# Shadow tests
With Amazon SageMaker AI you can evaluate any changes to your model serving infrastructure by comparing its performance against the currently deployed infrastructure. This practice is known as shadow testing. Shadow testing can help you catch potential configuration errors and performance issues before they impact end users. With SageMaker AI, you don't need to invest in building your shadow testing infrastructure, so you can focus on model development.
You can use this capability to validate changes to any component of your production variant, namely the model, the container, or the instance, without any end user impact. It is useful in situations including but not limited to the following:
+ You are considering promoting a new model that has been validated offline to production, but want to evaluate operational performance metrics such as latency and error rate before making this decision.
+ You are considering changes to your serving infrastructure container, such as patching vulnerabilities or upgrading to newer versions, and want to assess the impact of these changes prior to promotion to production.
+ You are considering changing your ML instance and want to evaluate how the new instance would perform with live inference requests.
The SageMaker AI console provides a guided experience to manage the workflow of shadow testing. You can set up shadow tests for a predefined duration of time, monitor the progress of the test through a live dashboard, clean up upon completion, and act on the results. Select a production variant you want to test against, and SageMaker AI automatically deploys the new variant in shadow mode and routes a copy of the inference requests to it in real time within the same endpoint. Only the responses of the production variant are returned to the calling application. You can choose to discard or log the responses of the shadow variant for offline comparison. For more information on production and shadow variants, see [Validation of models in production](model-validation.md).
See [Create a shadow test](shadow-tests-create.md) for instructions on creating a shadow test.
**Note**
Certain endpoint features may make your endpoint incompatible with shadow tests. If your endpoint uses any of the following features, you cannot use shadow tests on your endpoint, and your request to set up shadow tests will lead to validation errors.
Serverless inference
Asynchronous inference
Marketplace containers
Multiple-container endpoints
Multi-model endpoints
Endpoints that use Inf1 (Inferentia-based) instances
# Create a shadow test
You can create a shadow test to compare the performance of a shadow variant against a production variant. You can run the test on an existing endpoint that is serving inference requests or you can create a new endpoint on which to run the test.
To create a shadow test you need to specify the following:
+ A *production variant* that receives and responds to 100 percent of the incoming inference requests.
+ A *shadow variant* that receives a percentage of the incoming requests, replicated from the production variant, but does not return any responses.
For each variant, you can use SageMaker AI to control the model, instance type, and instance count. You can configure the percentage of incoming requests, known as the traffic sampling percentage, that you want replicated to your shadow variant. SageMaker AI manages the replication of requests to your shadow variant and you can modify the traffic sampling percentage when your test is scheduled or running. You can also optionally turn on Data Capture to log requests and responses of your production and shadow variants.
**Note**
SageMaker AI supports a maximum of one shadow variant per endpoint. For an endpoint with a shadow variant, there can be a maximum of one production variant.
You can schedule the test to start at any time and continue for a specified duration. The default duration is 7 days and the maximum is 30 days. After the test is complete, the endpoint reverts to the state it was in prior to starting the test. This ensures that you do not have to manually clean up resources upon the completion of the test.
You can monitor a test that is running through a dashboard in the SageMaker AI console. The dashboard provides a side by side comparison of invocation metrics and instance metrics between the production and shadow variants, along with a tabular view with relevant metric statistics. This dashboard is also available for completed tests. Once you have reviewed the metrics, you can either choose to promote the shadow variant to be the new production variant or retain the existing production variant. Once you promote the shadow variant, it responds to all incoming requests. For more information, see [Promote a shadow variant](shadow-tests-complete.md#shadow-tests-complete-promote).
The following procedure describes how to create a shadow test through the SageMaker AI console. There are variations in the workflow depending on whether you want to use an existing endpoint or to create a new endpoint for the shadow test.
**Topics**
+ [Prerequisites](#shadow-tests-create-prerequisites)
+ [Enter shadow test details](#shadow-tests-create-console-shadow-test-details)
+ [Enter shadow test settings](#shadow-tests-create-console-shadow-test-settings)
## Prerequisites
Before creating a shadow test with the SageMaker AI console, you must have a SageMaker AI model ready to use. For more information about how to create a SageMaker AI model, see [Deploy models for real-time inference](realtime-endpoints-deploy-models.md).
You can get started with shadow tests with an existing endpoint with a production variant and a shadow variant, an existing endpoint with only a production variant, or just the SageMaker AI models you'd like to compare. Shadow tests support creating an endpoint and adding variants before your test begins.
**Note**
Certain endpoint features may make your endpoint incompatible with shadow tests. If your endpoint uses any of the following features, you cannot use shadow tests on your endpoint, and your request to set up shadow tests will lead to validation errors.
Serverless inference
Asynchronous inference
Marketplace containers
Multiple-container endpoints
Multi-model endpoints
Endpoints that use Inf1 (Inferentia-based) instances
## Enter shadow test details
To start creating your shadow test, fill out the **Enter shadow test details** page by doing the following:
1. Open the [SageMaker AI console](https://console.aws.amazon.com/sagemaker/).
1. In the left navigation panel, choose **Inference**, and then choose **Shadow tests**.
1. Choose **Create shadow test**.
1. Under **Name**, enter a name for the test.
1. (Optional) Under **Description**, enter a description for the test.
1. (Optional) Specify **Tags** using **Key** and **Value** pairs.
1. Choose **Next**.
## Enter shadow test settings
After filling out the **Enter shadow test details** page, fill out the **Enter shadow test settings** page. If you already have a SageMaker AI Inference endpoint and a production variant, follow the **Use an existing endpoint** workflow. If you don't already have an endpoint, follow the **Create a new endpoint** workflow.
------
#### [ Use an existing endpoint ]
If you want to use an existing endpoint for your test, fill out the **Enter shadow test settings** page by doing the following:
1. Choose a role that has the `AmazonSageMakerFullAccess` IAM policy attached.
1. Choose **Use an existing endpoint**, and then choose one of the available endpoints.
1. (Optional) To encrypt the storage volume on your endpoint, either choose an existing KMS key or choose **Enter a KMS key ARN** from the dropdown list under **Encryption key**. If you choose the second option, a field to enter the KMS key ARN appears. Enter the KMS key ARN in that field.
1. If you have multiple production variants behind that endpoint, remove the ones you don't want to use for the test. You can remove a model variant by selecting it and then choosing **Remove**.
1. If you do not already have a shadow variant, add a shadow variant. To add a shadow variant, do the following:
1. Choose **Add**.
1. Choose **Shadow variant**.
1. In the **Add model** dialog box, choose the model you want to use for your shadow variant.
1. Choose **Save**.
1. (Optional) In the preceding step, the shadow variant is added with the default settings. To modify these settings, select the shadow variant and choose **Edit**. The **Edit shadow variant** dialog box appears. For more information on filling out this dialog box, see [Edit a shadow test](shadow-tests-view-monitor-edit-individual.md).
1. In the **Schedule** section, enter the duration of the test by doing the following:
1. Choose the box under **Duration**. A popup calender appears.
1. Select the start and end dates from the calender, or enter the start and end dates in the fields for **Start date** and **End date**, respectively.
1. (Optional) For the fields **Start time** and **End time**, enter the start and end times, respectively, in the 24 hour format.
1. Choose **Apply**.
The minimum duration is 1 hour, and the maximum duration is 30 days.
1. (Optional) Turn on **Enable data capture** to save inference request and response information from your endpoint to an Amazon S3 bucket, and then enter the location of the Amazon S3 bucket.
1. Choose **Create shadow test**.
------
#### [ Create a new endpoint ]
If you don't have an existing endpoint, or you want to create a new endpoint for your test, fill out the **Enter shadow test settings** page by doing the following:
1. Choose a role that has the `AmazonSageMakerFullAccess` IAM policy attached.
1. Choose **Create a new endpoint**.
1. Under **Name**, enter a name for the endpoint.
1. Add one production variant and one shadow variant to the endpoint:
+ To add a production variant choose **Add**, and then choose **Production variant**. In the **Add model** dialog box, choose the model you want to use for your production variant, and then choose **Save**.
+ To add a shadow variant choose **Add**, and then choose **Shadow variant**. In the **Add model** dialog box, choose the model you want to use for your shadow variant, and then choose **Save**.
1. (Optional) In the preceding step, the shadow variant is added with the default settings. To modify these settings, select the shadow variant and choose **Edit**. The **Edit shadow variant** dialog box appears. For more information on filling out this dialog box, see [Edit a shadow test](shadow-tests-view-monitor-edit-individual.md).
1. In the **Schedule** section, enter the duration of the test by doing the following:
1. Choose the box under **Duration**. A popup calender appears.
1. Select the start and end dates from the calender, or enter the start and end dates under **Start date** and **End date**, respectively.
1. (Optional) Under **Start time** and **End time**, enter the start and end times, respectively, in the 24 hour format.
1. Choose **Apply**.
The minimum duration is 1 hour, and the maximum duration is 30 days.
1. (Optional) Turn on **Enable data capture** to save inference request and response information from your endpoint to an Amazon S3 bucket, and then enter the location of the Amazon S3 bucket.
1. Choose **Create shadow test**.
------
After completing the preceding procedures, you should now have a test scheduled to begin at your specified start date and time. You can view the progress of the test from a dashboard. For more information about viewing your test and the actions you can take, see [How to view, monitor, and edit shadow tests](shadow-tests-view-monitor-edit.md).
# How to view, monitor, and edit shadow tests
You can view the statuses of your shadow tests, monitor their progress from a dashboard, and perform actions, such as starting or stopping an test early or deleting an test. The following topics show how you can view and modify your shadow tests using the SageMaker AI console.
**Topics**
+ [View shadow tests](shadow-tests-view-monitor-edit-list.md)
+ [Monitor a shadow test](shadow-tests-view-monitor-edit-dashboard.md)
+ [Start a shadow test early](shadow-tests-view-monitor-edit-start.md)
+ [Delete a shadow test](shadow-tests-view-monitor-edit-delete.md)
+ [Edit a shadow test](shadow-tests-view-monitor-edit-individual.md)
# View shadow tests
You can view the statuses of all of your shadow tests on the **Shadow tests** page on the SageMaker AI console.
To view your tests in the console, do the following:
1. Open the [SageMaker AI console](https://console.aws.amazon.com/sagemaker/).
1. In the navigation panel, choose **Inference**.
1. Choose **Shadow tests** to view the page that lists all of your shadow tests. The page should look like the following screenshot, with all the tests listed under the **Shadow test** section.
![\[List of all shadow tests.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/juxtaposer/shadow-test-landing-page.png)
You can see the status of a test in the console on the **Shadow tests** page by checking the **Status** field for the test.
The following are the possible statuses for a test:
+ `Creating` – SageMaker AI is creating your test.
+ `Created` – SageMaker AI has finished creating your test, and it will begin at the scheduled time.
+ `Updating` – When you make changes to your test, your test shows as updating.
+ `Starting` – SageMaker AI is beginning your test.
+ `Running` – Your test is in progress.
+ `Stopping` – SageMaker AI is stopping your test.
+ `Completed` – Your test has completed.
+ `Cancelled` – When you conclude your test early, it shows as cancelled.
# Monitor a shadow test
You can view the details of a shadow test and monitor it while it is in progress or after it has completed. SageMaker AI presents a live dashboard comparing the operational metrics like model latency, and error rate aggregated, of the production and shadow variants.
To view the details of an individual test in the console, do the following:
1. Select the test you want to monitor from the **Shadow test** section on the **Shadow tests** page.
1. From the **Actions** dropdown list, choose **View**. An overview page with the details of the test and a metrics dashboard appears.
The overview page has the following three sections.
**Summary**
This section summarizes the progress and status of the test. It also shows the summary statistics of the metric chosen from the **Select metric** dropdown list in the **Metrics** subsection. The following screenshot shows this section.
![\[Summary section of the overview page.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/juxtaposer/shadow-test-summary.png)
In the preceding screenshot, the **Settings**, and **Details** tabs show the settings that you selected, and the details that you entered when creating the test.
**Analysis**
This section shows a metrics dashboard with separate graphs for the following metrics:
+ `Invocations`
+ `InvocationsPerInstance`
+ `ModelLatency`
+ `Invocation4XXErrors`
+ `Invocation5XXErrors`
+ `InvocationModelErrors`
+ `CPUUtilization`
+ `MemoryUtilization`
+ `DiskUtilization`
The last three metrics monitor the model container runtime resource usage. The rest are CloudWatch metrics that you can use to analyse the performance of your variant. In general, fewer errors indicate a more stable model. A lower latency indicates either a faster model or a faster infrastructure. For more information about CloudWatch metrics, see [SageMaker AI endpoint invocation metrics](monitoring-cloudwatch.md#cloudwatch-metrics-endpoint-invocation). The following screenshot shows the metrics dashboard.
![\[Metrics analysis dashboard.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/juxtaposer/shadow-test-analysis.png)
**Environment**
This section shows the variants that you compared in the test. If you are satisfied by the performance of the shadow variant, based on the aforementioned metrics, you can promote the shadow variant to production, by choosing **Deploy shadow variant**. For more details about deploying a shadow variant, see [Promote a shadow variant](shadow-tests-complete.md#shadow-tests-complete-promote). You can also change the traffic sampling percentage, and continue testing, by choosing **Edit traffic**. For more details about editing a shadow variant, see [Edit a shadow test](shadow-tests-view-monitor-edit-individual.md). The following screenshot shows this section.
![\[Environment section of the overview page.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/juxtaposer/shadow-test-environment.png)
# Start a shadow test early
You can start your test before its scheduled start time. If the new duration of the test exceeds 30 days, SageMaker AI automatically sets the end of the test to 30 days after the new start time. This action starts the test immediately. If you want to change the start or end time of the test, see [Edit a shadow test](shadow-tests-view-monitor-edit-individual.md).
To immediately start your test, before its scheduled start time, through the console, do the following:
1. Select the test you want to start immediately from the **Shadow test** section on the **Shadow tests** page.
1. From the **Actions** dropdown list, choose **Start**. The **Start shadow test?** dialog box appears.
1. Choose **Start now**.
# Delete a shadow test
You can delete a test that you no longer need. Deleting your test only deletes the test metadata and not your endpoint, variants, or data captured in Amazon S3. If you want your endpoint to stop running, you must delete your endpoint. For more information about deleting an endpoint, see [Delete Endpoints and Resources](realtime-endpoints-delete-resources.md)
To delete a test through the console, do the following:
1. Select the test you want to delete from the **Shadow test** section on the **Shadow tests** page.
1. From the **Actions** dropdown list, choose **Delete**. The **Delete shadow test** dialog box appears.
1. In the **To confirm deletion, type *delete* in the field.** text box, enter **delete**.
1. Choose **Delete**.
# Edit a shadow test
You can modify both scheduled and in-progress tests. Before your test starts, you can change the description, the shadow variant configuration, the start date, and the end date of the test. You can also turn on or turn off data capture.
After your test starts, you can only change the description, the traffic sampling percentage for the shadow variant, and the end date.
To edit the details of your test through the console, do the following:
1. Select the test you want to edit from the **Shadow test** section on the **Shadow tests** page.
1. From the **Actions** dropdown list, choose **Edit**. The **Enter shadow test details** page appears.
1. (Optional) Under **Description**, enter a description of your test.
1. Choose **Next**. The **Enter shadow test settings** page appears.
1. (Optional) To edit your shadow variant, do the following:
1. Select the shadow variant and choose **Edit**. The **Edit shadow variant** dialog box appears. If your test has already started, then you can only change the traffic sampling percentage.
1. (Optional) Under **Name**, enter the new name to replace the old name.
1. (Optional) Under **Traffic sample**, enter the new traffic sampling percentage to replace the old traffic sampling percentage.
1. (Optional) Under **Instance type**, select the new instance type from the dropdown list.
1. (Optional) Under **Instance count**, enter the new instance count to replace the old instance count.
1. Choose **Apply**.
You cannot change the model in your shadow variant using the above procedure. If you want to change the model, first remove the shadow variant by selecting it and choosing **Remove**. Then add a new shadow variant.
1. (Optional) To edit the duration of the test, do the following:
1. Choose the box under **Duration** in the **Schedule** section. A popup calender appears.
1. If your test is yet to start, you can change both the start and end dates. Select the new start and end dates from the calender, or enter the new start and end dates under **Start date** and **End date**, respectively.
If your test has already started, you can only change the end date. Enter the new end date under **End date**.
1. (Optional) If your test is yet to start, you can change both the start and end times. Enter the new start and end times under **Start time**, and **End time**, respectively, in the 24 hour format.
If your test has already started, you can only change the end time. Enter the new end time under **End time**, in the 24 hour format.
1. Choose **Apply**.
1. (Optional) Turn on or turn off **Enable data capture**.
1. Choose **Update shadow test**.
# Complete a shadow test
Your test automatically completes at the end of the scheduled duration, or you can stop an in-progress test early. After your test has completed, the test’s status in the **Shadow tests** section on the **Shadow tests** page shows as **Complete**. Then you can review and analyze the final metrics of your test.
You can use the metrics dashboard to decide whether to promote the shadow variant to production. For more information about analyzing the metrics dashboard of your test, see [Monitor a shadow test](shadow-tests-view-monitor-edit-dashboard.md).
For instructions on how to complete your test before the end of its scheduled completion time, see [Complete a shadow test early](#shadow-tests-complete-early).
For instructions on promoting your shadow variant to production, see [Promote a shadow variant](#shadow-tests-complete-promote).
## Complete a shadow test early
One reason you might want to complete an in-progress shadow test is if you’ve decided that the metrics for your shadow variant look good and you want to promote it to production. You might also decide to complete the test if one or more of the variants aren’t performing well.
To complete your test before its scheduled end date, do the following:
1. Select the test you want to mark complete from the **Shadow tests** section on the **Shadow tests** page.
1. From the **Actions** dropdown list, choose **Complete**, and the **Complete shadow test** dialog box appears.
1. In the dialog box, choose one of the following options:
+ **Yes, deploy shadow variant**
+ **No, remove shadow variant**
1. (Optional) In the **Comment** text box, enter your reason for completing the test before its scheduled end time.
1.
1. If you decided to deploy the shadow variant, choose **Complete and proceed to deploy**. The **Deploy shadow variant** page appears. For instructions on how to fill out this page, see [Promote a shadow variant](#shadow-tests-complete-promote).
1. If you decide to remove the shadow variant, choose **Confirm**.
## Promote a shadow variant
If you’ve decided that you want to replace your production variant with your shadow variant, you can update your endpoint and promote your shadow variant to respond to inference requests. This removes your current production variant from production and replaces it with your shadow variant.
If your shadow test is still in-progress, you must first complete your test. To complete your shadow test before its scheduled end, follow the instructions in [Complete a shadow test early](#shadow-tests-complete-early) before continuing with this section.
When you promote a shadow variant to production, you have the following options for the instance count of the shadow variant.
+ You can retain the instance count and type from the production variant. If you select this option, then your shadow variant launches in production with the current instance count, ensuring that your model can continue to process request traffic at the same scale.
+ You can retain the instance count and type of your shadow variant. If you want to use this option, we recommend that you shadow test with 100 percent traffic sampling to ensure that the shadow variant can process request traffic at the current scale.
+ You can use custom values for the instance count and type. If you want to use this option, we recommend that you shadow test with 100 percent traffic sampling to ensure that the shadow variant can process request traffic at the current scale.
Unless you are validating the instance type or count or both of the shadow variant, we highly recommend that you retain the instance count and type from the production variant when promoting your shadow variant.
To promote your shadow variant, do the following:
1. If your test has completed, do the following:
1. Select the test from the **Shadow test** section on the **Shadow tests** page.
1. From the **Actions** dropdown list, choose **View**. The dashboard appears.
1. Choose **Deploy shadow variant** in the **Environment** section. The **Deploy shadow variant** page appears.
If your test has not completed, see [Complete a shadow test early](#shadow-tests-complete-early) to complete it.
1. In the **Variant settings** section, select one of the following options:
+ **Retain production settings**
+ **Retain shadow settings**
+ **Custom instance settings**
If you selected **Custom instance settings**, do the following:
1. Select the instance type from the **Instance type** dropdown list.
1. Under **Instance count**, enter the number of instances.
1. In **Enter 'deploy' to confirm deployment** text box, enter **deploy**.
1. Choose **Deploy shadow variant**.
Your SageMaker AI Inference endpoint is now using the shadow variant as your production variant, and your production variant has been removed from the endpoint.
# Best practices
When creating an inference experiment, keep the following information in mind:
+ **Traffic sampling percentage** – Sampling 100 percent of the inference requests lets you validate that your shadow variant can handle production traffic when promoted. You may start off with a lower traffic sampling percentage and dial up as you gain confidence in your variant, but it is best practice to ensure that you’ve increased the traffic to 100 percent prior to promotion.
+ **Instance type** – Unless you are using shadow variants to evaluate alternate instance types or sizes, we recommend that you use the same instance type, size, and count so that you can be certain that your shadow variant can handle the volume of inference requests after you promote it.
+ **Auto scaling** – To ensure that your shadow variant can respond to spikes in the number of inference requests or changes in inference requests patterns, we highly recommend that you configure autoscaling on your shadow variants. To learn how to configure autoscaling, see [Automatic scaling of Amazon SageMaker AI models](endpoint-auto-scaling.md). If you have configured autoscaling, you can also validate changes to autoscaling policies without causing impact to users.
+ **Metrics monitoring** – After you initiate a shadow experiment and have sufficient invocations, monitor the metrics dashboard to ensure that the metrics such as latency and error rate are within acceptable bounds. This helps you catch misconfigurations early and take corrective action. For information about how to monitor the metrics of an in-progress inference experiment, see [How to view, monitor, and edit shadow tests](shadow-tests-view-monitor-edit.md).