Create an experiment

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

In the navigation pane, choose Application Signals, Evidently.

Select the button next to the name of the project and choose Project actions, Create experiment.

For Experiment name, enter a name to be used to identify this feature within this project.

You can optionally add a description.

Choose either Select from existing features or Add new feature.

If you are using an existing feature, select it under Feature name.

If you choose Add new feature, do the following:

1. For Feature name, enter a name to be used to identify this feature within this project. You can also optionally enter a description.

2. For Feature variations, for Variation type choose Boolean, Long, Double, or String. The type defines which type of value is used for each variation. For more information, see Variation types.

3. Add up to five variations for your feature. The Value for each variation must be valid for the Variation type that you selected.

   Specify one of the variations to be the default. This is the baseline that the other variations will be compared to, and should be the variation that is being served to your users now. If you stop an experiment that uses this feature, the default variation is then served to the percentage of users that were in the experiment previously.

4. Choose Sample code. The code example shows what you need to add to your application to set up the variations and assign user sessions to them. You can choose between JavaScript, Java, and Python for the code.

   You don't need to add the code to your application right now, but you must do so before you start the experiment. For more information, see Adding code to your application.

For Audience, optionally select a segment that you have created if you want this experiment to apply only to the users who match that segment. For more information about segments, see Use segments to focus your audience.

For Traffic split for the experiment, specify the percentage of the selected audience whose sessions will be used in the experiment. Then allocate the traffic for the different variations that the experiment uses.

If a launch and an experiment are both running at the same time for the same feature, the audience is first directed to the launch. Then, the percentage of traffic specified for the launch is taken from the overall audience. After that, the percentage that you specify here is the percentage of the remaining audience that is used for the experiment. Any remaining traffic after that is served the default variation.

For Metrics, choose the metrics to use to evaluate the variations during the experiment. You must use at least one metric for evaluation.

1. For Metric source, choose whether to use CloudWatch RUM metrics or custom metrics.

2. Enter a name for the metric. For Goal, choose Increase if you want a higher value for the metric to indicate a better variation. Choose Decrease if you want a lower value for the metric to indicate a better variation.

3. If you are using a custom metric, you can create the metric here using an Amazon EventBridge rule. To create a custom metric, do the following:

   • Under Metric rule, for Entity ID, enter a way to identify the entity, This can be a user or session that does an action that causes a metric value to be recorded. An example is userDetails.userID.

   • For Value key, enter the value that is to be tracked to produce the metric.

   • Optionally, enter a name for the units for the metric. This unit name is for display purposes only, for use on graphs in the Evidently console.

   You can use RUM metrics only if you have set up RUM to monitor this application. For more information, see CloudWatch RUM.

   Note

   If you use RUM metrics, and the app monitor is not configured to sample 100% of user sessions, then not all of the user sessions in the experiment will send metrics to Evidently. To ensure that the experiment metrics are accurate, we recommend that the app monitor uses 100% of user sessions for sampling.

4. (Optional) To add more metrics to evaluate, choose Add metric. You can evaluate as many as three metrics during the experiment.

(Optional) To create CloudWatch alarms to use with this experiment, choose CloudWatch alarms. The alarms can monitor whether the difference in results between each variation and the default variation is larger than a threshold that you specify. If a variation's performance is worse than the default variation, and the difference is greater than your threshold, it goes into alarm state and notifies you.

Creating an alarm here creates one alarm for each variation that is not the default variation.

If you create an alarm, specify the following:

You can choose Add new alarm again to add more alarms to the experiment.

Next, choose Set notifications for the alarm and select or create an Amazon Simple Notification Service topic to send alarm notifications to. For more information, see Setting up Amazon SNS notifications,

(Optional) To add tags to this experiment, choose Tags, Add new tag.

Then, for Key, enter a name for the tag. You can add an optional value for the tag in Value.

To add another tag, choose Add new tag again.

For more information, see Tagging AWS Resources.

Choose Create experiment.

If you haven't already, build the feature variants into your application.

Choose Done. The experiment does not start until you start it.

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

In the navigation pane, choose Application Signals, Evidently.

Choose the name of the project.

Choose the Experiments tab.

Choose the button next to the name of the experiment, and choose Actions, Start experiment.

(Optional) To view or modify the experiment settings you made when you created it, choose Experiment setup.

Choose a time for the experiment to end.

Choose Start experiment.

The experiment starts immediately.