After careful consideration, we have decided to discontinue Amazon Kinesis Data Analytics for SQL applications:
1. From **September 1, 2025**, we won't provide any bug fixes for Amazon Kinesis Data Analytics for SQL applications because we will have limited support for it, given the upcoming discontinuation.
2. From **October 15, 2025**, you will not be able to create new Kinesis Data Analytics for SQL applications.
3. We will delete your applications starting **January 27, 2026**. You will not be able to start or operate your Amazon Kinesis Data Analytics for SQL applications. Support will no longer be available for Amazon Kinesis Data Analytics for SQL from that time. For more information, see [Amazon Kinesis Data Analytics for SQL Applications discontinuation](discontinuation.md).
# Monitoring with Amazon CloudWatch
You can monitor applications using Amazon CloudWatch. CloudWatch collects and processes raw data from into readable, near real-time metrics. These statistics are retained for a period of two weeks. You can access the historical information and gain a better perspective on how your web application or service is performing. By default, metric data is automatically sent to CloudWatch. For more information, see [What Is Amazon CloudWatch?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatch.html) in the *Amazon CloudWatch User Guide*.
**Topics**
+ [Metrics and Dimensions](monitoring-metrics.md)
+ [Viewing Metrics and Dimensions](metrics-dimensions.md)
+ [Creating CloudWatch Alarms to Monitor](creating-alarms.md)
+ [Working with Amazon CloudWatch Logs](cloudwatch-logs.md)
# Metrics and Dimensions
The `AWS/KinesisAnalytics` namespace includes the following metrics.
| Metric | Description |
| --- | --- |
| `Bytes` | The number of bytes read (per input stream) or written (per output stream). Levels: Per input stream and per output stream |
| `KPUs` | The number of Kinesis Processing Units that are used to run your stream processing application. The average number of KPUs used each hour determines the billing for your application. Levels: Application-level |
| `MillisBehindLatest` | Indicates how far behind from the current time an application is reading from the streaming source. Levels: Application-level |
| `Records` | The number of records read (per input stream) or written (per output stream). Levels: Per input stream and per output stream |
| `Success` | 1 for each successful delivery attempt to the destination configured for your application; 0 for every failed delivery attempt. The average value of this metric indicates how many successful deliveries are performed. Levels: Per destination. |
| `InputProcessing.Duration` | The time taken for each AWS Lambda function invocation performed by . Levels: Per input stream |
| `InputProcessing.OkRecords ` | The number of records returned by a Lambda function that were marked with `Ok` status. Levels: Per input stream |
| `InputProcessing.OkBytes ` | The sum of bytes of the records returned by a Lambda function that were marked with `Ok` status. Levels: Per input stream |
| `InputProcessing.DroppedRecords ` | The number of records returned by a Lambda function that were marked with `Dropped` status. Levels: Per input stream |
| `InputProcessing.ProcessingFailedRecords ` | The number of records returned by a Lambda function that were marked with `ProcessingFailed` status. Levels: Per input stream |
| `InputProcessing.Success` | The number of successful Lambda invocations by . Levels: Per input stream |
| `LambdaDelivery.OkRecords` | The number of records returned by a Lambda function that were marked with `Ok` status. Levels: Per Lambda destination |
| `LambdaDelivery.DeliveryFailedRecords` | The number of records returned by a Lambda function that were marked with `DeliveryFailed` status. Levels: Per Lambda destination |
| `LambdaDelivery.Duration` | The time taken for each Lambda function invocation performed by . Levels: Per Lambda destination |
provides metrics for the following dimensions.
| Dimension | Description |
| --- | --- |
| `Flow` | Per input stream: Input Per output stream: Output |
| `Id` | Per input stream: Input Id Per output stream: Output Id |
# Viewing Metrics and Dimensions
When your application processes data streams, sends the following metrics and dimensions to CloudWatch. You can use the following procedures to view the metrics for .
On the console, metrics are grouped first by service namespace, and then by the dimension combinations within each namespace.
**To view metrics using the CloudWatch console**
1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. In the navigation pane, choose **Metrics**.
1. In the **CloudWatch Metrics by Category** pane for , choose a metrics category.
1. In the upper pane, scroll to view the full list of metrics.
**To view metrics using the AWS CLI**
+ At a command prompt, use the following command.
```
1. aws cloudwatch list-metrics --namespace "AWS/KinesisAnalytics" --region region
```
metrics are collected at the following levels:
+ Application
+ Input stream
+ Output stream
# Creating CloudWatch Alarms to Monitor
You can create an Amazon CloudWatch alarm that sends an Amazon SNS message when the alarm changes state. An alarm watches a single metric over a time period you specify. It performs one or more actions based on the value of the metric relative to a given threshold over a number of time periods. The action is a notification sent to an Amazon SNS topic or Auto Scaling policy.
Alarms invoke actions for sustained state changes only. For a CloudWatch alarm to invoke an action, the state must have changed and been maintained for a specified amount of time.
You can set alarms using the AWS Management Console, CloudWatch AWS CLI, or CloudWatch API, as described following.
**To set an alarm using the CloudWatch console**
1. Sign in to the AWS Management Console and open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).
1. Choose **Create Alarm**. The **Create Alarm Wizard** starts.
1. Choose **Kinesis Analytics Metrics**. Then scroll through the metrics to locate the metric you want to place an alarm on.
To display just metrics, search for the file system ID of your file system. Choose the metric to create an alarm for, and then choose **Next**.
1. Enter values for **Name**, **Description**, and **Whenever** for the metric.
1. If you want CloudWatch to send you an email when the alarm state is reached, in the **Whenever this alarm:** field, choose **State is ALARM**. In the **Send notification to:** field, choose an existing SNS topic. If you select **Create topic**, you can set the name and email addresses for a new email subscription list. This list is saved and appears in the field for future alarms.
**Note**
If you use **Create topic** to create a new Amazon SNS topic, the email addresses must be verified before they receive notifications. Emails are only sent when the alarm enters an alarm state. If this alarm state change happens before the email addresses are verified, they do not receive a notification.
1. In the **Alarm Preview** section, preview the alarm you’re about to create.
1. Choose **Create Alarm** to create the alarm.
**To set an alarm using the CloudWatch CLI**
+ Call `[mon-put-metric-alarm](https://docs.aws.amazon.com/AmazonCloudWatch/latest/cli/cli-mon-put-metric-alarm.html)`. For more information, see the [Amazon CloudWatch CLI Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/cli/).
**To set an alarm using the CloudWatch API**
+ Call `[PutMetricAlarm](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricAlarm.html)`. For more information, see the [Amazon CloudWatch API Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/).
# Working with Amazon CloudWatch Logs
If an application is misconfigured, it can transition to a running state during application start. Or it can update but not process any data into the in-application input stream. By adding a CloudWatch log option to the application, you can monitor for application configuration problems.
can generate configuration errors under the following conditions:
+ The Kinesis data stream used for input doesn't exist.
+ The Amazon Data Firehose delivery stream used for input doesn't exist.
+ The Amazon S3 bucket used as a reference data source doesn't exist.
+ The specified file in the reference data source in the S3 bucket doesn't exist.
+ The correct resource is not defined in the AWS Identity and Access Management (IAM) role that manages related permissions.
+ The correct permission is not defined in the IAM role that manages related permissions.
+ doesn't have permission to assume the IAM role that manages related permissions.
For more information about Amazon CloudWatch, see the [http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/).
## Adding the PutLogEvents Policy Action
needs permissions to write misconfiguration errors to CloudWatch. You can add these permissions to the IAM role that assumes, as described following. For more information on using an IAM role for , see [Identity and Access Management in Kinesis Data Analytics](iam-role.md).
### Trust Policy
To grant permissions to assume an IAM role, you can attach the following trust policy to the role.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "kinesisanalytics.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
### Permissions Policy
To grant an application permissions to write log events to CloudWatch from a resource, you can use the following IAM permissions policy.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "Stmt0123456789000",
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:us-east-1:123456789012:log-group:my-log-group:log-stream:my-log-stream*"
]
}
]
}
```
------
## Adding Configuration Error Monitoring
Use the following API actions to add a CloudWatch log option to a new or existing application or change a log option for an existing application.
**Note**
You can currently only add a CloudWatch log option to an application by using API actions. You can't add CloudWatch log options by using the console.
### Adding a CloudWatch Log Option When Creating an Application
The following code example demonstrates how to use the `CreateApplication` action to add a CloudWatch log option when you create an application. For more information on `Create_Application`, see [CreateApplication](API_CreateApplication.md).
```
{
"ApplicationCode": "",
"ApplicationDescription": "",
"ApplicationName": "",
"Inputs": [ ... ],
"Outputs": [ ... ],
"CloudWatchLoggingOptions": [{
"LogStreamARN": "",
"RoleARN": ""
}]
}
```
### Adding a CloudWatch Log Option to an Existing Application
The following code example demonstrates how to use the `AddApplicationCloudWatchLoggingOption` action to add a CloudWatch log option to an existing application. For more information about `AddApplicationCloudWatchLoggingOption`, see [AddApplicationCloudWatchLoggingOption](API_AddApplicationCloudWatchLoggingOption.md).
```
{
"ApplicationName": "",
"CloudWatchLoggingOption": {
"LogStreamARN": "",
"RoleARN": ""
},
"CurrentApplicationVersionId":
}
```
### Updating an Existing CloudWatch Log Option
The following code example demonstrates how to use the `UpdateApplication` action to modify an existing CloudWatch log option. For more information about `UpdateApplication`, see [UpdateApplication](API_UpdateApplication.md).
```
{
"ApplicationName": "",
"ApplicationUpdate": {
"CloudWatchLoggingOptionUpdates": [
{
"CloudWatchLoggingOptionId": "",
"LogStreamARNUpdate": "",
"RoleARNUpdate": ""
}
],
},
"CurrentApplicationVersionId":
}
```
### Deleting a CloudWatch Log Option from an Application
The following code example demonstrates how to use the `DeleteApplicationCloudWatchLoggingOption` action to delete an existing CloudWatch log option. For more information about `DeleteApplicationCloudWatchLoggingOption`, see [DeleteApplicationCloudWatchLoggingOption](API_DeleteApplicationCloudWatchLoggingOption.md).
```
{
"ApplicationName": "",
"CloudWatchLoggingOptionId": "",
"CurrentApplicationVersionId":
}
```
## Configuration Errors
The following sections contain details about errors that you might see in Amazon CloudWatch Logs from a misconfigured application.
### Error Message Format
Error messages generated by application misconfiguration appear in the following format.
```
{
"applicationARN": "string",
"applicationVersionId": integer,
"messageType": "ERROR",
"message": "string",
"inputId": "string",
"referenceId": "string",
"errorCode": "string"
"messageSchemaVersion": "integer",
}
```
The fields in an error message contain the following information:
+ `applicationARN`: The Amazon Resource Name (ARN) of the generating application, for example: `arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp`
+ `applicationVersionId`: The version of the application at the time the error was encountered. For more information, see [ApplicationDetail](API_ApplicationDetail.md).
+ `messageType`: The message type. Currently, this type can be only `ERROR`.
+ `message`: The details of the error, for example:
```
There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.
```
+ `inputId`: The ID associated with the application input. This value is only present if this input is the cause of the error. This value is not present if `referenceId` is present. For more information, see [DescribeApplication](API_DescribeApplication.md).
+ `referenceId`: The ID associated with the application reference data source. This value is only present if this source is the cause of the error. This value is not present if `inputId` is present. For more information, see [DescribeApplication](API_DescribeApplication.md).
+ `errorCode`: The identifier for the error. This ID is either `InputError` or `ReferenceDataError`.
+ `messageSchemaVersion`: A value that specifies the current message schema version, currently `1`. You can check this value to see if the error message schema has been updated.
### Errors
The errors that might appear in CloudWatch Logs for include the following.
#### Resource Does Not Exist
If an ARN is specified for a Kinesis input stream that doesn't exist, but the ARN is syntactically correct, an error like the following is generated.
```
{
"applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp",
"applicationVersionId": "5",
"messageType": "ERROR",
"message": "There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.",
"inputId":"1.1",
"errorCode": "InputError",
"messageSchemaVersion": "1"
}
```
If an incorrect Amazon S3 file key is used for reference data, an error like the following is generated.
```
{
"applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp",
"applicationVersionId": "5",
"messageType": "ERROR",
"message": "There is a problem related to the configuration of your reference data. Please check that the bucket and the file exist, the role has the correct permissions to access these resources and that Kinesis Analytics can assume the role provided.",
"referenceId":"1.1",
"errorCode": "ReferenceDataError",
"messageSchemaVersion": "1"
}
```
#### Role Does Not Exist
If an ARN is specified for an IAM input role that doesn't exist, but the ARN is syntactically correct, an error like the following is generated.
```
{
"applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp",
"applicationVersionId": "5",
"messageType": "ERROR",
"message": "There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.",
"inputId":null,
"errorCode": "InputError",
"messageSchemaVersion": "1"
}
```
#### Role Does Not Have Permissions to Access the Resource
If an input role is used that doesn't have permission to access the input resources, such as a Kinesis source stream, an error like the following is generated.
```
{
"applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp",
"applicationVersionId": "5",
"messageType": "ERROR",
"message": "There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.",
"inputId":null,
"errorCode": "InputError",
"messageSchemaVersion": "1"
}
```