# Troubleshooting Amazon Keyspaces (for Apache Cassandra)
This guide covers troubleshooting steps for various scenarios when working with Amazon Keyspaces (for Apache Cassandra). It includes information on resolving general errors, connection issues, capacity management problems, and Data Definition Language (DDL) errors.
+ **General errors**
+ Troubleshooting top-level exceptions like `NoNodeAvailableException`, `NoHostAvailableException`, and `AllNodesFailedException`.
+ Isolating underlying errors from Java driver exceptions.
+ Implementing retry policies and configuring connections correctly.
+ **Connection issues**
+ Resolving errors when connecting to Amazon Keyspaces endpoints using `cqlsh` or Apache Cassandra client drivers.
+ Troubleshooting VPC endpoint connections, Cassandra-stress connections, and IAM configuration errors.
+ Handling connection losses during data imports.
+ **Capacity management errors**
+ Recognizing and resolving insufficient capacity errors related to tables, partitions, and connections.
+ Monitoring relevant Amazon Keyspaces metrics in Amazon CloudWatch Logs.
+ Optimizing connections and throughput for improved performance.
+ **Data Definition Language (DDL) errors**
+ Troubleshooting errors when creating, accessing, or restoring keyspaces and tables.
+ Handling failures related to custom Time to Live (TTL) settings, column limits, and range deletes.
+ Considerations for heavy delete workloads.
For troubleshooting guidance specific to IAM access, see [Troubleshooting Amazon Keyspaces identity and access](security_iam_troubleshoot.md). For more information about security best practices, see [Security best practices for Amazon Keyspaces](best-practices-security.md).
**Topics**
+ [General errors](troubleshooting.general.md)
+ [Connection errors](troubleshooting.connecting.md)
+ [Capacity management errors](troubleshooting.serverless.md)
+ [Data definition language errors](troubleshooting.cql.md)
# Troubleshooting general errors in Amazon Keyspaces
Getting general errors? Here are some common issues and how to resolve them.
## General errors
You're getting one of the following top-level exceptions that can occur due to many different reasons.
+ `NoNodeAvailableException`
+ `NoHostAvailableException`
+ `AllNodesFailedException`
These exceptions are generated by the client driver and can occur either when you're establishing the control connection or when you're performing read/write/prepare/execute/batch requests.
When the error occurs while you're establishing the control connection, it's a sign that all the contact points specified in your application are unreachable. When the error occurs while performing read/write/prepare/execute queries, it indicates that all of the retries for that request have been exhausted. Each retry is attempted on a different node when you're using the default retry policy.
### How to isolate the underlying error from top-level Java driver exceptions
These general errors can be caused either by connection issues or when performing read/write/prepare/execute operations. Transient failures have to be expected in distributed systems, and should be handled by retrying the request. The Java driver doesn't automatically retry when connection errors are encountered, so it's recommended to implement the retry policy when establishing the driver connection in your application. For a detailed overview of connection best practices, see [Optimize client driver connections for the serverless environment](connections.md).
By default, the Java driver sets `idempotence` to false for all request, which means the Java driver doesn't automatically retry failed read/write/prepare request. To set `idempotence` to `true` and tell the driver to retry failed requests, you can do so in a few different ways. Here's one example how you can set idempotence programmatically for a single request in your Java application.
```
Statement s = new SimpleStatement("SELECT * FROM my_table WHERE id = 1");
s.setIdempotent(true);
```
Or you can set the default idempotence for your entire Java application programmatically as shown in the following example.
```
// Make all statements idempotent by default:
cluster.getConfiguration().getQueryOptions().setDefaultIdempotence(true);
//Set the default idempotency to true in your Cassandra configuration
basic.request.default-idempotence = true
```
Another recommendation is to create a retry policy at the application level. In this case, the application needs to catch the `NoNodeAvailableException` and retry the request. We recommend 10 retries with exponential backoff starting at 10ms and working up to 100ms with a total time of 1 second for all retries.
Another option is to apply the Amazon Keyspaces exponential retry policy when establishing the Java driver connection available on [Github](https://github.com/aws-samples/amazon-keyspaces-java-driver-helpers/blob/main/src/main/java/com/aws/ssa/keyspaces/retry/AmazonKeyspacesExponentialRetryPolicy.java).
Confirm that you have established connections to more than one node when using the default retry policy. You can do so using the following query in Amazon Keyspaces.
```
SELECT * FROM system.peers;
```
If the response for this query is empty, this indicates that you're working with a single node for Amazon Keyspaces. If you're using the default retry policy, there will be no retries because the default retry always occurs on a different node. To learn more about establishing connections over VPC endpoints, see [How to configure connections over VPC endpoints in Amazon Keyspaces](connections.md#connections.VPCendpoints).
For a step-by-step tutorial that shows how to establish a connection to Amazon Keyspaces using the Datastax 4.x Cassandra driver, see [Step-by-step tutorial to connect to Amazon Keyspaces using the 4.x DataStax Java driver for Apache Cassandra and the SigV4 authentication plugin](using_java_driver.md#java_tutorial.SigV4).
# Troubleshooting connection errors in Amazon Keyspaces
Having trouble connecting? Here are some common issues and how to resolve them.
## Errors connecting to an Amazon Keyspaces endpoint
Failed connections and connection errors can result in different error messages. The following section covers the most common scenarios.
**Topics**
+ [I can't connect to Amazon Keyspaces with cqlsh](#troubleshooting.connection.cqlsh)
+ [I can't connect to Amazon Keyspaces using a Cassandra client driver](#troubleshooting.connection.driver)
### I can't connect to Amazon Keyspaces with cqlsh
**You're trying to connect to an Amazon Keyspaces endpoint using cqlsh and the connection fails with a `Connection error`.**
If you try to connect to an Amazon Keyspaces table and cqlsh hasn't been configured properly, the connection fails. The following section provides examples of the most common configuration issues that result in connection errors when you're trying to establish a connection using cqlsh.
**Note**
If you're trying to connect to Amazon Keyspaces from a VPC, additional permissions are required. To successfully configure a connection using VPC endpoints, follow the steps in the [Tutorial: Connect to Amazon Keyspaces using an interface VPC endpoint](vpc-endpoints-tutorial.md).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you get a connection `timed out` error.**
This might be the case if you didn't supply the correct port, which results in the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com 9140 -u "USERNAME" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.199': error(None, "Tried connecting to [('3.234.248.199', 9140)]. Last error: timed out")})
```
To resolve this issue, verify that you're using port 9142 for the connection.
**You're trying to connect to Amazon Keyspaces using cqlsh, but you get a `Name or service not known` error.**
This might be the case if you used an endpoint that is misspelled or doesn't exist. In the following example, the name of the endpoint is misspelled.
```
# cqlsh cassandra.us-east-1.amazon.com 9142 -u "USERNAME" -p "PASSWORD" --ssl
Traceback (most recent call last):
File "/usr/bin/cqlsh.py", line 2458, in >module>
main(*read_options(sys.argv[1:], os.environ))
File "/usr/bin/cqlsh.py", line 2436, in main
encoding=options.encoding)
File "/usr/bin/cqlsh.py", line 484, in __init__
load_balancing_policy=WhiteListRoundRobinPolicy([self.hostname]),
File "/usr/share/cassandra/lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/policies.py", line 417, in __init__
socket.gaierror: [Errno -2] Name or service not known
```
To resolve this issue when you're using public endpoints to connect, select an available endpoint from [Service endpoints for Amazon Keyspaces](programmatic.endpoints.md), and verify that the name of the endpoint doesn't have any errors. If you're using VPC endpoints to connect, verify that the VPC endpoint information is correct in your cqlsh configuration.
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive an `OperationTimedOut` error.**
Amazon Keyspaces requires that SSL is enabled for connections to ensure strong security. The SSL parameter might be missing if you receive the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com -u "USERNAME" -p "PASSWORD"
Connection error: ('Unable to connect to any servers', {'3.234.248.192': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})
#
```
To resolve this issue, add the following flag to the cqlsh connection command.
```
--ssl
```
**You're trying to connect to Amazon Keyspaces using cqlsh, and you receive a `SSL transport factory requires a valid certfile to be specified` error.**
In this case, the path to the SSL/TLS certificate is missing, which results in the following error.
```
# cat .cassandra/cqlshrc
[connection]
port = 9142
factory = cqlshlib.ssl.ssl_transport_factory
#
# cqlsh cassandra.us-east-1.amazonaws.com -u "USERNAME" -p "PASSWORD" --ssl
Validation is enabled; SSL transport factory requires a valid certfile to be specified. Please provide path to the certfile in [ssl] section as 'certfile' option in /root/.cassandra/cqlshrc (or use [certfiles] section) or set SSL_CERTFILE environment variable.
#
```
To resolve this issue, add the path to the certfile on your computer. For more information, see [How to manually configure `cqlsh` connections for TLS](programmatic.cqlsh.md#encrypt_using_tls).
```
certfile = path_to_file/keyspaces-bundle.pem
```
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive a `No such file or directory` error.**
This might be the case if the path to the certificate file on your computer is wrong, which results in the following error.
```
# cat .cassandra/cqlshrc
[connection]
port = 9142
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
validate = true
certfile = /root/wrong_path/keyspaces-bundle.pem
#
# cqlsh cassandra.us-east-1.amazonaws.com -u "USERNAME" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.192': IOError(2, 'No such file or directory')})
#
```
To resolve this issue, verify that the path to the certfile on your computer is correct. For more information, see [How to manually configure `cqlsh` connections for TLS](programmatic.cqlsh.md#encrypt_using_tls).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive a `[X509] PEM lib` error.**
This might be the case if the SSL/TLS certificate `pem` file is not valid, which results in the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com -u "USERNAME" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.241': error(185090057, u"Tried connecting to [('3.234.248.241', 9142)]. Last error: [X509] PEM lib (_ssl.c:3063)")})
#
```
To resolve this issue make sure that you downloaded the required digital certificates. For more information, see [How to manually configure `cqlsh` connections for TLS](programmatic.cqlsh.md#encrypt_using_tls).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive an `unknown` SSL error.**
This might be the case if the SSL/TLS certificate `pem` file is empty, which results in the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com -u "USERNAME" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.220': error(0, u"Tried connecting to [('3.234.248.220', 9142)]. Last error: unknown error (_ssl.c:3063)")})
#
```
To resolve this issue make sure that you downloaded the required digital certificates. You can confirm this using the steps in the following topic [How to manually configure `cqlsh` connections for TLS](programmatic.cqlsh.md#encrypt_using_tls).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive a `SSL: CERTIFICATE_VERIFY_FAILED` error.**
This might be the case if the SSL/TLS certificate file could not be verified, which results in the following error.
```
Connection error: ('Unable to connect to any servers', {'3.234.248.223': error(1, u"Tried connecting to [('3.234.248.223', 9142)]. Last error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)")})
```
To resolve this issue make sure that you downloaded the required digital certificates. You can confirm this using the steps in the following topic [How to manually configure `cqlsh` connections for TLS](programmatic.cqlsh.md#encrypt_using_tls).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you're receiving a `Last error: timed out` error.**
This might be the case if you didn't configure an outbound rule for Amazon Keyspaces in your Amazon EC2 security group, which results in the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com 9142 -u "USERNAME" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.206': error(None, "Tried connecting to [('3.234.248.206', 9142)]. Last error: timed out")})
#
```
To confirm that this issue is caused by the configuration of the Amazon EC2 instance and not `cqlsh`, you can try to connect to your keyspace using the AWS CLI, for example with the following command.
```
aws keyspaces list-tables --keyspace-name 'my_keyspace'
```
If this command also times out, the Amazon EC2 instance is not correctly configured.
To confirm that you have sufficient permissions to access Amazon Keyspaces, you can use the AWS CloudShell to connect with `cqlsh`. If that connections gets established, you need to configure the Amazon EC2 instance.
To resolve this issue, confirm that your Amazon EC2 instance has an outbound rule that allows traffic to Amazon Keyspaces. If that is not the case, you need to create a new security group for the EC2 instance, and add a rule that allows outbound traffic to Amazon Keyspaces resources. To update the outbound rule to allow traffic to Amazon Keyspaces, choose **CQLSH/CASSANDRA** from the **Type** drop-down menu.
After creating the new security group with the outbound traffic rule, you need to add it to the instance. Select the instance and then choose **Actions**, then **Security**, and then **Change security groups**. Add the new security group with the outbound rule, but make sure that the default group also remains available.
For more information about how to view and edit EC2 outbound rules, see [Add rules to a security group in the Amazon EC2 User Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#adding-security-group-rule).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive an `Unauthorized` error.**
This might be the case if you're missing Amazon Keyspaces permissions in the IAM user policy, which results in the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com 9142 -u "testuser-at-12345678910" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.241': AuthenticationFailed('Failed to authenticate to 3.234.248.241: Error from server: code=2100 [Unauthorized] message="User arn:aws:iam::12345678910:user/testuser has no permissions."',)})
#
```
To resolve this issue, ensure that the IAM user `testuser-at-12345678910` has permissions to access Amazon Keyspaces. For examples of IAM policies that grant access to Amazon Keyspaces, see [Amazon Keyspaces identity-based policy examples](security_iam_id-based-policy-examples.md).
For troubleshooting guidance that's specific to IAM access, see [Troubleshooting Amazon Keyspaces identity and access](security_iam_troubleshoot.md).
**You're trying to connect to Amazon Keyspaces using cqlsh, but you receive a `Bad credentials` error.**
This might be the case if the user name or password is wrong, which results in the following error.
```
# cqlsh cassandra.us-east-1.amazonaws.com 9142 -u "USERNAME" -p "PASSWORD" --ssl
Connection error: ('Unable to connect to any servers', {'3.234.248.248': AuthenticationFailed('Failed to authenticate to 3.234.248.248: Error from server: code=0100 [Bad credentials] message="Provided username USERNAME and/or password are incorrect"',)})
#
```
To resolve this issue, verify that the *USERNAME* and *PASSWORD* in your code match the user name and password you obtained when you generated [service-specific credentials](programmatic.credentials.ssc.md).
**Important**
If you continue to see errors when trying to connect with cqlsh, rerun the command with the `--debug` option and include the detailed output when contacting Support.
### I can't connect to Amazon Keyspaces using a Cassandra client driver
The following sections shows the most common errors when connecting with a Cassandra client driver.
**You're trying to connect to an Amazon Keyspaces table using the DataStax Java driver, but you receive an `NodeUnavailableException` error.**
If the connection on which the request is attempted is broken, it results in the following error.
```
[com.datastax.oss.driver.api.core.NodeUnavailableException: No connection was available to Node(endPoint=vpce-22ff22f2f22222fff-aa1bb234.cassandra.us-west-2.vpce.amazonaws.com/11.1.1111.222:9142, hostId=1a23456b-c77d-8888-9d99-146cb22d6ef6, hashCode=123ca4567)]
```
To resolve this issue, find the heartbeat value and lower it to 30 seconds if it's higher.
```
advanced.heartbeat.interval = 30 seconds
```
Then look for the associated time out and ensure the value is set to at least 5 seconds.
```
advanced.connection.init-query-timeout = 5 seconds
```
**You're trying to connect to an Amazon Keyspaces table using a driver and the SigV4 plugin, but you receive an `AttributeError` error.**
If the credentials are not correctly configured, it results in the following error.
```
cassandra.cluster.NoHostAvailable: (‘Unable to connect to any servers’,
{‘44.234.22.154:9142’: AttributeError(“‘NoneType’ object has no attribute ‘access_key’“)})
```
To resolve this issue, verify that you're passing the credentials associated with your IAM user or role when using the SigV4 plugin. The SigV4 plugin requires the following credentials.
+ `AWS_ACCESS_KEY_ID` – Specifies an AWS access key associated with an IAM user or role.
+ `AWS_SECRET_ACCESS_KEY`– Specifies the secret key associated with the access key. This is essentially the "password" for the access key.
To learn more about access keys and the SigV4 plugin, see [Create and configure AWS credentials for Amazon Keyspaces](access.credentials.md).
**You're trying to connect to an Amazon Keyspaces table using a driver, but you receive a `PartialCredentialsError` error.**
If the `AWS_SECRET_ACCESS_KEY` is missing, it can result in the following error.
```
cassandra.cluster.NoHostAvailable: (‘Unable to connect to any servers’, {‘44.234.22.153:9142’:
PartialCredentialsError(‘Partial credentials found in config-file, missing: aws_secret_access_key’)})
```
To resolve this issue, verify that you're passing both the `AWS_ACCESS_KEY_ID` and the `AWS_SECRET_ACCESS_KEY` when using the SigV4 plugin. To learn more about access keys and the SigV4 plugin, see [Create and configure AWS credentials for Amazon Keyspaces](access.credentials.md).
**You're trying to connect to an Amazon Keyspaces table using a driver, but you receive an `Invalid signature` error.**
This might be the case if any of the components required for the signature are wrong or not correctly defined for the session.
+ `AWS_ACCESS_KEY_ID`
+ `AWS_SECRET_ACCESS_KEY`
+ `AWS_DEFAULT_REGION`
The following error is an examples of invalid access keys.
```
cassandra.cluster.NoHostAvailable: (‘Unable to connect to any servers’, {‘11.234.11.234:9142’:
AuthenticationFailed(‘Failed to authenticate to 11.234.11.234:9142: Error from server: code=0100
[Bad credentials] message=“Authentication failure: Invalid signature”’)})
```
To resolve this issue, verify that the access keys and the AWS Region have been correctly configured for the SigV4 plugin to access Amazon Keyspaces. To learn more about access keys and the SigV4 plugin, see [Create and configure AWS credentials for Amazon Keyspaces](access.credentials.md).
#### My VPC endpoint connection doesn't work properly
**You're trying to connect to Amazon Keyspaces using VPC endpoints, but you're receiving token map errors or you are experiencing low throughput. **
This might be the case if the VPC endpoint connection isn't correctly configured.
To resolve these issues, verify the following configuration details. To follow a step-by-step tutorial to learn how to configure a connection over interface VPC endpoints for Amazon Keyspaces see [Tutorial: Connect to Amazon Keyspaces using an interface VPC endpoint](vpc-endpoints-tutorial.md).
1. Confirm that the IAM entity used to connect to Amazon Keyspaces has read/write access to the user table and read access to the system tables as shown in the following example.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"cassandra:Select",
"cassandra:Modify"
],
"Resource":[
"arn:aws:cassandra:us-east-1:111122223333:/keyspace/mykeyspace/table/mytable",
"arn:aws:cassandra:us-east-1:111122223333:/keyspace/system*"
]
}
]
}
```
1. Confirm that the IAM entity used to connect to Amazon Keyspaces has the required read permissions to access the VPC endpoint information on your Amazon EC2 instance as shown in the following example.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"ListVPCEndpoints",
"Effect":"Allow",
"Action":[
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeVpcEndpoints"
],
"Resource":"*"
}
]
}
```
**Note**
The managed policies `AmazonKeyspacesReadOnlyAccess_v2` and `AmazonKeyspacesFullAccess` include the required permissions to let Amazon Keyspaces access the Amazon EC2 instance to read information about available interface VPC endpoints.
For more information about VPC endpoints, see [Using interface VPC endpoints for Amazon Keyspaces](vpc-endpoints.md#using-interface-vpc-endpoints)
1. Confirm that the SSL configuration of the Java driver sets hostname validation to false as shown in this example.
```
hostname-validation = false
```
For more information about driver configuration, see [Step 2: Configure the driver](using_java_driver.md#java_tutorial.driverconfiguration).
1. To confirm that the VPC endpoint has been configured correctly, you can run the following statement from *within* your VPC.
**Note**
You can't use your local developer environment or the Amazon Keyspaces CQL editor to confirm this configuration, because they use the public endpoint.
```
SELECT peer FROM system.peers;
```
The output should look similar to this example and return between 2 to 6 nodes with private IPv4 addresses when connecting from an IPv4 network, depending on your VPC setup and AWS Region.
```
peer
---------------
192.0.2.0.15
192.0.2.0.24
192.0.2.0.13
192.0.2.0.7
192.0.2.0.8
(5 rows)
```
#### I can't connect using `cassandra-stress`
**You're trying to connect to Amazon Keyspaces using the `cassandra-stress` command, but you're receiving an `SSL context` error. **
This happens if you try to connect to Amazon Keyspaces, but you don't have the trustStore setup correctly. Amazon Keyspaces requires the use of Transport Layer Security (TLS) to help secure connections with clients.
In this case, you see the following error.
```
Error creating the initializing the SSL Context
```
To resolve this issue, follow the instructions to set up a trustStore as shown in this topic [Before you begin](using_java_driver.md#using_java_driver.BeforeYouBegin).
Once the trustStore is setup, you should be able to connect with the following command.
```
./cassandra-stress user profile=./profile.yaml n=100 "ops(insert=1,select=1)" cl=LOCAL_QUORUM -node "cassandra.eu-north-1.amazonaws.com" -port native=9142 -transport ssl-alg="PKIX" truststore="./cassandra_truststore.jks" truststore-password="trustStore_pw" -mode native cql3 user="user_name" password="password"
```
#### I can't connect using IAM identities
**You're trying to connect to an Amazon Keyspaces table using an IAM identity, but you're receiving an `Unauthorized` error. **
This happens if you try to connect to an Amazon Keyspaces table using an IAM identity (for example, an IAM user) without implementing the policy and giving the user the required permissions first.
In this case, you see the following error.
```
Connection error: ('Unable to connect to any servers', {'3.234.248.202': AuthenticationFailed('Failed to authenticate to 3.234.248.202:
Error from server: code=2100 [Unauthorized] message="User arn:aws:iam::1234567890123:user/testuser has no permissions."',)})
```
To resolve this issue, verify the permissions of the IAM user. To connect with a standard driver, a user must have at least `SELECT` access to the system tables, because most drivers read the system keyspaces/tables when they establish the connection.
For example IAM policies that grant access to Amazon Keyspaces system and user tables, see [Accessing Amazon Keyspaces tables](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-access-one-table).
To review the troubleshooting section specific to IAM, see [Troubleshooting Amazon Keyspaces identity and access](security_iam_troubleshoot.md).
#### I'm trying to import data with cqlsh and the connection to my Amazon Keyspaces table is lost
**You're trying to upload data to Amazon Keyspaces with cqlsh, but you're receiving connection errors.**
The connection to Amazon Keyspaces fails after the cqlsh client receives three consecutive errors of any type from the server. The cqlsh client fails with the following message.
```
Failed to import 1 rows: NoHostAvailable - , will retry later, attempt 3 of 100
```
To resolve this error, you need to make sure that the data to be imported matches the table schema in Amazon Keyspaces. Review the import file for parsing errors. You can try using a single row of data by using an INSERT statement to isolate the error.
The client automatically attempts to reestablish the connection.
# Troubleshooting capacity management errors in Amazon Keyspaces
Having trouble with serverless capacity? Here are some common issues and how to resolve them.
## Serverless capacity errors
This section outlines how to recognize errors related to serverless capacity management and how to resolve them. For example, you might observe insufficient capacity events when your application exceeds your provisioned throughput capacity.
Because Apache Cassandra is cluster-based software that is designed to run on a fleet of nodes, it doesn’t have exception messages related to serverless features such as throughput capacity. Most drivers only understand the error codes that are available in Apache Cassandra, so Amazon Keyspaces uses that same set of error codes to maintain compatibility.
To map Cassandra errors to the underlying capacity events, you can use Amazon CloudWatch to monitor the relevant Amazon Keyspaces metrics. Insufficient-capacity events that result in client-side errors can be categorized into these three groups based on the resource that is causing the event:
+ **Table** – If you choose **Provisioned** capacity mode for a table, and your application exceeds your provisioned throughput, you might observe insufficient-capacity errors. For more information, see [Configure read/write capacity modes in Amazon Keyspaces](ReadWriteCapacityMode.md).
+ **Partition** – You might experience insufficient-capacity events if traffic against a given partition exceeds 3,000 RCUs or 1,000 WCUs. We recommend distributing traffic uniformly across partitions as a best practice. For more information, see [Data modeling best practices: recommendations for designing data models](data-modeling.md).
+ **Connection** – You might experience insufficient throughput if you exceed the quota for the maximum number of operations per second, per connection. To increase throughput, you can increase the number of default connections when configuring the connection with the driver.
To learn how to configure connections for Amazon Keyspaces, see [How to configure connections in Amazon Keyspaces](connections.md#connections.howtoconfigure). For more information about optimizing connections over VPC endpoints, see [How to configure connections over VPC endpoints in Amazon Keyspaces](connections.md#connections.VPCendpoints).
To determine which resource is causing the insufficient-capacity event that is returning the client-side error, you can check the dashboard in the Amazon Keyspaces console. By default, the console provides an aggregated view of the most common capacity and traffic related CloudWatch metrics in the **Capacity and related metrics** section on the **Capacity** tab for the table.
To create your own dashboard using Amazon CloudWatch, check the following Amazon Keyspaces metrics.
+ `PerConnectionRequestRateExceeded` – Requests to Amazon Keyspaces that exceed the quota for the per-connection request rate. Each client connection to Amazon Keyspaces can support up to 3000 CQL requests per second. You can perform more than 3000 requests per second by creating multiple connections.
+ `ReadThrottleEvents` – Requests to Amazon Keyspaces that exceed the read capacity for a table.
+ `StoragePartitionThroughputCapacityExceeded` – Requests to an Amazon Keyspaces storage partition that exceed the throughput capacity of the partition. Amazon Keyspaces storage partitions can support up to 1000 WCU/WRU per second and 3000 RCU/RRU per second. To mitigate these exceptions, we recommend that you review your data model to distribute read/write traffic across more partitions.
+ `WriteThrottleEvents` – Requests to Amazon Keyspaces that exceed the write capacity for a table.
To learn more about CloudWatch, see [Monitoring Amazon Keyspaces with Amazon CloudWatch](monitoring-cloudwatch.md). For a list of all available CloudWatch metrics for Amazon Keyspaces, see [Amazon Keyspaces metrics and dimensions](metrics-dimensions.md).
**Note**
To get started with a custom dashboard that shows all commonly observed metrics for Amazon Keyspaces, you can use a prebuilt CloudWatch template available on GitHub in the [AWS samples](https://github.com/aws-samples/amazon-keyspaces-cloudwatch-cloudformation-templates) repository.
**Topics**
+ [Client-side errors](#troubleshooting.serverless.clientside)
+ [Write timeout errors during data import](#troubleshooting.serverless.writetimeout)
+ [Keyspace or table storage size](#troubleshooting.serverless.storagesize)
### I'm receiving `NoHostAvailable` insufficient capacity errors from my client driver
**You're seeing `Read_Timeout` or `Write_Timeout` exceptions for a table.**
Repeatedly trying to write to or read from an Amazon Keyspaces table with insufficient capacity can result in client-side errors that are specific to the driver.
Use CloudWatch to monitor your provisioned and actual throughput metrics, and insufficient capacity events for the table. For example, a read request that doesn’t have enough throughput capacity fails with a `Read_Timeout` exception and is posted to the `ReadThrottleEvents` metric. A write request that doesn’t have enough throughput capacity fails with a `Write_Timeout` exception and is posted to the `WriteThrottleEvents` metric. For more information about these metrics, see [Amazon Keyspaces metrics and dimensions](metrics-dimensions.md).
To resolve these issues, consider one of the following options.
+ Increase the *provisioned throughput* for the table, which is the maximum amount of throughput capacity an application can consume. For more information, see [Read capacity units and write capacity units](ReadWriteCapacityMode.Provisioned.md#ReadWriteCapacityMode.Provisioned.Units).
+ Let the service manage throughput capacity on your behalf with automatic scaling. For more information, see [Manage throughput capacity automatically with Amazon Keyspaces auto scaling](autoscaling.md).
+ Chose **On-demand** capacity mode for the table. For more information, see [Configure on-demand capacity mode](ReadWriteCapacityMode.OnDemand.md).
If you need to increase the default capacity quota for your account, see [Quotas for Amazon Keyspaces (for Apache Cassandra)](quotas.md).
**You're seeing errors related to exceeded partition capacity.**
When you're seeing the error `StoragePartitionThroughputCapacityExceeded` the partition capacity is temporarily exceeded. This might be automatically handled by adaptive capacity or on-demand capacity. We recommend reviewing your data model to distribute read/write traffic across more partitions to mitigate these errors. Amazon Keyspaces storage partitions can support up to 1000 WCU/WRU per second and 3000 RCU/RRU per second. To learn more about how to improve your data model to distribute read/write traffic across more partitions, see [Data modeling best practices: recommendations for designing data models](data-modeling.md).
`Write_Timeout` exceptions can also be caused by an elevated rate of concurrent write operations that include static and nonstatic data in the same logical partition. If traffic is expected to run multiple concurrent write operations that include static and nonstatic data within the same logical partition, we recommend writing static and nonstatic data separately. Writing the data separately also helps to optimize the throughput costs.
**You're seeing errors related to exceeded connection request rate.**
You're seeing `PerConnectionRequestRateExceeded` due to one of the following causes.
+ You might not have enough connections configured per session.
+ You might be getting fewer connections than available peers, because you don't have the VPC endpoint permissions configured correctly. For more information about VPC endpoint policies, see [Using interface VPC endpoints for Amazon Keyspaces](vpc-endpoints.md#using-interface-vpc-endpoints).
+ If you're using a 4.x driver, check to see if you have hostname validation enabled. The driver enables TLS hostname verification by default. This configuration leads to Amazon Keyspaces appearing as a single-node cluster to the driver. We recommend that you turn hostname verification off.
We recommend that you follow these best practices to ensure that your connections and throughput are optimized:
+ **Configure CQL query throughput tuning.**
Amazon Keyspaces supports up to 3,000 CQL queries per TCP connection per second, but there is no limit on the number of connections a driver can establish.
Most open-source Cassandra drivers establish a connection pool to Cassandra and load balance queries over that pool of connections. Amazon Keyspaces exposes 9 peer IP addresses to drivers. The default behavior of most drivers is to establish a single connection to each peer IP address. Therefore, the maximum CQL query throughput of a driver using the default settings will be 27,000 CQL queries per second.
To increase this number, we recommend that you increase the number of connections per IP address that your driver is maintaining in its connection pool. For example, setting the maximum connections per IP address to 2 will double the maximum throughput of your driver to 54,000 CQL queries per second.
+ **Optimize your single-node connections.**
By default, most open-source Cassandra drivers establish one or more connections to every IP address advertised in the `system.peers` table when establishing a session. However, certain configurations can lead to a driver connecting to a single Amazon Keyspaces IP address. This can happen if the driver is attempting SSL hostname validation of the peer nodes (for example, DataStax Java drivers), or when it's connecting through a VPC endpoint.
To get the same availability and performance as a driver with connections to multiple IP addresses, we recommend that you do the following:
+ Increase the number of connections per IP to 9 or higher depending on the desired client throughput.
+ Create a custom retry policy that ensures that retries are run against the same node. For more information, see
[How to configure the retry policy for connections in Amazon Keyspaces](connections.md#connections.retry-policies).
+ If you use VPC endpoints, grant the IAM entity that is used to connect to Amazon Keyspaces access permissions to query your VPC for the endpoint and network interface information. This improves load balancing and increases read/write throughput. For more information, see [Populating `system.peers` table entries with interface VPC endpoint information](vpc-endpoints.md#system_peers).
### I'm receiving write timeout errors during data import
**You're receiving a timeout error when uploading data using the `cqlsh` `COPY` command.**
```
Failed to import 1 rows: WriteTimeout - Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses]
message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 2, 'write_type': 'SIMPLE', 'consistency':
'LOCAL_QUORUM'}, will retry later, attempt 1 of 100
```
Amazon Keyspaces uses the `ReadTimeout` and `WriteTimeout` exceptions to indicate when a write request fails due to insufficient throughput capacity. To help diagnose insufficient capacity exceptions, Amazon Keyspaces publishes the following metrics in Amazon CloudWatch.
+ `WriteThrottleEvents`
+ `ReadThrottledEvents`
+ `StoragePartitionThroughputCapacityExceeded`
To resolve insufficient-capacity errors during a data load, lower the write rate per worker or the total ingest rate, and then retry to upload the rows. For more information, see [Step 4: Configure `cqlsh COPY FROM` settings](bulk-upload-config.md). For a more robust data upload option, consider using DSBulk, which is available from the [GitHub repository](https://github.com/datastax/dsbulk). For step-by-step instructions, see [Tutorial: Loading data into Amazon Keyspaces using DSBulk](dsbulk-upload.md).
### I can't see the actual storage size of a keyspace or table
**You can't see the actual storage size of the keyspace or table.**
To learn more about the storage size of your table, see [Evaluate your costs at the table level](CostOptimization_TableLevelCostAnalysis.md). You can also estimate storage size by starting to calculate the row size in a table. Detailed instructions for calculating the row size are available at [Estimate row size in Amazon Keyspaces](calculating-row-size.md).
# Troubleshooting data definition language errors in Amazon Keyspaces
Having trouble creating resources? Here are some common issues and how to resolve them.
## Data definition language errors
Amazon Keyspaces performs data definition language (DDL) operations asynchronously—for example, creating and deleting keyspaces and tables. If an application is trying to use the resource before it's ready, the operation fails.
You can monitor the creation status of new keyspaces and tables in the AWS Management Console, which indicates when a keyspace or table is pending or active. You can also monitor the creation status of a new keyspace or table programmatically by querying the system schema table. A keyspace or table becomes visible in the system schema when it's ready for use.
**Note**
To optimize the creation of keyspaces using CloudFormation, you can use this utility to convert CQL scripts into CloudFormation templates. The tool is available from the [GitHub repository](https://github.com/aws/amazon-keyspaces-cql-to-cfn-converter).
**Topics**
+ [Keyspace creation errors](#troubleshooting.cql.keyspace)
+ [Table creation errors](#troubleshooting.cql.table)
+ [I'm trying to restore a table using Amazon Keyspaces point-in-time recovery (PITR), but the restore fails](#troubleshooting.cql.pitr)
+ [I'm trying to use INSERT/UPDATE to edit custom Time to Live (TTL) settings, but the operation fails](#troubleshooting.cql.ttl)
+ [Columns exceeded](#troubleshooting.cql.upload)
+ [Range delete error](#troubleshooting.cql.rangedelete)
### I created a new keyspace, but I can't view or access it
**You're receiving errors from your application that is trying to access a new keyspace.**
If you try to access a newly created Amazon Keyspaces keyspace that is still being created asynchronously, you will get an error. The following error is an example.
```
InvalidRequest: Error from server: code=2200 [Invalid query] message="unconfigured keyspace mykeyspace"
```
The recommended design pattern to check when a new keyspace is ready for use is to poll the Amazon Keyspaces system schema tables (system\$1schema\$1mcs.\$1).
For more information, see [Check keyspace creation status in Amazon Keyspaces](keyspaces-create.md).
### I created a new table, but I can't view or access it
**You're receiving errors from your application that is trying to access a new table.**
If you try to access a newly created Amazon Keyspaces table that is still being created asynchronously, you will get an error. For example, trying to query a table that isn't available yet fails with an `unconfigured table` error.
```
InvalidRequest: Error from server: code=2200 [Invalid query] message="unconfigured table mykeyspace.mytable"
```
Trying to view the table with `sync_table()` fails with a `KeyError`.
```
KeyError: 'mytable'
```
The recommended design pattern to check when a new table is ready for use is to poll the Amazon Keyspaces system schema tables (system\$1schema\$1mcs.\$1).
This is the example output for a table that is being created.
```
user-at-123@cqlsh:system_schema_mcs> select table_name,status from system_schema_mcs.tables where keyspace_name='example_keyspace' and table_name='example_table';
table_name | status
------------+----------
example_table | CREATING
(1 rows)
```
This is the example output for a table that is active.
```
user-at-123@cqlsh:system_schema_mcs> select table_name,status from system_schema_mcs.tables where keyspace_name='example_keyspace' and table_name='example_table';
table_name | status
------------+----------
example_table | ACTIVE
(1 rows)
```
For more information, see [Check table creation status in Amazon Keyspaces](tables-create.md).
### I'm trying to restore a table using Amazon Keyspaces point-in-time recovery (PITR), but the restore fails
If you're trying to restore an Amazon Keyspaces table with point-in-time recovery (PITR), and you see the restore process begin but not complete successfully, you might not have configured all of the required permissions that are needed by the restore process for this particular table.
In addition to user permissions, Amazon Keyspaces might require permissions to perform actions during the restore process on your principal's behalf. This is the case if the table is encrypted with a customer managed key, or if you're using IAM policies that restrict incoming traffic.
For example, if you're using condition keys in your IAM policy to restrict source traffic to specific endpoints or IP ranges, the restore operation fails. To allow Amazon Keyspaces to perform the table restore operation on your principal's behalf, you must add an `aws:ViaAWSService` global condition key in the IAM policy.
For more information about permissions to restore tables, see [Configure restore table IAM permissions for Amazon Keyspaces PITR](howitworks_restore_permissions.md).
### I'm trying to use INSERT/UPDATE to edit custom Time to Live (TTL) settings, but the operation fails
If you're trying to insert or update a custom TTL value, the operation might fail with the following error.
```
TTL is not yet supported.
```
To specify custom TTL values for rows or columns by using `INSERT` or `UPDATE` operations, you must first enable TTL for the table. You can enable TTL for a table using the `ttl` custom property.
For more information about enabling custom TTL settings for tables, see [Update table with custom Time to Live (TTL)](TTL-how-to-enable-custom-alter.md).
### I'm trying to upload data to my Amazon Keyspaces table and I get an error about exceeding the number of columns
**You're uploading data and have exceeded the number of columns that can be updated simultaneously.**
This error occurs when your table schema exceeds the maximum size of 350 KB. For more information, see [Quotas for Amazon Keyspaces (for Apache Cassandra)](quotas.md).
### I'm trying to delete data in my Amazon Keyspaces table and the deletion fails for the range
**You're trying to delete data by partition key and receive a range delete error.**
This error occurs when you're trying to delete more than 1,000 rows in one delete operation.
```
Range delete requests are limited by the amount of items that can be deleted in a single range.
```
For more information, see [Range delete](functional-differences.md#functional-differences.range-delete).
To delete more than 1,000 rows within a single partition, consider the following options.
+ Delete by partition – If the majority of partitions are under 1,000 rows, you can attempt to delete data by partition. If the partitions contain more than 1,000 rows, attempt to delete by the clustering column instead.
+ Delete by clustering column – If your model contains multiple clustering columns, you can use the column hierarchy to delete multiple rows. Clustering columns are a nested structure, and you can delete many rows by operating against the top-level column.
+ Delete by individual row – You can iterate through the rows and delete each row by its full primary key (partition columns and clustering columns).
+ As a best practice, consider splitting your rows across partitions – In Amazon Keyspaces, we recommend that you distribute your throughput across table partitions. This distributes data and access evenly across physical resources, which provides the best throughput. For more information, see [Data modeling best practices: recommendations for designing data models](data-modeling.md).
Consider also the following recommendations when you're planning delete operations for heavy workloads.
+ With Amazon Keyspaces, partitions can contain a virtually unbounded number of rows. This allows you to scale partitions “wider” than the traditional Cassandra guidance of 100 MB. It’s not uncommon for time series or ledgers to grow over a gigabyte of data over time.
+ With Amazon Keyspaces, there are no compaction strategies or tombstones to consider when you have to perform delete operations for heavy workloads. You can delete as much data as you want without impacting read performance.