Troubleshooting for RDS Proxy - Amazon Aurora
Verifying connectivity for a proxyCommon issues and solutions

Troubleshooting for RDS Proxy

Following, you can find troubleshooting ideas for some common RDS Proxy issues and information on CloudWatch logs for RDS Proxy.

In the RDS Proxy logs, each entry is prefixed with the name of the associated proxy endpoint. This name can be the name that you specified for a user-defined endpoint. Or, it can be the special name default for the default endpoint of a proxy that performs read/write requests. For more information about proxy endpoints, see Working with Amazon RDS Proxy endpoints.

Verifying connectivity for a proxy

You can use the following commands to verify that all components such as the proxy, database, and compute instances in the connection can communicate with the each other.

Examine the proxy itself using the describe-db-proxies command. Also examine the associated target group using the describe-db-proxy-target-groups command. Check that the details of the targets match the Aurora cluster that you intend to associate with the proxy. Use commands such as the following. 

aws rds describe-db-proxies --db-proxy-name $DB_PROXY_NAME
aws rds describe-db-proxy-target-groups --db-proxy-name $DB_PROXY_NAME

To confirm that the proxy can connect to the underlying database, examine the targets specified in the target groups using the describe-db-proxy-targets command. Use a command such as the following. 

aws rds describe-db-proxy-targets --db-proxy-name $DB_PROXY_NAME

The output of the describe-db-proxy-targets command includes a TargetHealth field. You can examine the fields State, Reason, and Description inside TargetHealth to check if the proxy can communicate with the underlying DB instance.

  • A State value of AVAILABLE indicates that the proxy can connect to the DB instance.

  • A State value of UNAVAILABLE indicates a temporary or permanent connection problem. In this case, examine the Reason and Description fields. For example, if Reason has a value of PENDING_PROXY_CAPACITY, try connecting again after the proxy finishes its scaling operation. If Reason has a value of UNREACHABLE, CONNECTION_FAILED, or AUTH_FAILURE, use the explanation from the Description field to help you diagnose the issue.

  • The State field might have a value of REGISTERING for a brief time before changing to AVAILABLE or UNAVAILABLE.

If the following Netcat command (nc) is successful, you can access the proxy endpoint from the EC2 instance or other system where you're logged in. This command reports failure if you're not in the same VPC as the proxy and the associated database. You might be able to log directly in to the database without being in the same VPC. However, you can't log into the proxy unless you're in the same VPC. 

nc -zx MySQL_proxy_endpoint 3306

nc -zx PostgreSQL_proxy_endpoint 5432

You can use the following commands to make sure that your EC2 instance has the required properties. In particular, the VPC for the EC2 instance must be the same as the VPC for the RDS DB instance Aurora cluster that the proxy connects to. 

aws ec2 describe-instances --instance-ids your_ec2_instance_id

Examine the Secrets Manager secrets used for the proxy. 

aws secretsmanager list-secrets
aws secretsmanager get-secret-value --secret-id your_secret_id

Make sure that the SecretString field displayed by get-secret-value is encoded as a JSON string that includes the username and password fields. The following example shows the format of the SecretString field. 

{
  "ARN": "some_arn",
  "Name": "some_name",
  "VersionId": "some_version_id",
  "SecretString": '{"username":"some_username","password":"some_password"}',
  "VersionStages": [ "some_stage" ],
  "CreatedDate": some_timestamp
}

Common issues and solutions

This section describes some common issues and potential solutions when using RDS Proxy.

After running the aws rds describe-db-proxy-targets CLI command, if the TargetHealth description states Proxy does not have any registered credentials, verify the following:

  • There are credentials registered for the user to access the proxy.

  • The IAM role to access the Secrets Manager secret used by the proxy is valid.

You might encounter the following RDS events while creating or connecting to a DB proxy.

Category RDS event ID Description

failure

 RDS-EVENT-0243 RDS couldn't provision capacity for the proxy because there aren't enough IP addresses available in your subnets. To fix the issue, make sure that your subnets have the minimum number of unused IP addresses. To determine the recommended number for your instance class, see Planning for IP address capacity.

failure

 RDS-EVENT-0275

RDS throttled some connections to DB proxy name. The number of simultaneous connection requests from the client to the proxy has exceeded the limit.

You might encounter the following issues while creating a new proxy or connecting to a proxy.

Error Causes or workarounds

403: The security token included in the request is invalid

Select an existing IAM role instead of choosing to create a new one.

You might encounter the following issues while connecting to a MySQL proxy.

Error Causes or workarounds
ERROR 1040 (HY000): Connections rate limit exceeded (limit_value) The rate of connection requests from the client to the proxy has exceeded the limit.
ERROR 1040 (HY000): IAM authentication rate limit exceeded The number of simultaneous requests with IAM authentication from the client to the proxy has exceeded the limit.
ERROR 1040 (HY000): Number simultaneous connections exceeded (limit_value) The number of simultaneous connection requests from the client to the proxy exceeded the limit.

ERROR 1045 (28000): Access denied for user 'DB_USER'@'%' (using password: YES)

The Secrets Manager secret used by the proxy doesn't match the user name and password of an existing database user. Either update the credentials in the Secrets Manager secret, or make sure the database user exists and has the same password as in the secret.
ERROR 1105 (HY000): Unknown error An unknown error occurred.
ERROR 1231 (42000): Variable ''character_set_client'' can't be set to the value of value

The value set for the character_set_client parameter is not valid. For example, the value ucs2 is not valid because it can crash the MySQL server.

ERROR 3159 (HY000): This RDS Proxy requires TLS connections.

You enabled the setting Require Transport Layer Security in the proxy but your connection included the parameter ssl-mode=DISABLED in the MySQL client. Do either of the following:

  • Disable the setting Require Transport Layer Security for the proxy.

  • Connect to the database using the minimum setting of ssl-mode=REQUIRED in the MySQL client.

ERROR 2026 (HY000): SSL connection error: Internal Server Error

The TLS handshake to the proxy failed. Some possible reasons include the following:

  • SSL is required but the server doesn't support it.

  • An internal server error occurred.

  • A bad handshake occurred.

ERROR 9501 (HY000): Timed-out waiting to acquire database connection

The proxy timed-out waiting to acquire a database connection. Some possible reasons include the following:

  • The proxy is unable to establish a database connection because the maximum connections have been reached

  • The proxy is unable to establish a database connection because the database is unavailable.

You might encounter the following issues while connecting to a PostgreSQL proxy.

Error Cause Solution

ERROR 28000: IAM authentication is allowed only with SSL connections.

The user tried to connect to the database using IAM authentication with the setting sslmode=disable in the PostgreSQL client.

The user needs to connect to the database using the minimum setting of sslmode=require in the PostgreSQL client. For more information, see the PostgreSQL SSL support documentation.

ERROR 28000: This RDS proxy has no credentials for the role role_name. Check the credentials for this role and try again.

There is no Secrets Manager secret for this role.

Add a Secrets Manager secret for this role. For more information, see Setting up AWS Identity and Access Management (IAM) policies for RDS Proxy.

ERROR 28000: RDS supports only IAM, MD5, or SCRAM authentication.

The database client being used to connect to the proxy is using an authentication mechanism not currently supported by the proxy.

If you're not using IAM authentication, use the MD5 or SCRAM password authentication.

ERROR 28000: A user name is missing from the connection startup packet. Provide a user name for this connection.

The database client being used to connect to the proxy isn't sending a user name when trying to establish a connection.

Make sure to define a user name when setting up a connection to the proxy using the PostgreSQL client of your choice.

ERROR 28000: IAM is allowed only with SSL connections.

A client tried to connect using IAM authentication, but SSL wasn't enabled.

Enable SSL in the PostgreSQL client.

ERROR 28000: This RDS Proxy requires TLS connections.

The user enabled the option Require Transport Layer Security but tried to connect with sslmode=disable in the PostgreSQL client.

To fix this error, do one of the following:

  • Disable the proxy's Require Transport Layer Security option.

  • Connect to the database using the minimum setting of sslmode=allow in the PostgreSQL client.

ERROR 28P01: IAM authentication failed for user user_name. Check the IAM token for this user and try again.

This error might be due to the following reasons:

  • The client supplied the incorrect IAM user name.

  • The client supplied an incorrect IAM authorization token for the user.

  • The client is using an IAM policy that does not have the necessary permissions.

  • The client supplied an expired IAM authorization token for the user.

To fix this error, do the following:

  1. Confirm that the provided IAM user exists.

  2. Confirm that the IAM authorization token belongs to the provided IAM user.

  3. Confirm that the IAM policy has adequate permissions for RDS.

  4. Check the validity of the IAM authorization token used.

ERROR 28P01: The password that was provided for the role role_name is wrong.

The password for this role doesn't match the Secrets Manager secret.

Check the secret for this role in Secrets Manager to see if the password is the same as what's being used in your PostgreSQL client.

ERROR 28P01: The IAM authentication failed for the role role_name. Check the IAM token for this role and try again.

There is a problem with the IAM token used for IAM authentication.

Generate a new authentication token and use it in a new connection.

ERROR 0A000: Feature not supported: RDS Proxy supports only version 3.0 of the PostgreSQL messaging protocol.

The PostgreSQL client used to connect to the proxy uses a protocol older than 3.0.

Use a newer PostgreSQL client that supports the 3.0 messaging protocol. If you're using the PostgreSQL psql CLI, use a version greater than or equal to 7.4.

ERROR 0A000: Feature not supported: RDS Proxy currently doesn't support streaming replication mode.

The PostgreSQL client used to connect to the proxy is trying to use the streaming replication mode, which isn't currently supported by RDS Proxy.

Turn off the streaming replication mode in the PostgreSQL client being used to connect.

ERROR 0A000: Feature not supported: RDS Proxy currently doesn't support the option option_name.

Through the startup message, the PostgreSQL client used to connect to the proxy is requesting an option that isn't currently supported by RDS Proxy.

Turn off the option being shown as not supported from the message above in the PostgreSQL client being used to connect.

ERROR 53300: The IAM authentication failed because of too many competing requests.

The number of simultaneous requests with IAM authentication from the client to the proxy has exceeded the limit.

Reduce the rate in which connections using IAM authentication from a PostgreSQL client are established.

ERROR 53300: The maximum number of client connections to the proxy exceeded number_value.

The number of simultaneous connection requests from the client to the proxy exceeded the limit.

Reduce the number of active connections from PostgreSQL clients to this RDS proxy.

ERROR 53300: Rate of connection to proxy exceeded number_value.

The rate of connection requests from the client to the proxy has exceeded the limit.

Reduce the rate in which connections from a PostgreSQL client are established.

ERROR XX000: Unknown error.

An unknown error occurred.

Reach out to AWS Support to investigate the issue.

ERROR 08000: Timed-out waiting to acquire database connection.

The proxy timed-out waiting to acquire a database connection. Some possible reasons include the following:

  • The proxy can't establish a database connection because the maximum connections have been reached.

  • The proxy can't establish a database connection because the database is unavailable.

Possible solutions are the following:

  • Check the target of the RDS DB instance Aurora cluster status to see if it's unavailable.

  • Check if there are long-running transactions and/or queries being executed. They can use database connections from the connection pool for a long time.

ERROR XX000: Request returned an error: database_error.

The database connection established from the proxy returned an error.

The solution depends on the specific database error. One example is: Request returned an error: database "your-database-name" does not exist. This means that the specified database name doesn't exist on the database server. Or it means that the user name used as a database name (if a database name isn't specified) doesn't exist on the server.