# Troubleshooting Amazon MQ This section describes common issues you might encounter when using Amazon MQ brokers, and the steps you can take to resolve them. For general troubleshooting, see [Troubleshooting: General Amazon MQ](general.md). For troubleshooting your specific engine version, see the following sections. ## Troubleshooting ActiveMQ on Amazon MQ | Troubleshooting topic | Description | | --- | --- | | [General troubleshooting](troubleshooting-activemq.md) | Use the information in this section to help you diagnose and resolve common issues you might encounter when working with ActiveMQ on Amazon MQ brokers. | | [BROKER\$1ENI\$1DELETED](troubleshooting-action-required-codes-broker-eni-deleted.md) | ActiveMQ on Amazon MQ will raise a BROKER\$1ENI\$1DELETED alarm when you delete a broker’s Elastic Network Interface (ENI). | | [BROKER\$1OOM](troubleshooting-action-required-codes-broker-out-of-memory.md) | ActiveMQ on Amazon MQ will raise a BROKER\$1OOM alarm when the broker undergoes a restart loop due to the insufficient memory capacity | ## Troubleshooting RabbitMQ on Amazon MQ | Troubleshooting topic | Description | | --- | --- | | [General troubleshooting](troubleshooting-rabbitmq.md) | Diagnose common issues you might encounter when working with RabbitMQ brokers. | | [ RABBITMQ\$1MEMORY\$1ALARM](troubleshooting-action-required-codes-rabbitmq-memory-alarm.md) | RabbitMQ will raise a high memory alarm when the broker's memory usage, identified by CloudWatch metric RabbitMQMemUsed, exceeds the memory limit, identified by RabbitMQMemLimit. | | [RABBITMQ\$1INVALID\$1KMS\$1KEY](troubleshooting-action-required-codes-invalid-kms-key.md) | RabbitMQ on Amazon MQ will raise an INVALID\$1KMS\$1KEY critical action required code when a broker created with a customer managed AWS KMS key(CMK) detects that the AWS Key Management Service (KMS) key is disabled. | | [RABBITMQ\$1INVALID\$1ASSUMEROLE](troubleshooting-action-required-codes-invalid-assumerole.md) | RabbitMQ on Amazon MQ will raise an INVALID\$1ASSUMEROLE critical action required code when the IAM role ARN specified in aws.arns.assume\$1role\$1arn cannot be assumed by Amazon MQ. | | [RABBITMQ\$1INVALID\$1ARN\$1LDAP](troubleshooting-action-required-codes-invalid-arn-ldap.md) | RabbitMQ on Amazon MQ will raise an INVALID\$1ARN\$1LDAP critical action required code when the LDAP service account password ARN is invalid or inaccessible. | | [RABBITMQ\$1INVALID\$1ARN\$1HTTP](troubleshooting-action-required-codes-invalid-arn-http.md) | RabbitMQ on Amazon MQ will raise an INVALID\$1ARN\$1HTTP critical action required code when one or more ARNs of SSL certificates or key file for HTTP auth\$1backend are invalid or inaccessible. | | [RABBITMQ\$1INVALID\$1ARN\$1SSL](troubleshooting-action-required-codes-invalid-arn-ssl.md) | RabbitMQ on Amazon MQ will raise an INVALID\$1ARN\$1SSL critical action required code when one or more ARNs of CA certificate truststore for EXTERNAL auth\$1mechanism are invalid or inaccessible. | | [RABBITMQ\$1INVALID\$1ARN](troubleshooting-action-required-codes-invalid-arn.md) | RabbitMQ on Amazon MQ will raise an INVALID\$1ARN critical action required code when one or more ARNs in the broker configuration are invalid or inaccessible. | | [RABBITMQ\$1DISK\$1ALARM](troubleshooting-action-required-codes-disk-limit-alarm.md) | Disk limit alarm is an indication that the volume of disk used by a RabbitMQ node has decreased due to a high number of messages not consumed while new messages were added. | # Troubleshooting: General Amazon MQ Use the information in this section to help you diagnose common issues you might encounter when working with Amazon MQ brokers, such as issues connecting to your broker, and broker reboots. **Contents** + [I can't connect to my broker web console or endpoints.](#issues-connecting-to-console-or-endpoint) + [My broker is running, and I can verify connectivity using `telnet`, but my clients are unable to connect and are returning SSL exceptions.](#issues-ssl-certificate-exception) + [I created a broker but broker creation failed.](#issues-creating-a-broker) + [My broker restarted and I'm not sure why.](#w2aac40b9c13) ## I can't connect to my broker web console or endpoints. If you're experiencing issues connecting to your broker using the web console or wire-level endpoints, we recommend the following steps. 1. Check whether you're attempting to connect to your broker from behind a firewall. You might need to configure the firewall to allow access to your broker. 1. Check whether you're trying to connect to your broker using a [FIPS](https://aws.amazon.com/compliance/fips/) endpoint. Amazon MQ only supports FIPS endpoints when using API operations, and not for wire-level connections to the broker instance itself. 1. Check if the **Public Accessibility** option for your broker is set to **Yes**. If this is set to **No**, check your subnet's network [Access Control List (ACL)](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html) rules. If you've created custom network ACLs, you might need to change the network ACL rules to provide access to your broker. For more information on Amazon VPC networking, see [Enabling internet access](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html#vpc-igw-internet-access) in the *Amazon VPC User Guide* 1. Check your broker's Security Group rules. Make sure that you are allowing connections to the following ports: **Note** The following ports are grouped according to engine types because ActiveMQ on Amazon MQ and RabbitMQ on Amazon MQ use different ports for connections. **ActiveMQ on Amazon MQ** + Web console – Port `8162` + OpenWire – Port `61617` + AMQP – Port `5671` + STOMP – Port `61614` + MQTT – Port `8883` + WSS – Port `61619` **RabbitMQ on Amazon MQ** + Web console and management API – Port ` 443` and `15671` + AMQP – Port `5671` 1. Run the following network connectivity tests for your broker engine type. **Note** For brokers without public accessibility, run the tests from an Amazon EC2 instance within the same Amazon VPC as your Amazon MQ broker and evaluate the responses. ------ #### [ ActiveMQ on Amazon MQ ] **To test your ActiveMQ on Amazon MQ broker's network connectivity** 1. Open a new terminal or command line window. 1. Run the following `nslookup` command to query your broker DNS record. For [active/standby](amazon-mq-broker-architecture.md#active-standby-broker-deployment) deployments, test both the active and standby endpoints. The active/standby endpoints are identified with a suffix, `-1` or `-2` added to the unique broker ID. Replace the endpoint with your information. ``` $ nslookup b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com ``` If the query succeeds, you will see an output similar to the following. ``` Non-authoritative answer: Server: dns-resolver-corp-sfo-1.sfo.corp.amazon.com Address: 172.10.123.456 Name: ec2-12-345-123-45.us-west-2.compute.amazonaws.com Address: 12.345.123.45 Aliases: b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com ``` The resolved IP address should match the IP addresses provided in the Amazon MQ console. This indicates that the domain name is resolving correctly on the DNS server, and you can move on to the next step. 1. Run the following `telnet` command to test the network path for your broker. Replace the endpoint with your information. Replace *port* with port number `8162` for the web console, or other wire-level ports to test additional protocols as needed. **Note** For active/standby deployments, you will receive a `Connect failed` error message if you run `telnet` with the standby endpoint. This is expected, as the standby instance itself is running, but the ActiveMQ process is not running and does not have access to the broker's Amazon EFS storage volume. Run the command for both `-1` and `-2` endpoints to ensure you test both the active and the standby instances. ``` $ telnet b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com port ``` For the active instance, you will see an output similar to the following. ``` Connected to b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com. Escape character is '^]'. ``` 1. Do one of the following. + If the `telnet` command succeeds, check the [`EstablishedConnectionsCount`](activemq-logging-monitoring.md#security-logging-monitoring-cloudwatch-metrics) metric and confirm that the broker has not reached the maximum [Wire-level connection limit](amazon-mq-limits.md). You can also confirm if the limit has been reached by reviewing the broker `General` logs. If this metric is greater than zero, then there is at least one client currently connected to the broker. If the metric shows zero connections, then perform the `telnet` path test again and wait at least one minute before disconnecting, as broker metrics are published every minute. + If the `telnet` command fails, check the status of your broker's [elastic network interface](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html), and confirm that the status is `in-use`. [Create an Amazon VPC flow log](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-flow-logs.html#create-flow-log) for each instance's network interface, and review the generated flow logs. Look for the broker's IP addresses when you ran the `telnet` command, and confirm the connection packets are `ACCEPTED`, including a return packet. For more information, and to see a flow log example, see [Flow log record examples](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html) in the *Amazon VPC Developer Guide*. 1. Run the following `curl` command to check connectivity to the ActiveMQ admin web console. ``` $ curl https://b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com:8162/index.html ``` If the command succeeds, the output should be an HTML document similar to the following. ``` Apache ActiveMQ ... ``` ------ #### [ RabbitMQ on Amazon MQ ] **To test your RabbitMQ on Amazon MQ broker's network connectivity** 1. Open a new terminal or command line window. 1. Run the following `nslookup` command to query your broker DNS record. Replace the endpoint with your information. ``` $ nslookup b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com ``` If the query succeeds, you will see an output similar to the following. ``` Non-authoritative answer: Server: dns-resolver-corp-sfo-1.sfo.corp.amazon.com Address: 172.10.123.456 Name: rabbit-broker-1c23e456ca78-b9000123b4ebbab5.elb.us-west-2.amazonaws.com Addresses: 52.12.345.678 52.23.234.56 41.234.567.890 54.123.45.678 Aliases: b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com ``` 1. Run the following `telnet` command to test the network path for your broker. Replace the endpoint with your information. You can replace *port* with port `443` for the web console, and `5671` to test the wire-level AMQP connection. ``` $ telnet b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com port ``` If the command succeeds, you'll see an output similar to the following. ``` Connected to b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com. Escape character is '^]'. ``` **Note** The telnet connection will close automatically after a few seconds. 1. Do one of the following. + If the `telnet` command succeeds, check the [`ConnectionCount`](rabbitmq-logging-monitoring.md#security-logging-monitoring-cloudwatch-metrics-rabbitmq) metric and confirm that the broker has not reached the value set in the [`max-connections`](rabbitmq-resource-limits-configuration.md) default policy. You can also confirm if the limit has been reached by reviewing the broker `Connection.log` log group. If this metric is greater than zero, there is at least one client currently connected to the broker. If the metric shows zero connections, then perform the `telnet` path test again. You may need to repeat this process if the connection closes before your broker has published new connection metrics to CloudWatch. Metrics are published every minute. + For brokers without public accessibility, if the `telnet` command fails, check the status of your broker's [elastic network interfaces](https://docs.aws.amazon.com/UserGuide/using-eni.html?icmpid=docs_ec2_console), and confirm that the status is `in-use`. [Create an Amazon VPC flow log](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-flow-logs.html#create-flow-log) for each network interface, and review the generated flow logs. Look for the broker's private IP addresses when you the `telnet` command was invoked, and confirm the connection packets are `ACCEPTED`, including a return packet. For more information, and to see a flow log example, see [Flow log record examples](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html) in the *Amazon VPC Developer Guide*. **Note** This step does not apply to RabbitMQ on Amazon MQ brokers with public accessibility. 1. Run the following `curl` command to check connectivity to the RabbitMQ admin web console. ``` $ curl https://b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9-1.mq.us-west-2.amazonaws.com:443/index.html ``` If the command succeeds, the output should be an HTML document similar to the following. ``` RabbitMQ Management ... ``` ------ ## My broker is running, and I can verify connectivity using `telnet`, but my clients are unable to connect and are returning SSL exceptions. Your broker endpoint certificate may have been updated during the broker [maintenance window](maintaining-brokers.md). Amazon MQ broker certificates are rotated periodically to ensure continued availability and security of brokers. We recommend using the Amazon root certificate authority (CA) in [Amazon Trust Services](https://www.amazontrust.com/repository/) to authenticate against in your clients' trust store. All Amazon MQ broker certificates are signed with this root CA. By using an Amazon root CA, you will no longer need to download the new Amazon MQ broker certificate every time there is a certificate update on the broker. ## I created a broker but broker creation failed. If your broker is in a `CREATION_FAILED` status, do the following. + Check your IAM permissions. To create a broker must either use the AWS managed IAM policy `AmazonMQFullAccess` or have the correct set of Amazon EC2 permissions in your custom IAM policy. To learn more about the required Amazon EC2 permissions you need, see [ IAM permissions required to create an Amazon MQ broker](security-api-authentication-authorization.md#security-permissions-required-to-create-broker). + Check if the subnet you are choosing for your broker is in a shared Amazon Virtual Private Cloud (VPC). To create an Amazon MQ broker in a shared Amazon VPC, you must create it in the account that owns the Amazon VPC. ## My broker restarted and I'm not sure why. If your broker has restarted automatically, it may be due to one of the following reasons. + Your broker may have restarted because of a scheduled weekly maintenance window. Periodically, Amazon MQ performs maintenance to the hardware, operating system, or the engine software of a message broker. The duration of the maintenance varies, but can last up to two hours, depending on the operations that are scheduled for your message broker. Brokers might restart at any point during the two hour maintenance window. For more information on broker maintenance windows, see [Scheduling the maintenance window for an Amazon MQ broker](maintaining-brokers.md). + Your broker instance type might not be suitable to your application workload. For example, running a production workload on a `mq.t3.micro` might result in the broker running out of resources. High CPU utilization, or high broker memory usage can cause a broker to unexpectedly restart. To see how much CPU and memory is being utilized by your broker, use the following CloudWatch metrics for your engine type. + **ActiveMQ on Amazon MQ** – Check `CpuUtilization` for the percentage of allocated Amazon EC2 compute units that the broker currently uses. Check `HeapUsage`for the percentage of the ActiveMQ JVM memory limit that the broker currently uses. + **RabbitMQ on Amazon MQ** – Check `SystemCpuUtilization` for the percentage of allocated Amazon EC2 compute units that the broker currently uses. Check `RabbitMQMemUsed` for the volume of RAM used in Bytes, and divide by `RabbitMQMemLimit` for the percentage of memory used by the RabbitMQ node. For more information on broker instance types and how to choose the right instance type for your workload, see [Broker instance types](broker-instance-types.md). # Troubleshooting ActiveMQ on Amazon MQ Use the information in this section to help you diagnose and resolve common issues you might encounter when working with ActiveMQ on Amazon MQ brokers. **Contents** + [I can't see general or audit logs for my broker in CloudWatch Logs even though I’ve activated logging.](#issues-cw-logging-activemq) + [After broker restart or maintenance window, I can't connect to my broker even though the status is `RUNNING`. Why?](#issues-connection-after-restart) + [I see some of my clients connecting to the broker, while others are unable to connect.](#issues-connection-limit) + [I'm seeing exception `org.apache.jasper.JasperException: An exception occurred processing JSP page` on the ActiveMQ console when performing operations.](#issues-jsp-exception) ## I can't see general or audit logs for my broker in CloudWatch Logs even though I’ve activated logging. If you’re unable to view logs for your broker in CloudWatch Logs, do the following. 1. Check if the user who creates or reboots the broker has the `logs:CreateLogGroup` permission. If you don't add the `CreateLogGroup` permission to a user before the user creates or reboots the broker, Amazon MQ will not create the log group. 1. Check if you have configured a resource-based policy to allow Amazon MQ to publish logs to CloudWatch Logs. To allow Amazon MQ to publish logs to your CloudWatch Logs log group, configure a resource-based policy to give Amazon MQ access to the following CloudWatch Logs API actions: + [https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateLogStream.html](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateLogStream.html) – Creates a CloudWatch Logs log stream for the specified log group. + [https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html) – Delivers events to the specified CloudWatch Logs log stream. For more information on configuring ActiveMQ on Amazon MQ to publish logs to CloudWatch Logs, see [Configuring logging](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/configure-logging-monitoring-activemq.html). ## After broker restart or maintenance window, I can't connect to my broker even though the status is `RUNNING`. Why? You might be encountering connection issues after a broker restart you initiated, after a scheduled maintenance window is completed, or in a failure event, where the standby instance is activated. In either case, connection issues following a broker restart are most likely caused by unusually large numbers of messages persisted in your broker's Amazon EFS or Amazon EBS storage volume. During a restart, Amazon MQ moves persisted messages from storage to broker memory. To confirm this diagnosis, you can monitor the following metrics on CloudWatch for your Amazon MQ for ActiveMQ broker: + **`StoragePercentUsage`** — Large percentages at or close to 100 percent can cause the broker to refuse connections. + **`JournalFilesForFullRecovery`** — Indicates the number of of journal files that will be replayed following an unclean shutdown and restart. An increasing, or consistently higher than one, value indicates unresolved transactions that can cause connection issues after restart. + **`OpenTransactionCount`** — A number larger than zero following a restart indicates that the broker will attempt to store previously consumed messages, as a result causing connection issues. To resolve this issue, we recommend resolving your XA transactions with either a `rollback()` or a `commit()`. For more information, and to see a code example of resolving XA transactions using `rollback()`, see [recovering XA transactions](best-practices-activemq.md#recover-xa-transactions). ## I see some of my clients connecting to the broker, while others are unable to connect. If your broker is in the `RUNNING` status and some clients are able to connect to the broker successfully, while others are unable to do so, you may have reached the [wire-level connections](amazon-mq-limits.md#broker-limits) limit for the broker. To verify that you've reached the wire-level connections limit, do the following: + Check the *general* broker logs for your ActiveMQ on Amazon MQ broker in CloudWatch Logs. If the limit has been reached, you will see `Reached Maximum Connections` in the broker logs. For more information on CloudWatch Logs for ActiveMQ on Amazon MQ brokers, see [Understanding the structure of logging in CloudWatch Logs](configure-logging-monitoring-activemq.md#security-logging-monitoring-configure-cloudwatch-structure). Once the wire-level connections limit is reached, the broker will actively refuse additional incoming connections. To resolve this issue, we recommend upgrading your broker instance type. For more information on choosing the best instance type for your workload, see [Broker instance types](broker-instance-types.md). If you've confirmed that the number of your wire-level connections is less than the broker connection limit, the issue might be related to rebooting clients. Check your broker logs for numerous and frequent entries of `... Inactive for longer than 600000 ms - removing ...`. The log entry is indicative of rebooting clients or connectivity issues. This effect is more evident when clients connect to the broker via a Network Load Balancer (NLB) with clients that frequently disconnect and reconnect to the broker. This is more typically observed in container based clients. Check your client-side logs for further details. The broker will clean up inactive TCP connections after 600000 ms, and free up the connection socket. ## I'm seeing exception `org.apache.jasper.JasperException: An exception occurred processing JSP page` on the ActiveMQ console when performing operations. If you are using simple authentication and configuring `AuthorizationPlugin` for queue and topic authorization, make sure to use the `AuthorizationEntries` element in your XML configuration file, and allow the `activemq-webconsole` group permission to all queues and topics. This ensures that the ActiveMQ web console can communicate with the ActiveMQ broker. The following example `AuthorizationEntry` grants read and write permissions for all queues and topics to the `activemq-webconsole` group. ``` ``` Similarly when integrating your broker with LDAP, make sure to grant permission for the `amazonmq-console-admins` group. For more information on LDAP integration, see [How LDAP integration works](security-authentication-authorization.md#ldap-support-details). # Troubleshooting: RabbitMQ on Amazon MQ Use the information in this section to help you diagnose and resolve common issues you might encounter when working with RabbitMQ on Amazon MQ brokers. **Contents** + [I can't see metrics for my queues or virtual hosts in CloudWatch.](#issues-cw-metrics-rabbitmq) + [How do I enable plugins in RabbitMQ on Amazon MQ?](#issues-enabling-plugins-rabbitmq) + [I'm unable to change Amazon VPC configuration for the broker.](#issues-changing-vpc-configration-rabbitmq) + [Cluster deployments have paused my queue synchronizations.](#addressing-paused-queue-sync) + [My Amazon MQ for RabbitMQ single-instance broker is in a restart loop.](#single-instance-broker-restart-loop) + [I've lost access to all administrator accounts on my broker.](#rabbitmq-broker-recovery) ## I can't see metrics for my queues or virtual hosts in CloudWatch. If you're unable to view metrics for your queues or virtual hosts in CloudWatch, check if your queue or virtual host names contain any blank spaces, tabs, or other non-ASCII characters. Amazon MQ cannot publish metrics for virtual hosts and queues with names containing blank spaces, tabs or other non-ASCII characters. For more information on dimension names, see [Dimension](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_Dimension.html#API_Dimension_Contents) in the *Amazon CloudWatch API Reference*. ## How do I enable plugins in RabbitMQ on Amazon MQ? RabbitMQ on Amazon MQ currently only supports the RabbitMQ management, shovel, federation, consistent-hash exchange plugin, which are enabled by default. For more information on using supported plugins, see [Plugins](rabbitmq-basic-elements-plugins.md). ## I'm unable to change Amazon VPC configuration for the broker. Amazon MQ does not support changing Amazon VPC configuration after your broker is created. Please note that you will need to create a new broker with the new Amazon VPC configuration and update the client connection URL with the new broker connection URL. ## Cluster deployments have paused my queue synchronizations. While addressing RabbitMQ's high memory alarms, you may find that messages on one or multiple queues cannot be consumed. These queues may be in the process of synchronizing messages between nodes, during which the respective queues become unavailable for publishing and consuming. Queue synchronizations might become paused due to the high memory alarm, and even contribute to the memory alarm. For information about stopping and retrying paused queue syncs, see [Resolving RabbitMQ paused queue synchronization](rabbitmq-queue-sync.md). ## My Amazon MQ for RabbitMQ single-instance broker is in a restart loop. An Amazon MQ for RabbitMQ single-instance broker that raises a high memory alarm is at risk of becoming unavailable if it restarts and doesn't have enough memory to start up. This can cause RabbitMQ to enter a restart loop and prevent any further interactions with the broker until the issue is resolved. If your broker is in a restart loop, you won't be able to apply the Amazon MQ recommended [best practices](troubleshooting-action-required-codes-rabbitmq-memory-alarm.md) to resolve the high memory alarm. To recover your broker, we recommend upgrading to a larger instance type with more memory. Unlike in cluster deployments, you can upgrade a single-instance broker while it's experiencing a high memory alarm because there are no queue synchronizations to perform between nodes during a restart. ## I've lost access to all administrator accounts on my broker. You can recover access using IAM authentication. Enable outbound web identity federation for your AWS account, create an IAM role with permissions to obtain web identity tokens, configure your broker to accept IAM authentication via OAuth 2.0, then use IAM credentials to obtain a JWT token and create a new administrator user. For detailed instructions, see [Using IAM authentication and authorization for Amazon MQ for RabbitMQ](rabbitmq-iam-tutorial.md). # ActiveMQ on Amazon MQ: Deleted Elastic Network Interface alarm ActiveMQ on Amazon MQ will raise a BROKER\$1ENI\$1DELETED alarm when you delete a broker’s Elastic Network Interface (ENI). When you first [create an Amazon MQ broker](getting-started-activemq.md), Amazon MQ provisions an [elastic network interface](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_ElasticNetworkInterfaces.html) in the [Virtual Private Cloud (VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Introduction.html) under your account and, thus, requires a number of [EC2 permissions](security-api-authentication-authorization.md). You must not modify or delete this network interface. Modifying or deleting the network interface can cause a permanent loss of connection between your VPC and your broker. If you wish to delete the network interface, you must delete the broker first. # ActiveMQ on Amazon MQ: Broker Out Of Memory alarm ActiveMQ on Amazon MQ will raise a BROKER\$1OOM alarm when the broker undergoes a restart loop due to the insufficient memory capacity. When a broker is in a restart loop, also called a bounce loop, the broker initiates repeated recovery attempts within a short time window. Brokers that cannot complete start-up due to insufficient memory capacity can enter a restart loop, during which interactions with the broker are limited. Amazon MQ enables metrics for your broker by default. You can view your broker metrics by accessing the Amazon CloudWatch console, or by using the CloudWatch API. The following metrics are useful when diagnosing the ActiveMQ BROKER\$1OOM alarm: | Amazon MQ CloudWatch metric | Reason for high memory use | | --- | --- | | TotalMessageCount | Messages are stored in memory until they are consumed or discarded. A high message count might indicate overutilization of resources and can lead to a high memory alarm. | | HeapUsage | The percentage of the ActiveMQ JVM memory limit that the broker currently uses. A higher percentage indicates the broker is using significant resources and may lead to an OOM alarm. | | ConnectionCount | Client connections utilize memory, and too many simultaneous connections can lead to a high memory alarm. | | CpuUtilization | The percentage of allocated EC2 compute units that the broker currently uses. | | TotalConsumerCount | For every consumer connected to the broker, a set number of messages are loaded from storage into memory before they are delivered to the consumer. A large number of consumer connections might cause high memory usage and lead to a high memory alarm. | To prevent restart loops and avoid the BROKER\$1OOM alarm, ensure that messages are consumed quickly. You can do this by choosing the most effective broker instance type, and also cleaning your [Dead Letter Queue](https://activemq.apache.org/message-redelivery-and-dlq-handling.html) to discard undeliverable or expired messages. You can learn more about ensuring effective performance at [ActiveMQ on Amazon MQ best practices](best-practices-activemq.md). # Amazon MQ for RabbitMQ: High memory alarm Amazon MQ for RabbitMQ will raise a high memory alarm when the broker's memory usage, identified by CloudWatch metric `RabbitMQMemUsed`, exceeds the memory limit, identified by `RabbitMQMemLimit`. A RabbitMQ broker that has raised a high memory alarm will block all clients that are publishing messages. Your broker may enter a [restart loop](troubleshooting-rabbitmq.md#single-instance-broker-restart-loop), experience [paused queue synchronization](troubleshooting-rabbitmq.md#addressing-paused-queue-sync), or develop other issues that complicate diagnosis and resolution of the alarm. To diagnose and resolve high memory alarm, first follow all [best practices](best-practices-rabbitmq.md) for RabbitMQ, then complete the following steps. **Important** `RabbitMQMemLimit` is set by Amazon MQ and is specifically tuned considering the memory available for each host instance type. Amazon MQ will not restart a broker experiencing a high memory alarm and will return an exception for [https://docs.aws.amazon.com/amazon-mq/latest/api-reference/brokers-broker-id-reboot.html](https://docs.aws.amazon.com/amazon-mq/latest/api-reference/brokers-broker-id-reboot.html) API operations as long as the broker continues to raise the alarm. ## Step 1: Diagnose high memory alarm There are two ways to diagnose high memory alarms on your Amazon MQ for RabbitMQ broker. We recommend that you check both the RabbitMQ web console and Amazon MQ metrics in CloudWatch. ### Diagnose high memory alarm using the RabbitMQ web console The RabbitMQ web console can generate and display detailed memory usage information for each node. You can find this information by doing the following: 1. Sign in to AWS Management Console and open your broker's RabbitMQ web console. 1. On the RabbitMQ console, on the **Overview** page, choose the name of a node from the **Nodes** list. 1. On the node detail page, choose **Memory details** to expand the section to view the node's memory usage information. The memory usage information that RabbitMQ provides in the web console can help you determine which resources might be consuming too much memory and contributing to the high memory alarm. For more information on the memory usage details available via the RabbitMQ web console, see [Reasoning About Memory Use ](https://www.rabbitmq.com/memory-use.html) on the RabbitMQ Server Documentation website. ### Diagnose high memory alarm using Amazon MQ metrics Amazon MQ enables metrics for your broker by default. You can [view your broker metrics](amazon-mq-accessing-metrics.md) by accessing the CloudWatch console, or by using the CloudWatch API. The following metrics are useful when diagnosing the RabbitMQ high memory alarm. | Amazon MQ CloudWatch metric | Reason for high memory use | | --- | --- | | MessageCount | Messages are stored in memory until they are consumed or discarded. A high message count might indicate overutilization of resources and can lead to a high memory alarm. | | QueueCount | Queues are stored in memory, and a high number of queues can lead to a high memory alarm. | | ConnectionCount | Client connections utilize memory, and too many simultaneous connections can lead to a high memory alarm. | | ChannelCount | Similar to connections, channels established using each connection are also stored in node memory, and a high number of channels can lead to a high memory alarm. | | ConsumerCount | For every consumer connected to the broker, a set number of messages are loaded from storage into memory before they are delivered to the consumer. A large number of consumer connections might cause high memory usage and lead to a high memory alarm. | | PublishRate | Publishing messages utilizes the broker' memory. If the rate at which messages are published to the broker is too high and significantly outpaces the rate at which the broker delivers messages to consumers, the broker might raise a high memory alarm. | ## Step 2: Address and prevent high memory alarm **Note** It may take up to several hours for the RABBITMQ\$1MEMORY\$1ALARM status to clear after you take the required actions. Follow all [best practices](best-practices-rabbitmq.md) for RabbitMQ as a general method of prevention. For each specific contributor that you identify, we recommend the following set of actions to address and prevent RabbitMQ high memory alarms. | Source of high memory use | Amazon MQ recommendation for addressing | Amazon MQ recommendation for preventing | | --- | --- | --- | | Number of messages | Consume messages published to the queues, purge messages from queues, or delete the queues from your broker. | Enable lazy queues, and set or reduce the [queue depth limit](rabbitmq-defaults-applying-policies.md). | | Number of queues | Reduce the number of queues. | Set or reduce the [queue count limit](rabbitmq-resource-limits-configuration.md). | | Number of connections | [Reduce the number of connections](reducing-connections-and-channels.md). | Set or reduce the [connection count limit](rabbitmq-resource-limits-configuration.md). | | Number of channels | [Reduce the number of channels](reducing-connections-and-channels.md). | Set a maximum number of channels per connection on client applications. | | Number of consumers | Reduce the number of consumers connected to the broker. | Set a small consumer [pre-fetch limit](rabbitmq-resource-limits-configuration.md). | | Message publishing rate | Reduce the rate at which publishers send messages to the broker. | Turn on [publisher confirms](best-practices-message-reliability.md#configure-confirmation-acknowledgement). | | Client connection attempt rate | Reduce the frequency at which clients attempt to connect to the broker in order to publish or consume messages, or configure the broker. | Use longer-lived connections to reduce the number and frequency of connection attempts. | After your broker's memory alarm is resolved, you can upgrade your host instance type to an instance with additional resources. For information on how to update your broker's instance type, see [https://docs.aws.amazon.com/amazon-mq/latest/api-reference/brokers-broker-id.html#brokers-broker-id-model-updatebrokerinput](https://docs.aws.amazon.com/amazon-mq/latest/api-reference/brokers-broker-id.html#brokers-broker-id-model-updatebrokerinput) in the *Amazon MQ REST API Reference*. **Note** You can't downgrade a broker from an `mq.m5.x` instance type to an `mq.t3.micro` instance type. To downgrade, you must delete your broker and create a new one. # RabbitMQ on Amazon MQ: Invalid AWS Key Management Service Key RabbitMQ on Amazon MQ will raise an INVALID\$1KMS\$1KEY critical action required code when a broker created with a customer managed AWS KMS key(CMK) detects that the AWS Key Management Service (KMS) key is disabled. A RabbitMQ broker with a CMK periodically verifies that the KMS key is enabled and the broker has all necessary grants. If RabbitMQ cannot verify that the key is enabled, the broker is quarantined and RabbitMQ will return INVALID\$1KMS\$1KEY. Without an active KMS key, the broker does not have basic permissions for customer managed KMS keys. The broker cannot perform cryptographic operations using your key until you re-enable your key and the broker restarts. A RabbitMQ broker with a disabled KMS key is quarantined to prevent deterioration. After RabbitMQ determines the KMS key is active again, your broker is removed from quarantine. Amazon MQ does not restart a broker with a disabled KMS key and returns an exception for `RebootBroker` API operations as long as the broker continues to have an invalid KMS key. ## Diagnosing and addressing INVALID\$1KMS\$1KEY To diagnose and address the INVALID\$1KMS\$1KEY action required code, you must use the AWS Command Line Interface (CLI) and the AWS Key Management Service console. **To re-enable your KMS key** 1. Call the `DescribeBroker` method to retrieve the `kmsKeyId` for your CMK broker. 1. Sign in to the AWS Key Management Service console. 1. On the **Customer managed keys** page, locate the KMS Key ID of the problematic broker and verify the status is **Enabled**. 1. If your KMS key has been disabled, re-enable the key by choosing **Key Actions**, then choose **Enable**. After your key has been re-enabled, you must wait for RabbitMQ to remove the broker from quarantine. To verify that the necessary grants are still associated with the broker's KMS key, call the `ListGrant`ListGrant method to verify that `mq_rabbit_grant` and `mq_grant` are present. If the KMS grant or key has been deleted, you must delete the broker and create a new one with all necessary grants. For steps on deleting a broker, see [Deleting a broker](https://docs.aws.amazon.com//amazon-mq/latest/developer-guide/amazon-mq-deleting-broker.html). To prevent the INVALID\$1KMS\$1KEY critical action required code, do not manually delete or disable a KMS key or CMK grant. If you wish to delete the key, delete the broker first. # RabbitMQ on Amazon MQ: Disk limit alarm Disk limit alarm is an indication that the volume of disk used by a RabbitMQ node has decreased due to a high number of messages not consumed while new messages were added. RabbitMQ will raise a disk limit alarm when the broker's free disk space, identified by Amazon CloudWatch metric `RabbitMQDiskFree`, reaches the disk limit, identified by `RabbitMQDiskFreeLimit`. `RabbitMQDiskFreeLimit` is set by Amazon MQ and has been defined considering the disk space available for each broker instance type. An RabbitMQ on Amazon MQ broker that has raised a disk limit alarm will become unavailable for new messages being published. If you have a publisher and consumer on the same connection, the consumer will also be unavailable to receive messages. When running RabbitMQ in a cluster, the disk alarm is cluster-wide. If one node goes under the limit, all other nodes will block incoming messages. Due to the lack of disk space, your broker might also experience other issues that complicate diagnosis and resolution of the alarm. Amazon MQ will not restart a broker experiencing a disk alarm and will return an exception for `RebootBroker` API operations as long as the broker continues to raise the alarm. **Note** You cannot downgrade a broker from an `mq.m5` instance type to an `mq.t3.micro` instance type. If you wish to downgrade, you must delete your broker and create a new one. ## Diagnosing and addressing disk limit alarm Amazon MQ enables metrics for your broker by default. You can [view your broker metrics](amazon-mq-accessing-metrics.md) by accessing the Amazon CloudWatch console, or by using the CloudWatch API.`MessageCount` is a useful metric when diagnosing the RabbitMQ disk limit alarm. Messages are stored in memory until they are consumed or discarded. A high message count indicates overutilization of disk storage and can lead to a disk alarm. To diagnose the disk limit alarm, use the Amazon MQ Management Console to: + Create a new connection to consume messages published to the queues. + Purge messages from queues. + Delete the queues from your broker. **Note** It may take up to several hours for the RABBITMQ\$1DISK\$1ALARM status to clear after you take the required actions. To prevent the disk limit alarm from reoccurring, you can upgrade your host [ instance type](rmq-broker-instance-types.md) to an instance with additional resources. For information on how to update your broker's instance type see `UpdateBrokerInput` in the Amazon MQ REST API Reference. We also recommend keeping your publishers and consumers on different connections. # Amazon MQ for RabbitMQ: Instance type change alarm `RABBITMQ_CLUSTER_DISK_USAGE_TOO_HIGH_FOR_INSTANCE_CHANGE` is an indication that a requested broker instance type change cannot proceed due to high disk usage on the current RabbitMQ node. Amazon MQ for RabbitMQ will raise this alarm when when the current disk usage exceeds what would be available on the requested instance type, as identified by the CloudWatch metric `RabbitMQDiskFree`. RabbitMQ brokers that enter `RABBITMQ_CLUSTER_DISK_USAGE_TOO_HIGH_FOR_INSTANCE_CHANGE` state will continue to be available for your applications, but the requested instance type change will not proceed. Amazon MQ allows broker restarts in this state, but you cannot change the instance type while the disk usage remains above the threshold for the requested instance type. The broker will return an exception for `ModifyBroker` API operations that attempt to change the instance type while in this state. ## Diagnosing and addressing instance type change alarm Amazon MQ enables metrics for your broker by default. You can view your broker metrics by accessing the CloudWatch console or by using the CloudWatch API. `MessageCount` and `RabbitMQDiskFree` metrics can be used to diagnose `RABBITMQ_CLUSTER_DISK_USAGE_TOO_HIGH_FOR_INSTANCE_CHANGE`. To resolve the quarantine state and allow your instance type change to proceed, use the Amazon MQ Management Console to: + Create a new connection to consume messages published to the queues. + Purge messages from queues. + Delete the queues from your broker. **Note** It may take up to several hours for the `RABBITMQ_CLUSTER_DISK_USAGE_TOO_HIGH_FOR_INSTANCE_CHANGE` status to clear after you take the required actions. # RabbitMQ on Amazon MQ: Invalid IAM Assume Role RabbitMQ on Amazon MQ will raise an INVALID\$1ASSUMEROLE critical action required code when the IAM role ARN specified in `aws.arns.assume_role_arn` is invalid or cannot be assumed by Amazon MQ. This can occur when the role does not exist, is in a different AWS account than the broker, or lacks the necessary trust relationship with mq.amazonaws.com. A broker in RABBITMQ\$1INVALID\$1ASSUMEROLE quarantine cannot retrieve credentials or certificates required for LDAP authentication, rendering LDAP authentication unavailable. If LDAP is the only configured authentication method, users will be unable to connect to the broker. The IAM role is required by Amazon MQ to access AWS resources referenced by ARNs in the broker configuration, such as AWS Secrets Manager secrets or Amazon S3 objects used for LDAP authentication. ## Diagnosing and addressing RABBITMQ\$1INVALID\$1ASSUMEROLE To diagnose and address the RABBITMQ\$1INVALID\$1ASSUMEROLE action required code, you must use Amazon CloudWatch Logs and the AWS Identity and Access Management console. **To resolve the invalid assume role issue** 1. Navigate to Amazon CloudWatch Logs Insights and run the following query against your broker's log group `/aws/amazonmq/broker//general`: ``` fields @timestamp, @message | sort @timestamp desc | filter @message like /error.*aws_arn_config/ | limit 10000 ``` 1. Look for error messages similar to: ``` [error] <0.254.0> aws_arn_config: {handle_assume_role,{error,{assume_role_failed,"AWS service is unavailable"}}} ``` 1. Check the IAM role configuration and fix any issues such as: + Ensure the role exists in the same AWS account as the broker + Verify the trust policy allows mq.amazonaws.com to assume the role + Confirm the role has appropriate permissions to access the required AWS resources 1. Validate the fix using the [ARN access validation](arn-support-rabbitmq-configuration.md#arn-support-validation) API endpoint before updating the broker configuration. 1. Update the broker configuration and reboot the broker. # RabbitMQ on Amazon MQ: Invalid LDAP ARN RabbitMQ on Amazon MQ will raise an INVALID\$1ARN\$1LDAP critical action required code when the ARN configured for the LDAP service account password is invalid or inaccessible. This applies to ARNs specified in `aws.arns.auth_ldap.dn_lookup_bind.password` or `aws.arns.auth_ldap.other_bind.password`, which must reference AWS Secrets Manager secrets containing plaintext passwords. A broker in RABBITMQ\$1INVALID\$1ARN\$1LDAP quarantine cannot authenticate with the LDAP service account, making LDAP authentication unavailable. If LDAP is the only configured authentication method, users will be unable to connect to the broker. Invalid ARNs can be caused by malformed ARN syntax, references to non-existent secrets, secrets located in a different AWS region than the broker, or insufficient secretsmanager:GetSecretValue permissions in the IAM role. ## Diagnosing and addressing RABBITMQ\$1INVALID\$1ARN\$1LDAP To diagnose and address the RABBITMQ\$1INVALID\$1ARN\$1LDAP action required code, you must use Amazon CloudWatch Logs and the console. **To resolve the invalid LDAP ARN issue** 1. Navigate to Amazon CloudWatch Logs Insights and run the following query against your broker's log group `/aws/amazonmq/broker//general`: ``` fields @timestamp, @message | sort @timestamp desc | filter @message like /error.*aws_arn_config/ | limit 10000 ``` 1. Look for error messages similar to: ``` [error] <0.254.0> aws_arn_config: {<<"could not resolve ARN 'arn:aws:secretsmanager:xxx' for configuration 'aws.arns.auth_ldap.dn_lookup_bind.password', error: \"AWS service is unavailable\"">>,{error,"AWS service is unavailable"}} ``` 1. Check the Secrets Manager secret and fix any issues such as: + Verify the secret exists in the same AWS region as the broker + Confirm the ARN syntax is correct + Ensure the IAM role has secretsmanager:GetSecretValue permissions 1. Validate the fix using the [ARN access validation](arn-support-rabbitmq-configuration.md#arn-support-validation) API endpoint before updating the broker configuration. 1. Update the broker configuration and reboot the broker. # RabbitMQ on Amazon MQ: Invalid HTTP ARN RabbitMQ on Amazon MQ will raise an INVALID\$1ARN\$1HTTP critical action required code when one or more ARNs of SSL certificates or key file for HTTP auth\$1backend are invalid or inaccessible. This applies to ARNS specified in `aws.arns.auth_http.ssl_options.cacertfile`, `aws.arns.auth_http.ssl_options.certfile` or `aws.arns.auth_http.ssl_options.keyfile`, which must reference Amazon S3 objects and AWS Secrets Manager secrets containing certificates and private key. A broker in RABBITMQ\$1INVALID\$1ARN\$1HTTP quarantine cannot authenticate via the HTTP server. If HTTP is the only configured authentication method, users will be unable to connect to the broker. Invalid ARNs can be caused by malformed ARN syntax, references to non-existent secrets, secrets located in a different AWS region than the broker, or insufficient s3:GetObject/secretsmanager:GetSecretValue permissions in the IAM role. ## Diagnosing and addressing RABBITMQ\$1INVALID\$1ARN\$1HTTP To diagnose and address the RABBITMQ\$1INVALID\$1ARN\$1HTTP action required code, you must use Amazon CloudWatch Logs and the console. **To resolve the invalid HTTP ARN issue** 1. Navigate to Amazon CloudWatch Logs Insights and run the following query against your broker's log group `/aws/amazonmq/broker//general`: ``` fields @timestamp, @message | sort @timestamp desc | filter @message like /error.*aws_arn_config/ | limit 10000 ``` 1. Look for error messages similar to: ``` [error] <0.209.0> aws_arn_config: {<<"could not resolve ARN 'arn:aws:s3:::xxxx' for configuration 'aws.arns.auth_http.ssl_options.certfile', error: \"AWS service is unavailable\"">>,{error,"AWS service is unavailable"}} ``` 1. Check the S3 Object/Secrets Manager secret and fix any issues such as: + Verify the resource exists in the same AWS region as the broker + Confirm the ARN syntax is correct + Ensure the IAM role has s3:GetObject and secretsmanager:GetSecretValue permissions 1. Validate the fix using the [ARN access validation](arn-support-rabbitmq-configuration.md#arn-support-validation) API endpoint before updating the broker configuration. 1. Update the broker configuration and reboot the broker. # RabbitMQ on Amazon MQ: Invalid SSL ARN RabbitMQ on Amazon MQ will raise an INVALID\$1ARN\$1SSL critical action required code when one or more ARNs of CA certificate truststore for EXTERNAL auth\$1mechanism are invalid or inaccessible. This applies to ARNS specified in `aws.arns.ssl_options.cacertfile` or `aws.arns.management.ssl.cacertfile`, which must reference Amazon S3 or ACM PCA object containing the certificate. A broker in RABBITMQ\$1INVALID\$1ARN\$1SSL quarantine cannot authenticate client certificates during mutual TLS handshakes because no valid truststore is configured. If EXTERNAL auth mechanism is the only configured authentication method, users will be unable to connect to the broker. Invalid ARNs can be caused by malformed ARN syntax, references to non-existent S3 objects, S3 objects located in a different AWS region than the broker, or insufficient s3:GetObject/acm-pca:GetCertificateAuthorityCertificate permissions in the IAM role. ## Diagnosing and addressing RABBITMQ\$1INVALID\$1ARN\$1SSL To diagnose and address the RABBITMQ\$1INVALID\$1ARN\$1SSL action required code, you must use Amazon CloudWatch Logs and the console. **To resolve the invalid SSL ARN issue** 1. Navigate to Amazon CloudWatch Logs Insights and run the following query against your broker's log group `/aws/amazonmq/broker//general`: ``` fields @timestamp, @message | sort @timestamp desc | filter @message like /error.*aws_arn_config/ | limit 10000 ``` 1. Look for error messages similar to: ``` [error] <0.209.0> aws_arn_config: {<<"could not resolve ARN 'arn:aws:acm-pca:xxxx' for configuration 'aws.arns.ssl_options.cacertfile', error: \"AWS service is unavailable\"">>,{error,"AWS service is unavailable"}} ``` 1. Check the S3/ACM-PCA Object and fix any issues such as: + Verify the secret exists in the same AWS region as the broker + Confirm the ARN syntax is correct + Ensure the IAM role has s3:GetObject/acm-pca:GetCertificateAuthorityCertificate permissions 1. Validate the fix using the [ARN access validation](arn-support-rabbitmq-configuration.md#arn-support-validation) API endpoint before updating the broker configuration. 1. Update the broker configuration and reboot the broker. # RabbitMQ on Amazon MQ: Invalid ARN RabbitMQ on Amazon MQ will raise an INVALID\$1ARN critical action required code when one or more ARNs configured in the broker are invalid or inaccessible. This applies to ARNs used for SSL certificates, AWS Secrets Manager secrets, Amazon S3 objects, or other AWS resource references not covered by more specific quarantine codes such as RABBITMQ\$1INVALID\$1ARN\$1LDAP or RABBITMQ\$1INVALID\$1ASSUMEROLE. A broker in RABBITMQ\$1INVALID\$1ARN quarantine may experience degraded functionality depending on which ARNs are invalid. Features that depend on the inaccessible resources will be unavailable, and the broker will log errors indicating which ARN failed to resolve. The impact on broker availability depends on whether the invalid ARN is required for critical broker operations. ## Diagnosing and addressing RABBITMQ\$1INVALID\$1ARN To diagnose and address the RABBITMQ\$1INVALID\$1ARN action required code, you must use Amazon CloudWatch Logs and the appropriate AWS service console for the affected resource. **To resolve the invalid ARN issue** 1. Navigate to Amazon CloudWatch Logs Insights and run the following query against your broker's log group `/aws/amazonmq/broker//general`: ``` fields @timestamp, @message | sort @timestamp desc | filter @message like /error.*aws_arn_config/ | limit 10000 ``` 1. Look for error messages similar to: ``` [error] <0.254.0> aws_arn_config: {<<"could not resolve ARN 'arn:aws:s3:::bucket-name/certificate.pem' for configuration 'aws.arns.auth_ldap.ssl_options.cacertfile', error: \"AWS service is unavailable\"">>,{error,"AWS service is unavailable"}} ``` 1. Check the AWS resource and fix any issues such as: + Verify the resource exists in the same AWS region as the broker + Confirm the ARN syntax is correct + Ensure the IAM role has appropriate permissions to access the resource 1. Validate the fix using the [ARN access validation](arn-support-rabbitmq-configuration.md#arn-support-validation) API endpoint before updating the broker configuration. 1. Update the broker configuration and reboot the broker.