Automate RabbitMQ configuration in Amazon MQ
Created by Yogesh Bhatia (AWS) and Afroz Khan (AWS)
Environment: PoC or pilot | Technologies: Messaging & communications; DevOps; Infrastructure | AWS services: Amazon MQ; AWS CloudFormation |
Summary
Amazon MQ is a managed message broker service that provides compatibility with many popular message brokers. Using Amazon MQ with RabbitMQ provides a robust RabbitMQ cluster managed in the Amazon Web Services (AWS) Cloud with multiple brokers and configuration options. Amazon MQ provides a highly available, secure, and scalable infrastructure, and can process a large number of messages per second with ease. Multiple applications can use the infrastructure with different virtual hosts, queues, and exchanges. However, managing these configuration options or creating the infrastructure manually can require time and effort. This pattern describes a way to manage configurations for RabbitMQ in one step, through a single file. You can embed the code provided with this pattern within any continuous integration (CI) tool such as Jenkins or Bamboo.
You can use this pattern to configure any RabbitMQ cluster. All it requires is connectivity to the cluster. Although there are many other ways to manage RabbitMQ configurations, this solution creates entire application configurations in one step, so you can manage queues and other details easily.
Prerequisites and limitations
Prerequisites
AWS Command Line Interface (AWS CLI) installed and configured to point to your AWS account (for instructions, see AWS CLI documentation)
Ansible installed, so you can run playbooks to create the configuration
rabbitmqadmin installed (for instructions, see the RabbitMQ documentation
) A RabbitMQ cluster in Amazon MQ, created with healthy Amazon CloudWatch metrics
Additional requirements
Make sure to create the configurations for virtual hosts and users separately and not as part of JSON.
Make sure that the configuration JSON is part of the repository and is version-controlled.
The version of the rabbitmqadmin CLI must be the same as the version of the RabbitMQ server, so the best option is to download the CLI from the RabbitMQ console.
As part of the pipeline, make sure that JSON syntax is validated before each run.
Product versions
AWS CLI version 2.0
Ansible version 2.9.13
rabbitmqadmin version 3.9.13 (must be the same as the RabbitMQ server version)
Architecture
Source technology stack
An RabbitMQ cluster running on an existing on-premises virtual machine (VM) or a Kubernetes cluster (on premises or in the cloud)
Target technology stack
Automated RabbitMQ configurations on Amazon MQ for RabbitMQ
Target architecture
There are many ways to configure RabbitMQ. This pattern uses the import configuration functionality, where a single JSON file contains all the configurations. This file applies all settings and can be managed by a version-control system such as Bitbucket or Git. This pattern uses Ansible to implement the configuration through the rabbitmqadmin CLI.
![Automating RabbitMQ configuration in Amazon MQ](/images/prescriptive-guidance/latest/patterns/images/pattern-img/294120b6-c95f-4cc5-bf85-5ad7e2abdad5/images/292e1284-5c9e-4c82-bb41-010fa84d8d74.png)
Tools
Tools
rabbitmqadmin
is a command-line tool for the RabbitMQ HTTP-based API. It is used to manage and monitor RabbitMQ nodes and clusters. Ansible
is an open-source tool for automating applications and IT infrastructure. AWS CLI enables you to interact with AWS services by using commands in a command-line shell.
AWS services
Amazon MQ is a managed message broker service that makes it easy to set up and operate message brokers in the cloud.
AWS CloudFormation helps you set up your AWS infrastructure and speed up cloud provisioning with infrastructure as code.
Code
The JSON configuration file used in this pattern and a sample Ansible playbook are provided in the attachment.
Epics
Task | Description | Skills required |
---|---|---|
Create a RabbitMQ cluster on AWS. | If you don't already have a RabbitMQ cluster, you can use AWS CloudFormation to create the stack on AWS. Or, you can use the Cloudformation module in Ansible | AWS CloudFormation, Ansible |
Task | Description | Skills required |
---|---|---|
Create a properties file. | Download the JSON configuration file ( - Creates two queues: - Creates two exchanges: - Implements the binding between the queues and exchanges These configurations are performed under the root (/) virtual host, as required by rabbitmqadmin. | JSON |
Retrieve the details of the Amazon MQ for RabbitMQ infrastructure. | Retrieve the following details for the RabbitMQ infrastructure on AWS:
You can use the AWS Management Console or the AWS CLI to retrieve this information. These details enable the Ansible playbook to connect to your AWS account and use the RabbitMQ cluster to run commands. Important: The computer that runs the Ansible playbook must be able to access your AWS account, and AWS CLI must already be configured, as described in the Prerequisites section. | AWS CLI, Amazon MQ |
Create the hosts_var file. | Create the
| Ansible |
Create an Ansible playbook. | For a sample playbook, see Follow best practices for Ansible playbooks, such as securing passwords. Use Ansible Vault for password encryption, and retrieve the RabbitMQ password from the encrypted file. | Ansible |
Task | Description | Skills required |
---|---|---|
Run the playbook. | Run the Ansible playbook that you created in the previous epic.
You can verify the new configurations on the RabbitMQ console. | RabbitMQ, Amazon MQ, Ansible |
Related resources
Migrating from RabbitMQ to Amazon MQ
(AWS blog post) Management Command Line Tool
(RabbitMQ documentation) Create or delete an AWS CloudFormation stack
(Ansible documentation) Migrating message driven applications to Amazon MQ for RabbitMQ
(AWS blog post)
Attachments
To access additional content that is associated with this document, unzip the following file: attachment.zip