Automate RabbitMQ configuration in Amazon MQ - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsRelated resourcesAttachments

Automate RabbitMQ configuration in Amazon MQ

Created by Yogesh Bhatia (AWS) and Afroz Khan (AWS)

Environment: PoC or pilot

Technologies: Messaging & communications; DevOps; Infrastructure

AWS services: Amazon MQ; AWS CloudFormation

Summary

Amazon MQ is a managed message broker service that provides compatibility with many popular message brokers. Using Amazon MQ with RabbitMQ provides a robust RabbitMQ cluster managed in the Amazon Web Services (AWS) Cloud with multiple brokers and configuration options. Amazon MQ provides a highly available, secure, and scalable infrastructure, and can process a large number of messages per second with ease. Multiple applications can use the infrastructure with different virtual hosts, queues, and exchanges. However, managing these configuration options or creating the infrastructure manually can require time and effort. This pattern describes a way to manage configurations for RabbitMQ in one step, through a single file. You can embed the code provided with this pattern within any continuous integration (CI) tool such as Jenkins or Bamboo. 

You can use this pattern to configure any RabbitMQ cluster. All it requires is connectivity to the cluster. Although there are many other ways to manage RabbitMQ configurations, this solution creates entire application configurations in one step, so you can manage queues and other details easily.

Prerequisites and limitations

Prerequisites

  • AWS Command Line Interface (AWS CLI) installed and configured to point to your AWS account (for instructions, see AWS CLI documentation)

  • Ansible installed, so you can run playbooks to create the configuration

  • rabbitmqadmin installed (for instructions, see the RabbitMQ documentation)

  • A RabbitMQ cluster in Amazon MQ, created with healthy Amazon CloudWatch metrics

Additional requirements

  • Make sure to create the configurations for virtual hosts and users separately and not as part of JSON.

  • Make sure that the configuration JSON is part of the repository and is version-controlled.

  • The version of the rabbitmqadmin CLI must be the same as the version of the RabbitMQ server, so the best option is to download the CLI from the RabbitMQ console.

  • As part of the pipeline, make sure that JSON syntax is validated before each run.

Product versions

  • AWS CLI version 2.0

  • Ansible version 2.9.13

  • rabbitmqadmin version 3.9.13 (must be the same as the RabbitMQ server version)

Architecture

Source technology stack  

  • An RabbitMQ cluster running on an existing on-premises virtual machine (VM) or a Kubernetes cluster (on premises or in the cloud)

Target technology stack  

  • Automated RabbitMQ configurations on Amazon MQ for RabbitMQ

Target architecture 

There are many ways to configure RabbitMQ. This pattern uses the import configuration functionality, where a single JSON file contains all the configurations. This file applies all settings and can be managed by a version-control system such as Bitbucket or Git. This pattern uses Ansible to implement the configuration through the rabbitmqadmin CLI.

Automating RabbitMQ configuration in Amazon MQ

Tools

Tools

  • rabbitmqadmin is a command-line tool for the RabbitMQ HTTP-based API. It is used to manage and monitor RabbitMQ nodes and clusters.

  • Ansible is an open-source tool for automating applications and IT infrastructure.

  • AWS CLI enables you to interact with AWS services by using commands in a command-line shell. 

AWS services

  • Amazon MQ is a managed message broker service that makes it easy to set up and operate message brokers in the cloud.

  • AWS CloudFormation helps you set up your AWS infrastructure and speed up cloud provisioning with infrastructure as code.

Code

The JSON configuration file used in this pattern and a sample Ansible playbook are provided in the attachment.

Epics

TaskDescriptionSkills required

Create a RabbitMQ cluster on AWS.

If you don't already have a RabbitMQ cluster, you can use AWS CloudFormation to create the stack on AWS. Or, you can use the Cloudformation module in Ansible to create the stack. With the latter approach, you can use Ansible for both tasks: to create the RabbitMQ infrastructure and to manage configurations. 

AWS CloudFormation, Ansible
TaskDescriptionSkills required

Create a properties file.

Download the JSON configuration file (rabbitmqconfig.json) in the attachment, or export it from the RabbitMQ console.  Modify it to configure queues, exchanges, and bindings. This configuration file demonstrates the following:

-  Creates two queues: sample-queue1 and sample-queue2 

-  Creates two exchanges: sample-exchange1 and sample-exchange2

-  Implements the binding between the queues and exchanges

These configurations are performed under the root (/) virtual host, as required by rabbitmqadmin. 

JSON

Retrieve the details of the Amazon MQ for RabbitMQ infrastructure.

Retrieve the following details for the RabbitMQ infrastructure on AWS:

  • Broker name

  • RabbitMQ host

  • RabbitMQ user name (the administrator user created during cluster creation)

  • RabbitMQ password

You can use the AWS Management Console or the AWS CLI to retrieve this information. These details enable the Ansible playbook to connect to your AWS account and use the RabbitMQ cluster to run commands.

Important: The computer that runs the Ansible playbook must be able to access your AWS account, and AWS CLI must already be configured, as described in the Prerequisites section.

AWS CLI, Amazon MQ

Create the hosts_var file.

Create the hosts_var file for Ansible and make sure that all the variables are defined in the file. Consider using Ansible Vault to store the password. You can configure the hosts_var file as follows (replace the asterisks with your information):

RABBITMQ_HOST: "***********.mq.us-east-2.amazonaws.com"
RABBITMQ_VHOST: "/"
RABBITMQ_USERNAME: "admin"
RABBITMQ_PASSWORD: "*******"
Ansible

Create an Ansible playbook.

For a sample playbook, see ansible-rabbit-config.yaml in the attachment. Download and save this file. The Ansible playbook imports and manages all RabbitMQ configurations, such as queues, exchanges, and bindings, that applications require. 

Follow best practices for Ansible playbooks, such as securing passwords. Use Ansible Vault for password encryption, and retrieve the RabbitMQ password from the encrypted file.

Ansible
TaskDescriptionSkills required

Run the playbook.

Run the Ansible playbook that you created in the previous epic.

ansible-playbook ansible-rabbit-config.yaml

You can verify the new configurations on the RabbitMQ console.

RabbitMQ, Amazon MQ, Ansible

Related resources

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip