Set up a Prometheus host on an Amazon EC2 instance

Download the Prometheus server from https://prometheus.io/download/#prometheus to your Amazon EC2 instance.

Extract the downloaded file to a directory and go to that directory.

Create a file with the following contents and name it prometheus.yml.

# file: prometheus.yml
# my global config
global:
  scrape_interval:     60s

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    # 9090 is the prometheus server port
    - targets: ['localhost:9090']
  - job_name: 'broker'
    file_sd_configs:
    - files:
      - 'targets.json'

Use the ListNodes operation to get a list of your cluster's brokers.

Create a file named targets.json with the following JSON. Replace broker_dns_1, broker_dns_2, and the rest of the broker DNS names with the DNS names you obtained for your brokers in the previous step. Include all of the brokers you obtained in the previous step. Amazon MSK uses port 11001 for the JMX Exporter and port 11002 for the Node Exporter.

[
  {
    "labels": {
      "job": "jmx"
    },
    "targets": [
      "broker_dns_1:11001",
      "broker_dns_2:11001",
      .
      .
      .
      "broker_dns_N:11001"
    ]
  },
  {
    "labels": {
      "job": "node"
    },
    "targets": [
      "broker_dns_1:11002",
      "broker_dns_2:11002",
      .
      .
      .
      "broker_dns_N:11002"
    ]
  }
]

To start the Prometheus server on your Amazon EC2 instance, run the following command in the directory where you extracted the Prometheus files and saved prometheus.yml and targets.json.

./prometheus

Find the IPv4 public IP address of the Amazon EC2 instance where you ran Prometheus in the previous step. You need this public IP address in the following step.

To access the Prometheus web UI, open a browser that can access your Amazon EC2 instance, and go to Prometheus-Instance-Public-IP:9090, where Prometheus-Instance-Public-IP is the public IP address you got in the previous step.