Starting from EMR 7.0, Amazon EMR provide the Amazon CloudWatch Agent to send metrics to CloudWatch or Prometheus, replacing the Ganglia monitoring system. You can find more details in the Amazon CloudWatch Agent Configuration Guide.
The EMR 7.0 Amazon CloudWatch agent provided basic integration and setting it up required manual effort, like downloading components and using bootstrap actions. However, from EMR 7.1 onwards, the process is simplified. Using the Amazon CloudWatch Agent in EMR 7.1 makes it easy to monitor both system-level and application-specific metrics, including those for HBase. By using the EMR configuration API, you can quickly set up and customize the metrics collection process, and choose where to send the data—either to Amazon CloudWatch or Prometheus. This flexibility helps you keep a close eye on your HBase clusters, ensuring they run smoothly and efficiently.
Using the EMR configuration API to handle the entire setup, making the process much smoother. The CloudWatch Agent in EMR 7.1 supports three main types of metrics:
System Metrics – These include key indicators of system performance, such as: CPU usage, Disk usage, Memory usage, Network I/O, Processes, and Swap usage.
Hadoop Daemon Metrics – These metrics relate to the various components of Hadoop, including: DataNode metrics, NameNode metrics, YARN NodeManager metrics, and YARN ResourceManager metrics.
HBase Metrics – These metrics provide insights into HBase’s performance: HBase Master metrics, HBase Region Server metrics, HBase REST Server metrics, and HBase Thrift Server metrics.
All the metrics for Hadoop and HBase are JMX-based, which means they use Java Management Extensions to provide detailed insights. Here’s how you can set up Amazon CloudWatch Agent to monitor HBase:
Refer to Prerequisites before creating a cluster with Amazon CloudWatch Agent. Use a
create-clustercommand similar to the sample that appears after this list.Refer the Configurations supported for an HBase cluster.
Refer to the sample that follows for example configurations to set up HBase monitoring. Refer to the example configurations for the
--configurationinput.
aws emr create-cluster --name "HBase cluster with CloudWatch agent" \
--release-label emr-7.1.0 \
--applications Name=HBase Name=AmazonCloudWatchAgent \
--ec2-attributes KeyName=myKey --instance-type m7g.2xlarge \
--configurations file://./configurations.json \
--instance-count 3 --use-default-rolesFor more information on JSON sample configurations for metrics, see Set up metrics.
