Solution for monitoring Kafka applications with Amazon Managed Grafana

Your Amazon Managed Service for Prometheus workspace to store Kafka and Java Virtual Machine (JVM) metrics from your Amazon EKS cluster.

Gathering specific Kafka and JVM metrics using the CloudWatch agent, as well as a CloudWatch agent add-on. The metrics are configured to be sent to the Amazon Managed Service for Prometheus workspace.

Your Amazon Managed Grafana workspace to pull those metrics, and create dashboards to help you monitor your cluster.

Amazon Managed Grafana workspace access by users. For more information about pricing, see Amazon Managed Grafana pricing.

Amazon Managed Service for Prometheus metric ingestion and storage, and metric analysis (query sample processing). The number of metrics used by this solution depends on your application configuration and usage.

You can view the ingestion and storage metrics in Amazon Managed Service for Prometheus using CloudWatch For more information, see CloudWatch metrics in the Amazon Managed Service for Prometheus User Guide.

You can estimate the cost using the pricing calculator on the Amazon Managed Service for Prometheus pricing page. The number of metrics will depend on the number of nodes in your cluster, and the metrics your applications produce.

Networking costs. You may incur standard AWS network charges for cross availability zone, Region, or other traffic.

jvm.classes.loaded

jvm.gc.collections.count

jvm.gc.collections.elapsed

jvm.memory.heap.init

jvm.memory.heap.max

jvm.memory.heap.used

jvm.memory.heap.committed

jvm.memory.nonheap.init

jvm.memory.nonheap.max

jvm.memory.nonheap.used

jvm.memory.nonheap.committed

jvm.memory.pool.init

jvm.memory.pool.max

jvm.memory.pool.used

jvm.memory.pool.committed

jvm.threads.count

kafka.message.count

kafka.request.count

kafka.request.failed

kafka.request.time.total

kafka.request.time.50p

kafka.request.time.99p

kafka.request.time.avg

kafka.network.io

kafka.purgatory.size

kafka.partition.count

kafka.partition.offline

kafka.partition.under_replicated

kafka.isr.operation.count

kafka.max.lag

kafka.controller.active.count

kafka.leader.election.rate

kafka.unclean.election.rate

kafka.request.queue

kafka.logs.flush.time.count

kafka.logs.flush.time.median

kafka.logs.flush.time.99p

kafka.consumer.fetch-rate

kafka.consumer.records-lag-max

kafka.consumer.total.bytes-consumed-rate

kafka.consumer.total.fetch-size-avg

kafka.consumer.total.records-consumed-rate

kafka.consumer.bytes-consumed-rate

kafka.consumer.fetch-size-avg

kafka.consumer.records-consumed-rate

kafka.producer.io-wait-time-ns-avg

kafka.producer.outgoing-byte-rate

kafka.producer.request-latency-avg

kafka.producer.request-rate

kafka.producer.response-rate

kafka.producer.byte-rate

kafka.producer.compression-rate

kafka.producer.record-error-rate

kafka.producer.record-retry-rate

kafka.producer.record-send-rate

You must complete all Prerequisites before installing the solution.

The cluster must have at least one node in it before attempting to create the solution or access the metrics.

Your Amazon EKS cluster must have the AWS CNI, CoreDNS and kube-proxy add-ons installed. If they are not installed, the solution will not work correctly. They are installed by default, when creating the cluster through the console. You may need to install them if the cluster was created through an AWS SDK.

Amazon EKS pods installation timed out. This can happen if there is not enough node capacity available. There are multiple causes of these issues, including:

You do not see any dashboards in Grafana: using the incorrect Grafana workspace ID.

Run the following command to get information about Grafana:

kubectl describe grafanas external-grafana -n grafana-operator

You can check the results for the correct workspace URL. If it is not the one you are expecting, re-deploy with the correct workspace ID.

Spec:
  External:
    API Key:
      Key:   GF_SECURITY_ADMIN_APIKEY
      Name:  grafana-admin-credentials
    URL:     https://g-123example.grafana-workspace.aws-region.amazonaws.com
Status:
  Admin URL:  https://g-123example.grafana-workspace.aws-region.amazonaws.com
  Dashboards:
    ...

You do not see any dashboards in Grafana: You are using an expired API key.

To look for this case, you will need to get the grafana operator and check the logs for errors. Get the name of the Grafana operator with this command:

kubectl get pods -n grafana-operator

This will return the operator name, for example:

NAME                               READY   STATUS    RESTARTS   AGE
grafana-operator-1234abcd5678ef90   1/1     Running   0          1h2m

Use the operator name in the following command:

kubectl logs grafana-operator-1234abcd5678ef90 -n grafana-operator

Error messages such as the following indicate an expired API key:

ERROR   error reconciling datasource    {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "datasource": "grafanadatasource-sample-amp", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDatasourceReconciler).Reconcile

In this case, create a new API key and deploy the solution again. If the problem persists, you can force synchronization by using the following command before redeploying:

kubectl delete externalsecret/external-secrets-sm -n grafana-operator

Missing SSM parameter. If you see an error like the following, run cdk bootstrap and try again.

Deployment failed: Error: aws-observability-solution-kafka-eks-$EKS_CLUSTER_NAME: SSM 
parameter /cdk-bootstrap/xxxxxxx/version not found. Has the environment been 
bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/
guide/bootstrapping.html)