Architecture overview - Scalable Analytics Using Apache Druid on AWS
Architecture diagram

Architecture overview

This section provides a reference implementation architecture diagram for the components deployed with this solution.

Architecture diagram

Deploying this solution with the default parameters deploys the following components in your AWS account.

Scalable Analytics using Apache Druid on AWS - Architecture diagram.

Scalable Analytics using Apache Druid on AWS - Architecture diagram

Note

AWS CloudFormation resources are created from AWS Cloud Development Kit (AWS CDK) (AWS CDK) constructs.

The high-level process flow for the solution components deployed with the AWS CDK constructs is as follows. The numbers and description matches the number designated in the following architecture diagram.

The solution deploys the following components that work together to provide a production-ready Druid cluster:

  1. AWS Web Application Firewall (AWS WAF) to protect the Druid web console and Druid API endpoints against common web exploits and bots that may affect availability, compromise security, or consume excessive resources. AWS WAF is only provisioned and deployed for internet facing clusters.

  2. A security hardened Linux server (Bastion host) to manage access to the Druid servers running in a private network separate from an external network. It can also be used to access the Druid web console through SSH tunneling where a private Application Load Balancer (ALB) is deployed.

  3. An ALB serves as the single point of contact for clients. The load balancer distributes incoming application traffic across multiple query servers in multiple Availability Zones.

  4. A private subnet consisting of:

    • Druid master Auto scaling group: An Auto scaling group contains a collection of Druid master servers. A Master server manages data ingestion and availability and is responsible for starting new ingestion jobs and coordinating availability of data on the data servers. Within a Master server, functionality is split between two processes: the Coordinator and Overlord.

    • Druid data Auto scaling group: An Auto scaling group contains a collection of Druid data servers. A data server runs ingestion jobs and stores queryable data. Within a data server, functionality is split between two processes: the Historical and MiddleManager.

    • Druid query Auto scaling group: An Auto scaling group contains a collection of Druid query servers. A query server provides the endpoints that users and client applications interact with, routing queries to data servers or other query servers. Within a Query server, functionality is split between two processes: the Broker and Router.

    • ZooKeeper Auto scaling group: An Auto scaling group contains a collection of ZooKeeper servers. Apache Druid uses Apache ZooKeeper (ZK) for management of current cluster state.

  5. An Amazon Simple Storage Service (Amazon S3) bucket provides deep storage for the Apache Druid cluster. Deep storage is the location where the segments are stored.

  6. AWS Secrets Manager stores the secrets used by Apache Druid including the RDS secret, and the administrator secret. It also stores the credentials for the system account the Druid components use to authenticate with each other.

  7. Amazon CloudWatch support logs, metrics, and dashboards.

  8. An Amazon Aurora PostgreSQL database provides the metadata storage for the Apache Druid cluster. Druid uses the metadata store to house only metadata about the system only, and does not store the actual data.

  9. The notification system, powered by Amazon Simple Notification Service (Amazon SNS), delivers alerts or alarms promptly when system events occur. This ensures immediate awareness and action when needed.