Data Lake Ingest and Processing with Downstream Analytics
This architecture shows how to monitor data processing pipelines within the data lake using downstream analytics. You can publish relevant logs, metrics, and events to detect abnormal behaviors and visualize trends in near real time.
Data Lake Ingest and Processing with Downstream Analytics
The following steps describe the architecture:
-
Data processing pipelines within the data lake produce data. Any number of source pipelines can produce in this step.
-
The data pipeline components publish relevant logs, metrics, or events to Amazon CloudWatch.
-
An AWS Lambda function fetches data from CloudWatch and pushes pre-processed data to Amazon Kinesis Data Streams.
-
The Kinesis data stream is the central storage for near real-time streaming data.
-
Amazon Kinesis Data Analytics analyzes data in near real time and saves aggregated metrics to an Amazon Timestream database for further analysis. A Kinesis Data Analytics query can also detect abnormal behaviors of data in the pipeline.
-
Amazon Timestream supports time series analytics and defines time series as a native data type. It supports advanced aggregates, window functions, and complex data types such as arrays and rows.
-
The Amazon Kinesis Data Analytics threshold query sends near real-time analytics to Lambda.
-
Based on the processed data from Kinesis Data Analytics, the Lambda function can send real-time alerts using Amazon Simple Notification Service.
-
Amazon Quick Sight dashboards visualize trends such as number of successful pipelines, error rate, error type distribution, and more.
Further reading
For additional information, refer to the following resources:
Diagram history
To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.
| Change | Description | Date |
|---|---|---|
Reference architecture diagram first published. | July 15, 2021 | |
Initial publication | Reference architecture diagram first published. | July 15, 2021 |
Note
To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.
