SQL Based Data Processing in Amazon ECS
Publication date: March 8, 2021 (Diagram history)
This architecture shows how to build a configuration-driven, codeless extract-transform-load (ETL) alternative using a containerized ETL framework (ARC
SQL Based Data Processing in Amazon ECS
The following steps describe the architecture:
-
A user creates an ETL data pipeline based on the ARC framework and SQL scripts in an interactive ARC Jupyter Notebook. The notebook runs on Amazon ECS with AWS Fargate.
-
The notebook and ETL jobs process batch and stream data through AWS PrivateLink. Traffic between ETL processes and data stores does not leave the Amazon network.
-
The ARC Jupyter notebook produces a job flow configuration JSON file. The user uploads the file and SQL scripts to Amazon Simple Storage Service through a CI/CD automated deployment process or manually.
-
An Amazon S3 file arrival event triggers an AWS Lambda function.
-
The Lambda function spins up an Amazon ECS task to process batch data in a transient way, or to process stream data continuously in a long-running container. Each job uses isolated compute resources.
-
Amazon CloudWatch Events schedules and orchestrates regular ARC ETL jobs and Amazon ECS tasks with AWS Fargate or Amazon EC2 launch types.
-
ARC ETL jobs generate application logs for each data processing stage at a granular level. CloudWatch provides monitoring and alerting capabilities.
Further reading
For additional information, refer to the following resources:
Diagram history
To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.
| Change | Description | Date |
|---|---|---|
Initial publication | Reference architecture diagram first published. | March 8, 2021 |
Note
To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.
