The environment class you choose for your Amazon MWAA environment determines the size of the AWS-managed AWS Fargate containers where the
Celery Executor
Environment capabilities
The following section contains the default concurrent Apache Airflow tasks, Random Access Memory (RAM), and the virtual centralized processing units (vCPUs) for each environment class. The concurrent tasks listed assume that task concurrency does not exceed the Apache Airflow Worker capacity in the environment.
In the following table, DAG capacity refers to DAG definitions, not executions, and assumes that your DAGs are
dynamic
Task executions depend by how many are scheduled simultaneously, and assumes that the
number of DAG runs set to start at the same time does not exceed the default max_dagruns_per_loop_to_schedule
-
Up to 25 DAG capacity
-
3 concurrent tasks (by default)
-
Components:
-
Web server: 1 vCPU, 3GB RAM
-
Worker and scheduler: 1 vCPU, 3GB RAM
-
Database: 2 vCPU, 4GB RAM
Note
mw1.micro does not support auto-scaling.
-
You can use celery.worker_autoscale
to increase tasks per worker. For more information, see the Example high performance use case.
Apache Airflow Schedulers
The following section contains the Apache Airflow scheduler options available on the Amazon MWAA, and how the number of schedulers affects the number of triggerers.
In Apache Airflow, a triggerer
-
v2 - For environments larger than mw1.micro, accepts values from
2
to5
. Defaults to2
for all environment sizes except mw1.micro, which defaults to1
.