Upgrading the Apache Airflow version - Amazon Managed Workflows for Apache Airflow
Upgrade your workflow resourcesSpecify the new version

Upgrading the Apache Airflow version

Amazon MWAA supports minor version upgrades. This means you can upgrade your environment from version x.4.z to x.5.z. To perform a major version upgrade, for example from version 1.y.z to 2.y.z, you must create a new environment and migrate your resources. For more information on upgrading to a new major version of Apache Airflow, see Migrating to a new Amazon MWAA environment in the Amazon MWAA Migration Guide.

During the upgrade process, Amazon MWAA captures a snapshot of your environment metadata, upgrades the workers, schedulers, the web server to the new Apache Airflow version, and finally restores the metadata database using the snapshot.

Note

You cannot downgrade the Apache Airflow version for your environment.

Before you upgrade, make sure that your DAGs and other workflow resources are compatible with the new Apache Airflow version you are upgrading to. If you use a requirements.txt to manage dependencies, you must also ensure the dependencies you specify in your requirements are compatible with the new version.

Upgrade your workflow resources

Whenever you're changing Apache Airflow versions, ensure that you reference the correct --constraint URL in your requirements.txt.

Warning

Specifying requirements that are incompatible with your target Apache Airflow version during an upgrade might result in a lengthy rollback process to the previous version of Apache Airflow with the previous requirements version.

To migrate your workflow resources

  1. Create a fork of the aws-mwaa-local-runner repository, and clone a copy of the Amazon MWAA local runner.

  2. Checkout to the branch of the aws-mwaa-local-runner repository that matches the version you are upgrading to.

  3. Use the Amazon MWAA local runner CLI tool to build the Docker image and run Apache Airflow locally. For more information, see the local runner README in the GitHub repository.

  4. To update your requirements.txt, follow the best practices we recommend in Managing Python dependencies, in the Amazon MWAA User Guide.

  5. (Optional) To speed up the upgrade process, clean up the environment's metadata database. Environments with a large amount of metadata can take significantly longer to upgrade.

  6. After you have successfully tested your workflow resources, copy your DAGs, requirements.txt, and plugins to your environment's Amazon S3 bucket.

You are now ready to edit the environment, specify a new Apache Airflow version, and start the update procedure.

Specify the new version

After you have completed updating your workflow resources to ensure compatibility with the new Apache Airflow version, do the following to edit environment details and specify the version of Apache Airflow that you want to upgrade to.

Note

When you perform an upgrade, all tasks currently running on the environment are terminated during the procedure. The update procedure can take up to two hours, during which time your environment will be unavailable.

To specify a new version using the console

  1. Open the Environments page on the Amazon MWAA console.

  2. From the Environments list, choose the environment that you want to upgrade.

  3. On the environment page, choose Edit to edit the environment.

  4. In the Environment details section, for Airflow version, choose the new Apache Airflow version number that you want to upgrade the environment to from the dropdown list.

  5. Choose Next until you are on the Review and save page.

  6. On the Review and save page, review your changes, then choose Save.

When you apply changes, your environment begins the upgrade procedure. During this period, the status of your environment indicates what actions Amazon MWAA is taking, and whether the procedure is successful.

In a successful upgrade scenario, the status will show UPDATING, then CREATING_SNAPSHOT as Amazon MWAA captures a backup of your metadata. Finally, the status will return first to UPDATING, then to AVAILABLE when the procedure is done.

If the environment fails to upgrade, your environment status will show ROLLING_BACK. If the rollback is successful, the status will first show UPDATE_FAILED, indicating that the update failed but the environment is available. If the rollback fails, the status will show UNAVAILABLE, indicating that you cannot access the environment.