# Get started with Amazon Managed Workflows for Apache Airflow Amazon Managed Workflows for Apache Airflow uses the Amazon VPC, DAG files and supporting files in your Amazon S3 storage bucket to create an environment. This chapter describes the prerequisites and AWS resources needed to get started with Amazon MWAA. **Topics** + [Prerequisites](#prerequisites) + [About this guide](#prerequisites-infra) + [Before you begin](#prerequisites-before) + [Available regions](#regions) + [Create an Amazon S3 bucket for Amazon MWAA](mwaa-s3-bucket.md) + [Create the VPC network](vpc-create.md) + [Create an Amazon MWAA environment](create-environment.md) + [What's next?](#mwaa-s3-bucket-next-up) ## Prerequisites To create an Amazon MWAA environment, ensure you have permission to the AWS resources you need to create. + **AWS account** – An AWS account with permission to use Amazon MWAA and the AWS services and resources used by your environment. ## About this guide This guide covers the AWS infrastructure and resources you'll create. + **Amazon VPC** – The Amazon VPC networking components required by an Amazon MWAA environment. You can configure an existing VPC that meets these requirements (advanced) as found in [About networking on Amazon MWAA](networking-about.md), or create the VPC and networking components, as defined in [Create the VPC network](vpc-create.md). + **Amazon S3 bucket** – An Amazon S3 bucket to store your DAGs and associated files, such as `plugins.zip` and `requirements.txt`. Your Amazon S3 bucket must be configured to **Block all public access**, with **Bucket Versioning** enabled, as defined in [Create an Amazon S3 bucket for Amazon MWAA](mwaa-s3-bucket.md). + **Amazon MWAA environment** – An Amazon MWAA environment configured with the location of your Amazon S3 bucket, the path to your DAG code and any custom plugins or Python dependencies, and your Amazon VPC and its security group, as defined in [Create an Amazon MWAA environment](create-environment.md). ## Before you begin To create an Amazon MWAA environment, you can take additional steps to create and configure other AWS resources before you create your environment. To create an environment, you need the following: + **AWS KMS key** – An AWS KMS key for data encryption on your environment. You can choose the default option on the Amazon MWAA console to create an [AWS-owned key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk) when you create an environment, or specify an existing [Customer-managed key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk) with permissions to other AWS services used by your environment configured (advanced). To learn more, refer to [Using customer-managed keys for encryption](custom-keys-certs.md). + **Execution role** – An execution role that allows Amazon MWAA to access AWS resources in your environment. You can choose the default option on the Amazon MWAA console to create an execution role when you create an environment. To learn more, refer to [Amazon MWAA execution role](mwaa-create-role.md). + **VPC security group** – A VPC security group that allows Amazon MWAA to access other AWS resources in your VPC network. You can choose the default option on the Amazon MWAA console to create a security group when you create an environment, or provide a security group with the appropriate inbound and outbound rules (advanced). To learn more, refer to [Security in your VPC on Amazon MWAA](vpc-security.md). ## Available regions Amazon MWAA is available in the following AWS Regions. To learn more about each region, such as which are enabled or disabled by default, refer to [AWS Regions](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html). | Code | Name | | --- | --- | | us-east-1 | US East (N. Virginia) | | us-east-2 | US East (Ohio) | | us-west-1 | US West (N. California) | | us-west-2 | US West (Oregon) | | af-south-1 | Africa (Cape Town) | | ap-east-1 | Asia Pacific (Hong Kong) | | ap-south-2 | Asia Pacific (Hyderabad) | | ap-southeast-3 | Asia Pacific (Jakarta) | | ap-southeast-5 | Asia Pacific (Malaysia) | | ap-southeast-4 | Asia Pacific (Melbourne) | | ap-south-1 | Asia Pacific (Mumbai) | | ap-northeast-3 | Asia Pacific (Osaka) | | ap-northeast-2 | Asia Pacific (Seoul) | | ap-southeast-1 | Asia Pacific (Singapore) | | ap-southeast-2 | Asia Pacific (Sydney) | | ap-northeast-1 | Asia Pacific (Tokyo) | | ca-central-1 | Canada (Central) | | ca-west-1 | Canada West (Calgary) | | eu-central-1 | Europe (Frankfurt) | | eu-west-1 | Europe (Ireland) | | eu-west-2 | Europe (London) | | eu-south-1 | Europe (Milan) | | eu-west-3 | Europe (Paris) | | eu-south-2 | Europe (Spain) | | eu-north-1 | Europe (Stockholm) | | eu-central-2 | Europe (Zurich) | | il-central-1 | Israel (Tel Aviv) | | me-south-1 | Middle East (Bahrain) | | me-central-1 | Middle East (UAE) | | sa-east-1 | South America (São Paulo) | # Create an Amazon S3 bucket for Amazon MWAA This guide describes the steps to create an Amazon S3 bucket to store your Apache Airflow Directed Acyclic Graphs (DAGs), custom plugins in a `plugins.zip` file, and Python dependencies in a `requirements.txt` file. **Contents** + [Before you begin](#mwaa-s3-bucket-before) + [Create the bucket](#mwaa-s3-bucket-create) + [What's next?](#mwaa-s3-bucket-next-up) ## Before you begin + The Amazon S3 bucket name can't be changed after you create the bucket. To learn more, refer to [Rules for bucket naming](https://docs.aws.amazon.com/AmazonS3/latest/userguide/BucketRestrictions.html#bucketnamingrules) in the *Amazon Simple Storage Service User Guide*. + An Amazon S3 bucket used for an Amazon MWAA environment must be configured to **Block all public access**, with **Bucket Versioning** enabled. + An Amazon S3 bucket used for an Amazon MWAA environment must be located in the same AWS Region as an Amazon MWAA environment. To access a list of AWS Regions for Amazon MWAA, refer to [Amazon MWAA endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/mwaa.html) in the *AWS General Reference*. ## Create the bucket This section describes the steps to create the Amazon S3 bucket for your environment. **To create a bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. Choose **Create bucket**. 1. In **Bucket name**, enter a DNS-compliant name for your bucket. The bucket name must: + Be unique across all of Amazon S3. + Be between 3 and 63 characters long. + Not contain uppercase characters. + Start with a lowercase letter or number. **Important** Avoid including sensitive information, such as account numbers, in the bucket name. The bucket name is available in the URLs that point to the objects in the bucket. 1. Choose an AWS Region in **Region**. This must be the same AWS Region as your Amazon MWAA environment. 1. We recommend choosing a region close to you to minimize latency and costs and address regulatory requirements. 1. Choose **Block all public access**. 1. Choose **Enable** in **Bucket Versioning**. 1. **Optional** - *Tags*. Add key-value tag pairs to identify your Amazon S3 bucket in **Tags**. For example, `Bucket` : `Staging`. 1. **Optional** - *Server-side encryption*. You can optionally **Enable** one of the following encryption options on your Amazon S3 bucket. 1. Choose **Amazon S3 key (SSE-S3)** in **Server-side encryption** to enable server-side encryption for the bucket. 1. Choose **AWS Key Management Service key (SSE-KMS)** to use an AWS KMS key for encryption on your Amazon S3 bucket: 1. **AWS managed key (aws/s3)** - If you choose this option, you can either use an [AWS-owned key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk) managed by Amazon MWAA, or specify a [Customer-managed key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk) for encryption of your Amazon MWAA environment. 1. **Choose from your AWS KMS keys** or **Enter AWS KMS key ARN** - If you choose to specify a [Customer-managed key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk) in this step, you must specify an AWS KMS key ID or ARN. [AWS KMS aliases and multi-region keys are not supported by Amazon MWAA](custom-keys-certs.md). The AWS KMS key you specify must also be used for encryption on your Amazon MWAA environment. 1. **Optional** - *Advanced settings*. If you want to enable Amazon S3 Object Lock: 1. Choose **Advanced settings**, **Enable**. **Important** Enabling Object Lock will permanently allow objects in this bucket to be locked. To learn more, refer to [Locking Objects Using Amazon S3 Object Lock](https://docs.aws.amazon.com//AmazonS3/latest/dev/object-lock.html) in the *Amazon Simple Storage Service User Guide*. 1. Choose the acknowledgement. 1. Choose **Create bucket**. ## What's next? + Learn how to create the required Amazon VPC network for an environment in [Create the VPC network](vpc-create.md). + Learn how to how to manage access permissions in [How do I set ACL bucket permissions?](https://docs.aws.amazon.com//AmazonS3/latest/user-guide/set-bucket-permissions.html) + Learn how to delete a storage bucket in [How do I delete an S3 Bucket?](https://docs.aws.amazon.com//AmazonS3/latest/user-guide/delete-bucket.html). # Create the VPC network Amazon Managed Workflows for Apache Airflow requires an Amazon VPC and specific networking components to support an environment. This guide describes the different options to create the Amazon VPC network for an Amazon Managed Workflows for Apache Airflow environment. **Note** Apache Airflow works best in a low-latency network environment. If you are using an existing Amazon VPC which routes traffic to another region or to an on-premise environment, we recommended adding AWS PrivateLink endpoints for Amazon SQS, CloudWatch, Amazon S3, and AWS KMS. For more information about configuring AWS PrivateLink for Amazon MWAA, refer to [Creating an Amazon VPC network without internet access](#vpc-create-template-private-only). **Contents** + [Prerequisites](#vpc-create-prereqs) + [Before you begin](#vpc-create-how-networking) + [Options to create the Amazon VPC network](#vpc-create-options) + [Option one: Creating the VPC network on the Amazon MWAA console](#vpc-create-mwaa-console) + [Option two: Creating an Amazon VPC network *with* internet access](#vpc-create-template-private-or-public) + [Option three: Creating an Amazon VPC network *without* internet access](#vpc-create-template-private-only) + [What's next?](#create-vpc-next-up) ## Prerequisites The AWS Command Line Interface (AWS CLI) is an open source tool that you can use to interact with AWS services using commands in your command-line shell. To complete the steps on this page, you need the following: + [AWS CLI – Install version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). + [AWS CLI – Quick configuration with `aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). ## Before you begin + The [VPC network](#vpc-create) you specify for your environment can't be changed after the environment is created. + You can use private or public routing for your Amazon VPC and Apache Airflow webserver. To access a list of options, refer to [Example use cases for an Amazon VPC and Apache Airflow access mode](networking-about.md#networking-about-network-usecase). ## Options to create the Amazon VPC network The following section describes the options available to create the Amazon VPC network for an environment. **Note** Amazon MWAA does not support the use of `use1-az3` Availability Zone (AZ) in the US East (N. Virginia) Region. When creating the VPC for Amazon MWAA in the US East (N. Virginia) region, you must explicitly assign the `AvailabilityZone` in the CloudFormation (CFN) template. The assigned availability zone name must not be mapped to `use1-az3`. You can retrieve the detailed mapping of AZ names to their corresponding AZ IDs by running the following command: ``` aws ec2 describe-availability-zones --region us-east-1 ``` ### Option one: Creating the VPC network on the Amazon MWAA console The following section explains how to create an Amazon VPC network on the Amazon MWAA console. This option uses [Public routing over the internet](networking-about.md#networking-about-overview-public). It can be used for an Apache Airflow webserver with the **Private network** or **Public network** access modes. The following image depicts where you can find the **Create MWAA VPC** button on the Amazon MWAA console. ![\[This image depicts where you can find the Create MWAA VPC on the Amazon MWAA console.\]](http://docs.aws.amazon.com/mwaa/latest/userguide/images/mwaa-console-create-vpc.png) ### Option two: Creating an Amazon VPC network *with* internet access The following CloudFormation template creates an Amazon VPC network with internet access in your default AWS Region. This option uses [Public routing over the internet](networking-about.md#networking-about-overview-public). This template can be used for an Apache Airflow webserver with the **Private network** or **Public network** access modes. 1. Copy the contents of the following template and save locally as `cfn-vpc-public-private.yaml`. You can also [download the template](./samples/cfn-vpc-public-private.zip). ``` Description: This template deploys a VPC, with a pair of public and private subnets spread across two Availability Zones. It deploys an internet gateway, with a default route on the public subnets. It deploys a pair of NAT gateways (one in each AZ), and default routes for them in the private subnets. Parameters: EnvironmentName: Description: An environment name that is prefixed to resource names Type: String Default: mwaa- VpcCIDR: Description: Please enter the IP range (CIDR notation) for this VPC Type: String Default: 10.192.0.0/16 PublicSubnet1CIDR: Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone Type: String Default: 10.192.10.0/24 PublicSubnet2CIDR: Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone Type: String Default: 10.192.11.0/24 PrivateSubnet1CIDR: Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone Type: String Default: 10.192.20.0/24 PrivateSubnet2CIDR: Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone Type: String Default: 10.192.21.0/24 Resources: VPC: Type: AWS::EC2::VPC Properties: CidrBlock: !Ref VpcCIDR EnableDnsSupport: true EnableDnsHostnames: true Tags: - Key: Name Value: !Ref EnvironmentName InternetGateway: Type: AWS::EC2::InternetGateway Properties: Tags: - Key: Name Value: !Ref EnvironmentName InternetGatewayAttachment: Type: AWS::EC2::VPCGatewayAttachment Properties: InternetGatewayId: !Ref InternetGateway VpcId: !Ref VPC PublicSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 0, !GetAZs '' ] CidrBlock: !Ref PublicSubnet1CIDR MapPublicIpOnLaunch: true Tags: - Key: Name Value: !Sub ${EnvironmentName} Public Subnet (AZ1) PublicSubnet2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 1, !GetAZs '' ] CidrBlock: !Ref PublicSubnet2CIDR MapPublicIpOnLaunch: true Tags: - Key: Name Value: !Sub ${EnvironmentName} Public Subnet (AZ2) PrivateSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 0, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet1CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Subnet (AZ1) PrivateSubnet2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 1, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet2CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Subnet (AZ2) NatGateway1EIP: Type: AWS::EC2::EIP DependsOn: InternetGatewayAttachment Properties: Domain: vpc NatGateway2EIP: Type: AWS::EC2::EIP DependsOn: InternetGatewayAttachment Properties: Domain: vpc NatGateway1: Type: AWS::EC2::NatGateway Properties: AllocationId: !GetAtt NatGateway1EIP.AllocationId SubnetId: !Ref PublicSubnet1 NatGateway2: Type: AWS::EC2::NatGateway Properties: AllocationId: !GetAtt NatGateway2EIP.AllocationId SubnetId: !Ref PublicSubnet2 PublicRouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub ${EnvironmentName} Public Routes DefaultPublicRoute: Type: AWS::EC2::Route DependsOn: InternetGatewayAttachment Properties: RouteTableId: !Ref PublicRouteTable DestinationCidrBlock: 0.0.0.0/0 GatewayId: !Ref InternetGateway PublicSubnet1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref PublicSubnet1 PublicSubnet2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref PublicSubnet2 PrivateRouteTable1: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Routes (AZ1) DefaultPrivateRoute1: Type: AWS::EC2::Route Properties: RouteTableId: !Ref PrivateRouteTable1 DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NatGateway1 PrivateSubnet1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTable1 SubnetId: !Ref PrivateSubnet1 PrivateRouteTable2: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Routes (AZ2) DefaultPrivateRoute2: Type: AWS::EC2::Route Properties: RouteTableId: !Ref PrivateRouteTable2 DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NatGateway2 PrivateSubnet2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTable2 SubnetId: !Ref PrivateSubnet2 SecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupName: "mwaa-security-group" GroupDescription: "Security group with a self-referencing inbound rule." VpcId: !Ref VPC SecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref SecurityGroup IpProtocol: "-1" SourceSecurityGroupId: !Ref SecurityGroup Outputs: VPC: Description: A reference to the created VPC Value: !Ref VPC PublicSubnets: Description: A list of the public subnets Value: !Join [ ",", [ !Ref PublicSubnet1, !Ref PublicSubnet2 ]] PrivateSubnets: Description: A list of the private subnets Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]] PublicSubnet1: Description: A reference to the public subnet in the 1st Availability Zone Value: !Ref PublicSubnet1 PublicSubnet2: Description: A reference to the public subnet in the 2nd Availability Zone Value: !Ref PublicSubnet2 PrivateSubnet1: Description: A reference to the private subnet in the 1st Availability Zone Value: !Ref PrivateSubnet1 PrivateSubnet2: Description: A reference to the private subnet in the 2nd Availability Zone Value: !Ref PrivateSubnet2 SecurityGroupIngress: Description: Security group with self-referencing inbound rule Value: !Ref SecurityGroupIngress ``` 1. In your command prompt, navigate to the directory where `cfn-vpc-public-private.yaml` is stored. For example: ``` cd mwaaproject ``` 1. Use the [https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html) command to create the stack using the AWS CLI. ``` aws cloudformation create-stack --stack-name mwaa-environment --template-body file://cfn-vpc-public-private.yaml ``` **Note** It takes about 30 minutes to create the Amazon VPC infrastructure. ### Option three: Creating an Amazon VPC network *without* internet access The following CloudFormation template creates an Amazon VPC network *without internet access* in your default AWS Region. This option uses [Private routing without internet access](networking-about.md#networking-about-overview-private). This template can be used for an Apache Airflow webserver with the **Private network** access mode only. It creates the required [VPC endpoints for the AWS services used by an environment](vpc-vpe-create-access.md#vpc-vpe-create-view-endpoints-attach-services). 1. Copy the contents of the following template and save locally as `cfn-vpc-private.yaml`. You can also [download the template](./samples/cfn-vpc-private-no-ops.zip). ``` AWSTemplateFormatVersion: "2010-09-09" Parameters: VpcCIDR: Description: The IP range (CIDR notation) for this VPC Type: String Default: 10.192.0.0/16 PrivateSubnet1CIDR: Description: The IP range (CIDR notation) for the private subnet in the first Availability Zone Type: String Default: 10.192.10.0/24 PrivateSubnet2CIDR: Description: The IP range (CIDR notation) for the private subnet in the second Availability Zone Type: String Default: 10.192.11.0/24 Resources: VPC: Type: AWS::EC2::VPC Properties: CidrBlock: !Ref VpcCIDR EnableDnsSupport: true EnableDnsHostnames: true Tags: - Key: Name Value: !Ref AWS::StackName RouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub "${AWS::StackName}-route-table" PrivateSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 0, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet1CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub "${AWS::StackName} Private Subnet (AZ1)" PrivateSubnet2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 1, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet2CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub "${AWS::StackName} Private Subnet (AZ2)" PrivateSubnet1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref RouteTable SubnetId: !Ref PrivateSubnet1 PrivateSubnet2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref RouteTable SubnetId: !Ref PrivateSubnet2 S3VpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3" VpcEndpointType: Gateway VpcId: !Ref VPC RouteTableIds: - !Ref RouteTable SecurityGroup: Type: AWS::EC2::SecurityGroup Properties: VpcId: !Ref VPC GroupDescription: Security Group for Amazon MWAA Environments to access VPC endpoints GroupName: !Sub "${AWS::StackName}-mwaa-vpc-endpoints" SecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref SecurityGroup IpProtocol: "-1" SourceSecurityGroupId: !Ref SecurityGroup SqsVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.sqs" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup CloudWatchLogsVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.logs" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup CloudWatchMonitoringVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.monitoring" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup KmsVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.kms" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup Outputs: VPC: Description: A reference to the created VPC Value: !Ref VPC MwaaSecurityGroupId: Description: Associates the Security Group to the environment to allow access to the VPC endpoints Value: !Ref SecurityGroup PrivateSubnets: Description: A list of the private subnets Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]] PrivateSubnet1: Description: A reference to the private subnet in the 1st Availability Zone Value: !Ref PrivateSubnet1 PrivateSubnet2: Description: A reference to the private subnet in the 2nd Availability Zone Value: !Ref PrivateSubnet2 ``` 1. In your command prompt, navigate to the directory where `cfn-vpc-private.yml` is stored. For example: ``` cd mwaaproject ``` 1. Use the [https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html) command to create the stack using the AWS CLI. ``` aws cloudformation create-stack --stack-name mwaa-private-environment --template-body file://cfn-vpc-private.yml ``` **Note** It takes about 30 minutes to create the Amazon VPC infrastructure. 1. You'll need to create a mechanism to access these VPC endpoints from your computer. To learn more, refer to [Managing access to service-specific Amazon VPC endpoints on Amazon MWAA](vpc-vpe-access.md). **Note** You can further restrict outbound access in the CIDR of your Amazon MWAA security group. For example, you can restrict to itself by adding a self-referencing outbound rule, the [prefix list](https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gateway.html) for Amazon S3, and the CIDR of your Amazon VPC. ## What's next? + Learn how to create an Amazon MWAA environment in [Create an Amazon MWAA environment](create-environment.md). + Learn how to create a VPN tunnel from your computer to your Amazon VPC with private routing in [Tutorial: Configuring private network access using an AWS Client VPN](tutorials-private-network-vpn-client.md). # Create an Amazon MWAA environment Amazon Managed Workflows for Apache Airflow sets up Apache Airflow on an environment in your chosen version using the same open-source Apache Airflow and user interface available from Apache. This guide describes the steps to create an Amazon MWAA environment. **Contents** + [Before you begin](#create-environment-before) + [Apache Airflow versions](#create-environment-regions-aa-versions) + [Create an environment](#create-environment-start) + [Step one: Specify details](#create-environment-start-details) + [Step two: Configure advanced settings](#create-environment-start-advanced) + [Step three: Review and create](#create-environment-start-review) ## Before you begin + The [VPC network](vpc-create.md) you specify for your environment cannot be modified after the environment is created. + You need an Amazon S3 bucket configured to **Block all public access**, with **Bucket Versioning** enabled. + You need an AWS account with [permissions to use Amazon MWAA](manage-access.md), and permission in AWS Identity and Access Management (IAM) to create IAM roles. If you choose the **Private network** access mode for the Apache Airflow webserver, which limits Apache Airflow access within your Amazon VPC, you'll need permission in IAM to create Amazon VPC endpoints. **Note** Amazon MWAA dynamically determines the network during creation. If you use IPv6 subnets, Amazon MWAA creates IPv6 private link connectivity to the database and webserver. Amazon MWAA does not support transitioning between network types and cannot upgrade existing environments to IPv6. ## Apache Airflow versions The following Apache Airflow versions are supported on Amazon Managed Workflows for Apache Airflow. **Note** Effective December 30, 2025, Amazon MWAA will end support for Apache Airflow versions v2.4.3, v2.5.1, and v2.6.3. For more information, refer to [Apache Airflow version support and FAQ](airflow-versions.md#airflow-versions-support). Beginning with Apache Airflow v2.2.2, Amazon MWAA supports installing Python requirements, provider packages, and custom plugins directly on the Apache Airflow webserver. Beginning with Apache Airflow v2.7.2, your requirements file must include a `--constraint` statement. If you don't provide a constraint, Amazon MWAA will specify one for you to ensure the packages listed in your requirements are compatible with the version of Apache Airflow you're using. For more information about setting up constraints in your requirements file, refer to [Installing Python dependencies](working-dags-dependencies.md#working-dags-dependencies-syntax-create). | Apache Airflow version | Apache Airflow release date | Amazon MWAA availability date | Apache Airflow constraints | Python version | | --- | --- | --- | --- | --- | | [v2.11.0](https://airflow.apache.org/docs/apache-airflow/2.11.0) | [May 20, 2025](https://airflow.apache.org/docs/apache-airflow/2.11.0/release_notes.html#airflow-2-11-0-2022-05-20) | January 7, 2026 | [v2.11.0 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.11.0/constraints-3.12.txt) | [Python 3.12](https://peps.python.org/pep-0693/) | | [v3.0.6](https://airflow.apache.org/docs/apache-airflow/3.0.6) | [August 29, 2025](https://airflow.apache.org/docs/apache-airflow/3.0.6/release_notes.html#airflow-3-0-6-2025-08-29) | October 1, 2025 | [v3.0.6 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-3.0.6/constraints-3.12.txt) | [Python 3.12](https://peps.python.org/pep-0693/) | | [v2.10.3](https://airflow.apache.org/docs/apache-airflow/2.10.3) | [November 4, 2024](https://airflow.apache.org/docs/apache-airflow/2.10.3/release_notes.html#airflow-2-10-3-2024-11-04) | December 18, 2024 | [v2.10.3 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.10.3/constraints-3.11.txt) | [Python 3.11](https://peps.python.org/pep-0664/) | | [v2.10.1](https://airflow.apache.org/docs/apache-airflow/2.10.1) | [September 5, 2024](https://airflow.apache.org/docs/apache-airflow/2.10.1/release_notes.html#airflow-2-10-1-2024-09-05) | September 26, 2024 | [v2.10.1 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.10.1/constraints-3.11.txt) | [Python 3.11](https://peps.python.org/pep-0664/) | | [v2.9.2](https://airflow.apache.org/docs/apache-airflow/2.9.2) | [June 10, 2024](https://airflow.apache.org/docs/apache-airflow/2.10.1/release_notes.html#airflow-2-9-2-2024-06-10) | July 9, 2024 | [v2.9.2 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.9.2/constraints-3.11.txt) | [Python 3.11](https://peps.python.org/pep-0664/) | | [v2.8.1](https://airflow.apache.org/docs/apache-airflow/2.8.1) | [January 19, 2024](https://airflow.apache.org/docs/apache-airflow/2.10.1/release_notes.html#airflow-2-8-1-2024-01-19) | February 23, 2024 | [v2.8.1 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt) | [Python 3.11](https://peps.python.org/pep-0664/) | | [v2.7.2](https://airflow.apache.org/docs/apache-airflow/2.7.2) | [October 12, 2023](https://airflow.apache.org/docs/apache-airflow/2.10.1/release_notes.html#airflow-2-7-2-2023-10-12) | November 6, 2023 | [v2.7.2 constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt) | [Python 3.11](https://peps.python.org/pep-0664/) | For more information about migrating your self-managed Apache Airflow deployments, or migrating an existing Amazon MWAA environment, including instructions for backing up your metadata database, refer to the [Amazon MWAA Migration Guide](https://docs.aws.amazon.com/mwaa/latest/migrationguide/index.html). ## Create an environment The following section describes the steps to create an Amazon MWAA environment. ### Step one: Specify details **To specify details for the environment** 1. Open the [Amazon MWAA](https://console.aws.amazon.com/mwaa/home/) console. 1. Select your AWS Region. 1. Choose **Create environment**. 1. On the **Specify details** page, under **Environment details**: 1. Enter a unique name for your environment in **Name**. 1. Choose the Apache Airflow version in **Airflow version**. **Note** If no value is specified, defaults to the latest Apache Airflow version. The latest available version is Apache Airflow v3.0.6. 1. Under **DAG code in Amazon S3** specify the following: 1. **S3 Bucket**. Choose **Browse S3** and select your Amazon S3 bucket, or enter the Amazon S3 URI. 1. **DAGs folder**. Choose **Browse S3** and select the `dags` folder in your Amazon S3 bucket, or enter the Amazon S3 URI. 1. **Plugins file - *optional***. Choose **Browse S3** and select the `plugins.zip` file on your Amazon S3 bucket, or enter the Amazon S3 URI. 1. **Requirements file - *optional***. Choose **Browse S3** and select the `requirements.txt` file on your Amazon S3 bucket, or enter the Amazon S3 URI. 1. **Startup script file - *optional***, Choose **Browse S3** and select the script file on your Amazon S3 bucket, or enter the Amazon S3 URI. 1. Choose **Next**. ### Step two: Configure advanced settings **To configure advanced settings** 1. On the **Configure advanced settings** page, under **Networking**: 1. Choose your [Amazon VPC](vpc-create.md). This step populates two of the private subnets in your Amazon VPC. 1. Under **webserver access**, select your preferred [Apache Airflow access mode](configuring-networking.md): 1. **Private network**. This limits access of the Apache Airflow UI to users *within your Amazon VPC* that have been granted access to the [IAM policy for your environment](access-policies.md). You need permission to create Amazon VPC endpoints for this step. **Note** Choose the **Private network** option if your Apache Airflow UI is only accessed within a corporate network, and you do not require access to public repositories for webserver requirements installation. If you choose this access mode option, you need to create a mechanism to access your Apache Airflow webserver in your Amazon VPC. For more information, refer to [Accessing the VPC endpoint for your Apache Airflow webserver (private network access)](vpc-vpe-access.md#vpc-vpe-access-endpoints). 1. **Public network**. This allows the Apache Airflow UI to be accessed over the internet by users granted access to the [IAM policy for your environment](access-policies.md). 1. Under **Security groups**, choose the security group used to secure your [Amazon VPC](vpc-create.md): 1. By default, Amazon MWAA creates a security group in your Amazon VPC with specific inbound and outbound rules in **Create new security group**. 1. **Optional**. Deselect the check box in **Create new security group** to select up to 5 security groups. **Note** An existing Amazon VPC security group must be configured with specific inbound and outbound rules to allow network traffic. To learn more, refer to [Security in your VPC on Amazon MWAA](vpc-security.md). 1. Under **Environment class**, choose an [environment class](environment-class.md). We recommend choosing the smallest size necessary to support your workload. You can change the environment class at any time. 1. For **Maximum worker count**, specify the maximum number of Apache Airflow workers to run in the environment. For more information, refer to [Example high performance use case](mwaa-autoscaling.md#mwaa-autoscaling-high-volume). 1. Specify the **Maximum web server count** and **Minimum web server count** to configure how Amazon MWAA scales the Apache Airflow web servers in your environment. For more information about web server automatic scaling, refer to [Configuring Amazon MWAA webserver automatic scaling](mwaa-web-server-autoscaling.md). 1. Under **Encryption**, choose a data encryption option: 1. By default, Amazon MWAA uses an AWS-owned key to encrypt your data. 1. **Optional**. Choose **Customize encryption settings (advanced)** to choose a different AWS KMS key. If you choose to specify a [Customer-managed key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk) in this step, you must specify an AWS KMS key ID or ARN. [AWS KMS aliases and multi-region keys are not supported by Amazon MWAA](custom-keys-certs.md). If you specified an Amazon S3 key for server-side encryption on your Amazon S3 bucket, you must specify the same key for your Amazon MWAA environment. **Note** You must have permissions to the key to select it on the Amazon MWAA console. You must also grant permissions for Amazon MWAA to use the key by attaching the policy described in [Attach key policy](custom-keys-certs.md#custom-keys-certs-grant-policies-attach). 1. **Recommended**. Under **Monitoring**, choose one or more log categories for **Airflow logging configuration** to send Apache Airflow logs to CloudWatch Logs: 1. **Airflow task logs**. Choose the type of Apache Airflow task logs to send to CloudWatch Logs in **Log level**. 1. **Airflow webserver logs**. Choose the type of Apache Airflow webserver logs to send to CloudWatch Logs in **Log level**. 1. **Airflow scheduler logs**. Choose the type of Apache Airflow scheduler logs to send to CloudWatch Logs in **Log level**. 1. **Airflow worker logs**. Choose the type of Apache Airflow worker logs to send to CloudWatch Logs in **Log level**. 1. **Airflow DAG processing logs**. Choose the type of Apache Airflow DAG processing logs to send to CloudWatch Logs in **Log level**. 1. **Optional**. For **Airflow configuration options**, choose **Add custom configuration option**. You can choose from the suggested dropdown list of [Apache Airflow configuration options](configuring-env-variables.md) for your Apache Airflow version, or specify custom configuration options. For example, `core.default_task_retries` : `3`. 1. **Optional**. Under **Tags**, choose **Add new tag** to associate tags to your environment. For example, `Environment`: `Staging`. 1. Under **Permissions**, choose an execution role: 1. By default, Amazon MWAA creates an [execution role](mwaa-create-role.md) in **Create a new role**. You must have permission to create IAM roles to use this option. 1. **Optional**. Choose **Enter role ARN** to enter the Amazon Resource Name (ARN) of an existing execution role. 1. Choose **Next**. ### Step three: Review and create **To review an environment summary** + Review the environment summary, choose **Create environment**. **Note** It takes about twenty to thirty minutes to create an environment. ## What's next? + Learn how to create an Amazon S3 bucket in [Create an Amazon S3 bucket for Amazon MWAA](mwaa-s3-bucket.md).