# Tutorials for Amazon Managed Workflows for Apache Airflow This guide includes step-by-step tutorials to using and configuring an Amazon Managed Workflows for Apache Airflow environment. **Topics** + [Tutorial: Configuring private network access using an AWS Client VPN](tutorials-private-network-vpn-client.md) + [Tutorial: Configuring private network access using a Linux Bastion Host](tutorials-private-network-bastion.md) + [Tutorial: Restricting an Amazon MWAA user's access to a subset of DAGs](limit-access-to-dags.md) + [Tutorial: Automate managing your own environment endpoints on Amazon MWAA](tutorials-customer-managed-endpoints.md) # Tutorial: Configuring private network access using an AWS Client VPN This tutorial walks you through the steps to create a VPN tunnel from your computer to the Apache Airflow webserver for your Amazon Managed Workflows for Apache Airflow environment. To connect to the internet through a VPN tunnel, you'll first need to create a AWS Client VPN endpoint. Once set up, a Client VPN endpoint acts as a VPN server allowing a secure connection from your computer to the resources in your VPC. You'll then connect to the Client VPN from your computer using the [AWS Client VPN for Desktop](https://aws.amazon.com/vpn/client-vpn-download/). **Topics** + [Private network](#private-network-vpn-onconsole) + [Use cases](#private-network-vpn-usecases) + [Before you begin](#private-network-vpn-prereqs) + [Objectives](#private-network-vpn-objectives) + [(Optional) Step one: Identify your VPC, CIDR rules, and VPC security](#private-network-vpn-optional) + [Step two: Create the server and client certificates](#private-network-vpn-certs) + [Step three: Save the CloudFormation template locally](#private-network-vpn-template) + [Step four: Create the Client VPN CloudFormation stack](#private-network-vpn-create) + [Step five: Associate subnets to your Client VPN](#private-network-vpn-associate) + [Step six: Add an authorization ingress rule to your Client VPN](#private-network-vpn-autho) + [Step seven: Download the Client VPN endpoint configuration file](#private-network-vpn-download) + [Step eight: Connect to the AWS Client VPN](#private-network-vpn-connect) + [What's next?](#create-vpc-vpn-next-up) ## Private network This tutorial assumes you've chosen the **Private network** access mode for your Apache Airflow webserver. ![\[This image displays the architecture for an Amazon MWAA environment with a private webserver.\]](http://docs.aws.amazon.com/mwaa/latest/userguide/images/mwaa-private-web-server.png) The private network access mode limits access to the Apache Airflow UI to users *within your Amazon VPC* who have been granted access to the [IAM policy for your environment](access-policies.md). When you create an environment with private webserver access, you must package all of your dependencies in a Python wheel archive (`.whl`), then reference the `.whl` in your `requirements.txt`. For instructions on packaging and installing your dependencies using wheel, refer to [Managing dependencies using Python wheel](best-practices-dependencies.md#best-practices-dependencies-python-wheels). The following image depicts where to find the **Private network** option on the Amazon MWAA console. ![\[This image depicts where to find the Private network option on the Amazon MWAA console.\]](http://docs.aws.amazon.com/mwaa/latest/userguide/images/mwaa-console-private-network.png) ## Use cases You can use this tutorial before or after you've created an Amazon MWAA environment. You must use the same Amazon VPC, VPC security groups, and private subnets as your environment. If you use this tutorial after you've created an Amazon MWAA environment, once you've completed the steps, you can return to the Amazon MWAA console and change your Apache Airflow webserver access mode to **Private network**. ## Before you begin 1. Check for user permissions. Be sure that your account in AWS Identity and Access Management (IAM) has sufficient permissions to create and manage VPC resources. 1. Use your Amazon MWAA VPC. This tutorial assumes that you are associating the Client VPN to an existing VPC. The Amazon VPC must be in the same AWS Region as an Amazon MWAA environment and have two private subnets. If you haven't created an Amazon VPC, use the CloudFormation template in [Option three: Creating an Amazon VPC network *without* internet access](vpc-create.md#vpc-create-template-private-only). ## Objectives In this tutorial, you'll do the following: 1. Create a AWS Client VPN endpoint using a CloudFormation template for an existing Amazon VPC. 1. Generate server and client certificates and keys, and then upload the server certificate and key to AWS Certificate Manager in the same AWS Region as an Amazon MWAA environment. 1. Download and modify a Client VPN endpoint configuration file for your Client VPN, and use the file to create a VPN profile to connect using the Client VPN for Desktop. ## (Optional) Step one: Identify your VPC, CIDR rules, and VPC security The following section describes how to find IDs for your Amazon VPC, VPC security group, and a way to identify the CIDR rules you'll need to create your Client VPN in subsequent steps. ### Identify your CIDR rules The following section explains how to identify the CIDR rules, which you'll need to create your Client VPN. **To identify the CIDR for your Client VPN** 1. Open the [Your Amazon VPCs page](https://console.aws.amazon.com/vpc/home#/vpcs:) on the Amazon VPC console. 1. Use the region selector in the navigation bar to choose the same AWS Region as an Amazon MWAA environment. 1. Choose your Amazon VPC. 1. Assuming the CIDRs for your private subnets are: + Private Subnet 1: 10.192.10.0`/24` + Private Subnet 2: 10.192.11.0`/24` If the CIDR for your Amazon VPC is 10.192.0.0`/16`, then the **Client IPv4 CIDR** you'd specify for your Client VPN would be 10.192.0.0`/22`. 1. Save this CIDR value, and the value of your VPC ID for subsequent steps. ### Identify your VPC and security groups The following section explains how to find the ID of your Amazon VPC and security groups, which you'll need to create your Client VPN. **Note** You might be using more than one security group. You'll need to specify all of your VPC's security groups in subsequent steps. **To identify the security groups** 1. Open the [Security Groups page](https://console.aws.amazon.com/vpc/home#/securityGroups:) on the Amazon VPC console. 1. Use the region selector in the navigation bar to choose the AWS Region. 1. Search for the Amazon VPC in **VPC ID**, and identify the security groups associated with the VPC. 1. Save the ID of your security groups and VPC for subsequent steps. ## Step two: Create the server and client certificates A Client VPN endpoint supports 1024-bit and 2048-bit RSA key sizes only. The following section explains how to use OpenVPN easy-rsa to generate the server and client certificates and keys, and then upload the certificates to ACM using the AWS Command Line Interface (AWS CLI). **To create the client certificates** 1. Follow these quick steps to create and upload the certificates to ACM through the AWS CLI in [Client authentication and authorization: Mutual authentication](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html#mutual). 1. In these steps, you **must** specify the same AWS Region as an Amazon MWAA environment in the AWS CLI command when uploading your server and client certificates. Here's some examples of how to specify the region in these commands: 1. **Example region for server certificate** ``` aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crt --region us-west-2 ``` 1. **Example region for client certificate** ``` aws acm import-certificate --certificate fileb://client1.domain.tld.crt --private-key fileb://client1.domain.tld.key --certificate-chain fileb://ca.crt --region us-west-2 ``` 1. After these steps, save the value returned in the AWS CLI response for the server certificate and client certificate ARNs. You'll be specifying these ARNs in your CloudFormation template to create the Client VPN. 1. In these steps, a client certificate and a private key are saved to your computer. Here's an example of where to find these credentials: 1. **Example on macOS** On macOS the contents are saved at `/Users/your-user/custom_folder`. If you list all (`ls -a`) contents of this directory, you get something similar to the following: ``` . .. ca.crt client1.domain.tld.crt client1.domain.tld.key server.crt server.key ``` 1. After these steps, save the contents or note the location of the client certificate in `client1.domain.tld.crt`, and the private key in `client1.domain.tld.key`. You'll be adding these values to the configuration file for your Client VPN. ## Step three: Save the CloudFormation template locally The following section contains the CloudFormation template to create the Client VPN. You must specify the same Amazon VPC, VPC security groups, and private subnets as your Amazon MWAA environment. + Copy the contents of the following template and save locally as `mwaa_vpn_client.yaml`. You can also [download the template](./samples/mwaa_vpn_client.zip). Substitute the following values: + **YOUR\$1CLIENT\$1ROOT\$1CERTIFICATE\$1ARN** – The ARN for your **client1.domain.tld** certificate in `ClientRootCertificateChainArn`. + **YOUR\$1SERVER\$1CERTIFICATE\$1ARN** – The ARN for your **server** certificate in `ServerCertificateArn`. + The Client IPv4 CIDR rule in `ClientCidrBlock`. A CIDR rule of `10.192.0.0/22` is provided. + Your Amazon VPC ID in `VpcId`. A VPC of `vpc-010101010101` is provided. + Your VPC security group IDs in `SecurityGroupIds`. A security group of `sg-0101010101` is provided. ``` AWSTemplateFormatVersion: 2010-09-09 Description: This template deploys a VPN Client Endpoint. Resources: ClientVpnEndpoint: Type: 'AWS::EC2::ClientVpnEndpoint' Properties: AuthenticationOptions: - Type: "certificate-authentication" MutualAuthentication: ClientRootCertificateChainArn: "YOUR_CLIENT_ROOT_CERTIFICATE_ARN" ClientCidrBlock: 10.192.0.0/22 ClientConnectOptions: Enabled: false ConnectionLogOptions: Enabled: false Description: "MWAA Client VPN" DnsServers: [] SecurityGroupIds: - sg-0101010101 SelfServicePortal: '' ServerCertificateArn: "YOUR_SERVER_CERTIFICATE_ARN" SplitTunnel: true TagSpecifications: - ResourceType: "client-vpn-endpoint" Tags: - Key: Name Value: MWAA-Client-VPN TransportProtocol: udp VpcId: vpc-010101010101 VpnPort: 443 ``` **Note** If you're using more than one security group for your environment, you can specify multiple security groups in the following format: ``` SecurityGroupIds: - sg-0112233445566778b - sg-0223344556677889f ``` ## Step four: Create the Client VPN CloudFormation stack **To create the AWS Client VPN** 1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/home#). 1. Choose **Template is ready**, **Upload a template file**. 1. Choose **Choose file**, and select your `mwaa_vpn_client.yaml` file. 1. Choose **Next**, **Next**. 1. Select the acknowledgement, and then choose **Create stack**. ## Step five: Associate subnets to your Client VPN **To associate private subnets to the AWS Client VPN** 1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/home#). 1. Choose the **Client VPN Endpoints** page. 1. Select your Client VPN, and then choose the **Associations** tab, **Associate**. 1. Choose the following in the dropdown list: + Your Amazon VPC in **VPC**. + One of your private subnets in **Choose a subnet to associate**. 1. Choose **Associate**. **Note** It takes several minutes for the VPC and subnet to be associated to the Client VPN. ## Step six: Add an authorization ingress rule to your Client VPN You need to add an authorization ingress rule using the CIDR rule for your VPC to your Client VPN. If you want to authorize specific users or groups from your Active Directory Group or SAML-based Identity Provider (IdP), refer to the [Authorization rules](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-rules.html) in the *Client VPN guide*. **To add the CIDR to the AWS Client VPN** 1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/home#). 1. Choose the **Client VPN Endpoints** page. 1. Select your Client VPN, and then choose the **Authorization** tab, **Authorize Ingress**. 1. Specify the following: + Your Amazon VPC's CIDR rule in **Destination network to enable**. For example: ``` 10.192.0.0/16 ``` + Choose **Allow access to all users** in **Grant access to**. + Enter a descriptive name in **Description**. 1. Choose **Add Authorization rule**. **Note** Depending on the networking components for your Amazon VPC, you might also need to this authorization ingress rule to your network access control list (NACL). ## Step seven: Download the Client VPN endpoint configuration file **To download the configuration file** 1. Follow these quick steps to download the Client VPN configuration file at [Download the Client VPN endpoint configuration file](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-getting-started.html#cvpn-getting-started-config). 1. In these steps, you're asked to prepend a string to your Client VPN endpoint DNS name. Here's an example: 1. **Example endpoint DNS name** If your Client VPN endpoint DNS name is: ``` remote cvpn-endpoint-0909091212aaee1.prod.clientvpn.us-west-1.amazonaws.com 443 ``` You can add a string to identify your Client VPN endpoint like this: ``` remote mwaavpn.cvpn-endpoint-0909091212aaee1.prod.clientvpn.us-west-1.amazonaws.com 443 ``` 1. In these steps, you're asked to add the contents of the client certificate between a new set of `` tags and the contents of the private key between a new set of `` tags. Here's an example: 1. Open a command prompt and change directories to the location of your client certificate and private key. 1. **Example macOS client1.domain.tld.crt** To show the contents of the `client1.domain.tld.crt` file on macOS, you can use `cat client1.domain.tld.crt`. Copy the value from terminal and paste in `downloaded-client-config.ovpn` like this: ``` ZZZ1111dddaBBB -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- YOUR client1.domain.tld.crt -----END CERTIFICATE----- ``` 1. **Example macOS client1.domain.tld.key** To show the contents of the `client1.domain.tld.key`, you can use `cat client1.domain.tld.key`. Copy the value from terminal and paste in `downloaded-client-config.ovpn` like this: ``` ZZZ1111dddaBBB -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- YOUR client1.domain.tld.crt -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- YOUR client1.domain.tld.key -----END CERTIFICATE----- ``` ## Step eight: Connect to the AWS Client VPN The client for AWS Client VPN is provided free of charge. You can connect your computer directly to AWS Client VPN for an end-to-end VPN experience. **To connect to the Client VPN** 1. Download and install the [AWS Client VPN for Desktop](https://aws.amazon.com/vpn/client-vpn-download/). 1. Open the AWS Client VPN. 1. Choose **File**, **Managed profiles** in the VPN client menu. 1. Choose **Add profile**, and then choose the `downloaded-client-config.ovpn`. 1. Enter a descriptive name in **Display Name**. 1. Choose **Add profile**, **Done**. 1. Choose **Connect**. After you connect to the Client VPN, you'll need to disconnect from other VPNs to access any of the resources in your Amazon VPC. **Note** You might need to quit the client, and start again before you're able to get connected. ## What's next? + Learn how to create an Amazon MWAA environment in [Get started with Amazon Managed Workflows for Apache Airflow](get-started.md). You must create an environment in the same AWS Region as the Client VPN, and using the same VPC, private subnets, and security group as the Client VPN. # Tutorial: Configuring private network access using a Linux Bastion Host This tutorial walks you through the steps to create an SSH tunnel from your computer to the to the Apache Airflow webserver for your Amazon Managed Workflows for Apache Airflow environment. It assumes you've already created an Amazon MWAA environment. Once set up, a Linux Bastion Host acts as a jump server allowing a secure connection from your computer to the resources in your VPC. You'll then use a SOCKS proxy management add-on to control the proxy settings in your browser to access your Apache Airflow UI. **Topics** + [Private network](#private-network-lb-onconsole) + [Use cases](#private-network-lb-usecases) + [Before you begin](#private-network-lb-prereqs) + [Objectives](#private-network-lb-objectives) + [Step one: Create the bastion instance](#private-network-lb-create-bastion) + [Step two: Create the ssh tunnel](#private-network-lb-create-test) + [Step three: Configure the bastion security group as an inbound rule](#private-network-lb-create-sgsource) + [Step four: Copy the Apache Airflow URL](#private-network-lb-view-env) + [Step five: Configure proxy settings](#private-network-lb-browser-extension) + [Step six: Open the Apache Airflow UI](#private-network-lb-open) + [What's next?](#bastion-next-up) ## Private network This tutorial assumes you've chosen the **Private network** access mode for your Apache Airflow webserver. ![\[This image displays the architecture for an Amazon MWAA environment with a private webserver.\]](http://docs.aws.amazon.com/mwaa/latest/userguide/images/mwaa-private-web-server.png) The private network access mode limits access to the Apache Airflow UI to users *within your Amazon VPC* who have been granted access to the [IAM policy for your environment](access-policies.md). When you create an environment with private webserver access, you must package all of your dependencies in a Python wheel archive (`.whl`), then reference the `.whl` in your `requirements.txt`. For instructions on packaging and installing your dependencies using wheel, refer to [Managing dependencies using Python wheel](best-practices-dependencies.md#best-practices-dependencies-python-wheels). The following image depicts where to find the **Private network** option on the Amazon MWAA console. ![\[This image depicts where to find the Private network option on the Amazon MWAA console.\]](http://docs.aws.amazon.com/mwaa/latest/userguide/images/mwaa-console-private-network.png) ## Use cases You can use this tutorial after you've created an Amazon MWAA environment. You must use the same Amazon VPC, VPC security groups, and public subnets as your environment. ## Before you begin 1. Check for user permissions. Be sure that your account in AWS Identity and Access Management (IAM) has sufficient permissions to create and manage VPC resources. 1. Use your Amazon MWAA VPC. This tutorial assumes that you are associating the bastion host to an existing VPC. The Amazon VPC must be in the same region as your Amazon MWAA environment and have two private subnets, as defined in [Create the VPC network](vpc-create.md). 1. Create an SSH key. You need to create an Amazon EC2 SSH key (**.pem**) in the same Region as your Amazon MWAA environment to connect to the virtual servers. If you don't have an SSH key, refer to [Create or import a key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#prepare-key-pair) in the *Amazon EC2 User Guide*. ## Objectives In this tutorial, you'll do the following: 1. Create a Linux Bastion Host instance using a [CloudFormation template for an existing VPC](https://fwd.aws/vWMxm). 1. Authorize inbound traffic to the bastion instance's security group using an ingress rule on port `22`. 1. Authorize inbound traffic from an Amazon MWAA environment's security group to the bastion instance's security group. 1. Create an SSH tunnel to the bastion instance. 1. Install and configure the FoxyProxy add-on for the Firefox browser to access the Apache Airflow UI. ## Step one: Create the bastion instance The following section describes the steps to create the linux bastion instance using a [CloudFormation template for an existing VPC](https://fwd.aws/vWMxm) on the CloudFormation console. **To create the Linux Bastion Host** 1. Open the [Deploy Quick Start](https://fwd.aws/Jwzqv) page on the CloudFormation console. 1. Use the region selector in the navigation bar to choose the same AWS Region as your Amazon MWAA environment. 1. Choose **Next**. 1. Enter a name in the **Stack name** text field, such as `mwaa-linux-bastion`. 1. On the **Parameters**, **Network configuration** pane, choose the following options: 1. Choose your Amazon MWAA environment's **VPC ID**. 1. Choose your Amazon MWAA environment's **Public subnet 1 ID**. 1. Choose your Amazon MWAA environment's **Public subnet 2 ID**. 1. Enter the narrowest possible address range (for example, an internal CIDR range) in **Allowed bastion external access CIDR**. **Note** The simplest way to identify a range is to use the same CIDR range as your public subnets. For example, the public subnets in the CloudFormation template on the [Create the VPC network](vpc-create.md) page are `10.192.10.0/24` and `10.192.11.0/24`. 1. On the **Amazon EC2 configuration** pane, choose the following: 1. Choose your SSH key in the dropdown list in **Key pair name**. 1. Enter a name in **Bastion Host Name**. 1. Choose **true** for **TCP forwarding**. **Warning** TCP forwarding must be set to **true** in this step. Otherwise, you won't be able to create an SSH tunnel in the next step. 1. Choose **Next**, **Next**. 1. Select the acknowledgement, and then choose **Create stack**. To learn more about the architecture of your Linux Bastion Host, refer to [Linux Bastion Hosts on the AWS Cloud: Architecture](https://docs.aws.amazon.com/quickstart/latest/linux-bastion/architecture.html). ## Step two: Create the ssh tunnel The following steps describe how to create the ssh tunnel to your linux bastion. An SSH tunnel recieves the request from your local IP address to the linux bastion, which is why TCP forwarding for the linux bastion was set to `true` in previous steps. ------ #### [ macOS/Linux ] **To create a tunnel using the command line** 1. Open the [Instances](https://console.aws.amazon.com/ec2/v2/home#/Instances:) page on the Amazon EC2 console. 1. Choose an instance. 1. Copy the address in **Public IPv4 DNS**. For example, `ec2-4-82-142-1.compute-1.amazonaws.com`. 1. In your command prompt, navigate to the directory where your SSH key is stored. 1. Run the following command to connect to the bastion instance using ssh. Substitute the sample value with your SSH key name in `mykeypair.pem`. ``` ssh -i mykeypair.pem -N -D 8157 ec2-user@YOUR_PUBLIC_IPV4_DNS ``` ------ #### [ Windows (PuTTY) ] **To create a tunnel using PuTTY** 1. Open the [Instances](https://console.aws.amazon.com/ec2/v2/home#/Instances:) page on the Amazon EC2 console. 1. Choose an instance. 1. Copy the address in **Public IPv4 DNS**. For example, `ec2-4-82-142-1.compute-1.amazonaws.com`. 1. Open [PuTTY](https://www.putty.org/), select **Session**. 1. Enter the host name in **Host Name** as ec2-user@*YOUR\$1PUBLIC\$1IPV4\$1DNS* and the **port** as `22`. 1. Expand the **SSH** tab, select **Auth**. In **Private Key file for authentication**, choose your local "ppk" file. 1. Under SSH, choose the **Tunnels** tab, and then select the *Dynamic* and *Auto* options. 1. In **Source Port**, add the `8157` port (or any other unused port), and then leave the **Destination** port blank. Choose **Add**. 1. Choose the **Session** tab and enter a session name. For example `SSH Tunnel`. 1. Choose **Save**, **Open**. **Note** You might need to enter a pass phrase for your public key. ------ **Note** If you receive a `Permission denied (publickey)` error, we recommend using the [AWSSupport-TroubleshootSSH](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-awssupport-troubleshootssh.html) tool, and choose **Run this Automation (console)** to troubleshoot your SSH setup. ## Step three: Configure the bastion security group as an inbound rule Access to the servers and regular internet access from the servers is allowed with a special maintenance security group attached to those servers. The following steps describe how to configure the bastion security group as an inbound source of traffic to an environment's VPC security group. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. On the **Networking** pane, choose **VPC security group**. 1. Choose **Edit inbound rules**. 1. Choose **Add rule**. 1. Choose your VPC security group ID in the **Source** dropdown list. 1. Leave the remaining options blank, or set to their default values. 1. Choose **Save rules**. ## Step four: Copy the Apache Airflow URL The following steps describe how to open the Amazon MWAA console and copy the URL to the Apache Airflow UI. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Copy the URL in **Airflow UI** for subsequent steps. ## Step five: Configure proxy settings If you use an SSH tunnel with dynamic port forwarding, you must use a SOCKS proxy management add-on to control the proxy settings in your browser. For example, you can use the `--proxy-server` feature of Chromium to kick off a browser session, or use the FoxyProxy extension in the Mozilla FireFox browser. ### Option one: Setup an SSH Tunnel using local port forwarding If you do not wish to use a SOCKS proxy, you can set up an SSH tunnel using local port forwarding. The following example command accesses the Amazon EC2 *ResourceManager* web interface by forwarding traffic on local port 8157. 1. Open a new command prompt window. 1. Enter the following command to open an SSH tunnel. ``` ssh -i mykeypair.pem -N -L 8157:YOUR_VPC_ENDPOINT_ID-vpce.us-east-1.airflow.amazonaws.com:443 ubuntu@YOUR_PUBLIC_IPV4_DNS.us-east-1.compute.amazonaws.com ``` `-L` signifies the use of local port forwarding which you can use to specify a local port used to forward data to the identified remote port on the node's local webserver. 1. Enter `http://localhost:8157/` in your browser. **Note** You might need to use `https://localhost:8157/`. ### Option two: Proxies using the command line You can use most web browsers to configure proxies using a command line or configuration parameter. For example, with Chromium you can start the browser with the following command: ``` chromium --proxy-server="socks5://localhost:8157" ``` This starts a browser session which uses the ssh tunnel you created in previous steps to proxy its requests. You can open your Private Amazon MWAA environment URL (with *https://*) as follows: ``` https://YOUR_VPC_ENDPOINT_ID-vpce.us-east-1.airflow.amazonaws.com/home. ``` ### Option three: Proxies using FoxyProxy for Mozilla Firefox The following example demonstrates a FoxyProxy Standard (version 7.5.1) configuration for Mozilla Firefox. FoxyProxy provides a set of proxy management tools. It lets you use a proxy server for URLs that match patterns corresponding to domains used by the Apache Airflow UI. 1. In Firefox, open the [FoxyProxy Standard](https://addons.mozilla.org/en-US/firefox/addon/foxyproxy-standard/) extension page. 1. Choose **Add to Firefox**. 1. Choose **Add**. 1. Choose the FoxyProxy icon in your browser's toolbar, choose **Options**. 1. Copy the following code and save locally as `mwaa-proxy.json`. Substitute the sample value in *YOUR\$1HOST\$1NAME* with your **Apache Airflow URL**. ``` { "e0b7kh1606694837384": { "type": 3, "color": "#66cc66", "title": "airflow", "active": true, "address": "localhost", "port": 8157, "proxyDNS": false, "username": "", "password": "", "whitePatterns": [ { "title": "airflow-ui", "pattern": "YOUR_HOST_NAME", "type": 1, "protocols": 1, "active": true } ], "blackPatterns": [], "pacURL": "", "index": -1 }, "k20d21508277536715": { "active": true, "title": "Default", "notes": "These are the settings that are used when no patterns match a URL.", "color": "#0055E5", "type": 5, "whitePatterns": [ { "title": "all URLs", "active": true, "pattern": "*", "type": 1, "protocols": 1 } ], "blackPatterns": [], "index": 9007199254740991 }, "logging": { "active": true, "maxSize": 500 }, "mode": "patterns", "browserVersion": "82.0.3", "foxyProxyVersion": "7.5.1", "foxyProxyEdition": "standard" } ``` 1. On the **Import Settings from FoxyProxy 6.0\$1** pane, choose **Import Settings** and select the `mwaa-proxy.json` file. 1. Choose **OK**. ## Step six: Open the Apache Airflow UI The following steps describe how to open your Apache Airflow UI. 1. Open the [Environments](https://console.aws.amazon.com/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose **Open Airflow UI**. ## What's next? + Learn how to run Airflow CLI commands on an SSH tunnel to a bastion host in [Apache Airflow CLI command reference](airflow-cli-command-reference.md). + Learn how to upload DAG code to your Amazon S3 bucket in [Adding or updating DAGs](configuring-dag-folder.md). # Tutorial: Restricting an Amazon MWAA user's access to a subset of DAGs Amazon MWAA manages access to your environment by mapping your IAM principals to one or more of Apache Airflow's [default roles](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#default-roles). Use the following tutorial to restrict individual Amazon MWAA users to only access and interact with a specific DAG or a set of DAGs. **Note** The steps in this tutorial can be completed using federated access, as long as the IAM roles can be assumed. **Topics** + [Prerequisites](#limit-access-to-dags-prerequisites) + [Step one: Provide Amazon MWAA webserver access to your IAM principal with the default `Public` Apache Airflow role.](#limit-access-to-dags-apply-public-access) + [Step two: Create a new Apache Airflow custom role](#limit-access-to-dags-create-new-airflow-role) + [Step three: Assign the role you created to your Amazon MWAA user](#limit-access-to-dags-assign-role) + [Next steps](#limit-access-to-dags-next-up) + [Related resources](#limit-access-to-dags-related-resources) ## Prerequisites To complete the steps in this tutorial, you'll need the following: + An [Amazon MWAA environment with multiple DAGs](get-started.md) + An IAM principal, `Admin` with [AdministratorAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AdministratorAccess$jsonEditor) permissions, and an IAM user, `MWAAUser`, as the principal for which you can limit DAG access. For more information about admin roles, refer to [Administrator job function](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_administrator) in the *IAM User Guide* **Note** Don't attach permission policies directly to your IAM users. We recommend setting up IAM roles that users can assume to gain temporary access to your Amazon MWAA resources. + [AWS Command Line Interface version 2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install) installed. ## Step one: Provide Amazon MWAA webserver access to your IAM principal with the default `Public` Apache Airflow role. **To grant permission using the AWS Management Console** 1. Sign in to your AWS account with an `Admin` role and open the [IAM console](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Users**, then choose your Amazon MWAA IAM user from the users table. 1. On the user details page, under **Summary**, choose the **Permissions** tab, then choose **Permissions policies** to expand the card and choose **Add permissions**. 1. In the **Grant permissions** section, choose **Attach existing policies directly**, then choose **Create policy** to create and attach your own custom permissions policy. 1. On the **Create policy** page, choose **JSON**, then copy and paste the following JSON permissions policy in the policy editor. Tha policy grants webserver access to the user with the default `Public` Apache Airflow role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "airflow:CreateWebLoginToken", "Resource": [ "arn:aws:airflow:us-east-1:111122223333:role/YOUR_ENVIRONMENT_NAME/Public" ] } ] } ``` ------ ## Step two: Create a new Apache Airflow custom role **To create a new role using the Apache Airflow UI** 1. Using your administrator IAM role, open the [Amazon MWAA console](https://console.aws.amazon.com/mwaa/home) and launch your environment's Apache Airflow UI. 1. From the navigation pane at the top, hover on **Security** to open the dropdown list, then choose **List Roles** to access the default Apache Airflow roles. 1. From the roles list, select **User**, then at the beginning of the page choose **Actions** to open the dropdown. Choose **Copy Role**, and confirm **Ok** **Note** Copy the **Ops** or **Viewer** roles to grant more or less access, respectively. 1. Locate the new role you created in the table and choose **Edit record**. 1. On the **Edit Role** page, do the following: + For **Name**, enter a new name for the role in the text field. For example, **Restricted**. + For the list of **Permissions**, remove `can read on DAGs` and `can edit on DAGs`, then add read and write permissions for the set of DAGs you want to provide access to. For example, for a DAG, `example_dag.py`, add **`can read on DAG:example_dag`** and **`can edit on DAG:example_dag`**. Choose **Save**. Now you have a new role that limits access to a subset of DAGs available in your Amazon MWAA environment. You can assign this role to any existing Apache Airflow users. ## Step three: Assign the role you created to your Amazon MWAA user **To assign the new role** 1. Using access credentials for `MWAAUser`, run the following CLI command to retrieve your environment's webserver URL. ``` aws mwaa get-environment --name YOUR_ENVIRONMENT_NAME | jq '.Environment.WebserverUrl' ``` If successful, you'll refer to the following output: ``` "ab1b2345-678a-90a1-a2aa-34a567a8a901.c13.us-west-2.airflow.amazonaws.com" ``` 1. With `MWAAUser` signed in to the AWS Management Console, open a new browser window and access the following URl. Replace `Webserver-URL` with your information. ``` https:///home ``` If successful, you'll get a `Forbidden` error page because `MWAAUser` has not been granted permission to access the Apache Airflow UI yet. 1. With `Admin` signed in to the AWS Management Console, open the Amazon MWAA console again and launch your environment's Apache Airflow UI. 1. From the UI dashboard, expand the **Security** dropdown, and this time choose **List Users**. 1. In the users table, find the new Apache Airflow user and choose **Edit record**. The user's first name will match your IAM user name in the following pattern: `user/mwaa-user`. 1. On the **Edit User** page, in the **Role** section, add the new custom role you created, then choose **Save**. **Note** The **Last Name** field is required, but a space satisfies the requirement. The IAM `Public` principal grants the `MWAAUser` permission to access the Apache Airflow UI, while the new role provides the additional permissions needed to get their DAGs. **Important** Any of the 5 default roles (such as `Admin`) not authorized by IAM which are added using the Apache Airflow UI will be removed on next user log-in. ## Next steps + To learn more about managing access to your Amazon MWAA environment, and to get sample JSON IAM policies you can use for your environment users, refer to [Accessing an Amazon MWAA environment](access-policies.md) ## Related resources + [Access Control](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html) (Apache Airflow Documentation) – Learn more about the default Apache Airflow roles on the Apache Airflow documentation website. # Tutorial: Automate managing your own environment endpoints on Amazon MWAA If you use [AWS Organizations](https://docs.aws.amazon.com/) to manage multiple AWS accounts that share resources, Amazon MWAA lets you create and manage your own Amazon VPC endpoints. This means you can use stricter security policies that allow access only to the resources required by your environment. When you create an environment in a shared Amazon VPC, the account that owns the main Amazon VPC (*owner*) shares the two private subnets required by Amazon MWAA with other accounts (*participants*) that belong to the same organization. Participant accounts that share those subnets can then view, create, modify, and delete environments in the shared VPC. When you create an environment in a shared, or otherwise policy-restricted, Amazon VPC, Amazon MWAA will first create the service VPC resources, then enter a [https://docs.aws.amazon.com/mwaa/latest/API/API_Environment.html#mwaa-Type-Environment-Status](https://docs.aws.amazon.com/mwaa/latest/API/API_Environment.html#mwaa-Type-Environment-Status) state for up to 72 hours. When the environment status changes from `CREATING` to `PENDING`, Amazon MWAA sends an Amazon EventBridge notification of the change in state. This lets the owner account create the required endpoints on behalf of participants based on endpoint service information from the Amazon MWAA console or API, or programmatically In the following, we create new Amazon VPC endpoints using an Lambda function and an EventBridge rule that listens to Amazon MWAA state change notifications. Here, we create the new endpoints in the same Amazon VPC as the environment. To set up a shared Amazon VPC, create the EventBridge rule and Lambda function would in the owner account, and the Amazon MWAA environment in the participant account. **Topics** + [Prerequisites](#tutorials-customer-managed-endpoints-prerequisites) + [Create the Amazon VPC](#tutorials-customer-managed-endpoints-create-vpc) + [Create the Lambda function](#tutorials-customer-managed-endpoints-create-lambda-function) + [Create the EventBridge rule](#tutorials-customer-managed-endpoints-create-eb-rule) + [Create the Amazon MWAA environment](#tutorials-customer-managed-endpoints-create-mwaa) ## Prerequisites To complete the steps in this tutorial, you will need the following: + ... ## Create the Amazon VPC Use the following CloudFormation template and AWS CLI command to create a new Amazon VPC. The template sets up the Amazon VPC resources and modifies the endpoint policy to restrict access to a specific queue. 1. Download the CloudFormation [template](samples/cfn-vpc-private-network.zip), then unzip the `.yml` file. 1. In a new command prompt window, navigate to the folder where you saved the template, then use [https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html) to create the stack. The `--template-body` flag specifies the path to the template. ``` aws cloudformation create-stack --stack-name stack-name --template-body file://cfn-vpc-private-network.yml ``` In the next section, you'll create the Lambda function. ## Create the Lambda function Use the following Python code and IAM JSON policy to create a new Lambda function and execution role. This function creates Amazon VPC endpoints for a private Apache Airflow webserver and an Amazon SQS queue. Amazon MWAA uses Amazon SQS to queue tasks with Celery among multiple workers when scaling your environment. 1. Download the Python [function code](./samples/mwaa-lambda-shared-vpc.zip). 1. Download the IAM [permission policy](./samples/lambda-mwaa-shared-vpce-policy.zip), then unzip the file. 1. Open a command prompt, then navigate to the folder where you saved the JSON permission policy. Use the IAM [https://docs.aws.amazon.com/](https://docs.aws.amazon.com/) command to create the new role. ``` aws iam create-role --role-name function-role \ --assume-role-policy-document file://lambda-mwaa-vpce-policy.json ``` Note the role ARN from the AWS CLI response. In the next step, we specify this new role as the function's execution role using its ARN. 1. Navigate to the folder where you saved the function code, then use the[https://docs.aws.amazon.com/](https://docs.aws.amazon.com/) command to create a new function. ``` aws lambda create-function --function-name mwaa-vpce-lambda \ --zip-file file://mwaa-lambda-shared-vpc.zip --runtime python3.8 --role arn:aws:iam::123456789012:role/function-role --handler lambda_handler ``` Note the function ARN from the AWS CLI response. In the next step we specify the ARN to configure the function as a target for a new EventBridge rule. In the next section, you create the EventBridge rule that invokes this function when the environment enters a `PENDING` state. ## Create the EventBridge rule Do the following to create a new rule that listens for Amazon MWAA notifications and targets your new Lambda function. 1. Use the EventBridge `put-rule` command to create a new EventBridge rule. ``` aws events put-rule --name "mwaa-lambda-rule" \ --event-pattern "{\"source\":[\"aws.airflow\"],\"detail-type\":[\"MWAA Environment Status Change\"]}" ``` The event pattern listens for notifications that Amazon MWAA sends whenever an environment status changes. ``` { "source": ["aws.airflow"], "detail-type": ["MWAA Environment Status Change"] } ``` 1. Use the `put-targets` command to add the Lambda function as a target for the new rule. ``` aws events put-targets --rule "mwaa-lambda-rule" \ --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789012:function:mwaa-vpce-lambda" ``` You're ready to create a new Amazon MWAA environment with customer-managed Amazon VPC endpoints. ## Create the Amazon MWAA environment Use the Amazon MWAA console to create a new environment with customer-managed Amazon VPC endpoints. 1. Open the [Amazon MWAA](https://console.aws.amazon.com/mwaa/home/) console, and choose **Create an environment**. 1. For **Name** enter a unique name. 1. For **Airflow version** choose the latest version. 1. Choose an **Amazon S3 bucket** and a **DAGs folder**, such as `dags/` to use with the environment, then choose **Next**. 1. On the **Configure advanced settings** page, do the following: 1. For **Virtual Private Cloud**, choose the Amazon VPC you created in the [previous step](#tutorials-customer-managed-endpoints-create-vpc). 1. For **webserver access**, choose **Public network (internet accessible)**. 1. For **Security groups**, choose the security group you created with CloudFormation. Because the security groups for the AWS PrivateLink endpoints from the earlier step are self-referencing, you must choose the same security group for your environment. 1. For **Endpoint management**, choose **Customer managed endpoints**. 1. Keep the remaining default settings, then choose **Next**. 1. Review your selections, then choose **Create environment**. **Tip** For more information about setting up a new environment, refer to [Getting started with Amazon MWAA](get-started.md). When the environment is `PENDING`, Amazon MWAA sends a notification that matches the event pattern you set for your rule. The rule invokes your Lambda function. The function parses the notification event and gets the required endpoint information for the webserver and the Amazon SQS queue. It then creates the endpoints in your Amazon VPC. When the endpoints are available, Amazon MWAA resumes creating your environment. When ready, the environment status changes to `AVAILABLE` and you can access the Apache Airflow webserver using the Amazon MWAA console.