创建 VPC 网络 - Amazon Managed Workflows for Apache Airflow

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

创建 VPC 网络

Amazon MWAA 需要 Amazon VPC 和特定的网络组件来支持环境。本指南介绍了为 Amazon MWAA 环境创建 Amazon VPC 网络的不同选项。

注意

Apache Airflow 在低延迟网络环境中效果最好。如果您使用的是将流量路由到其他区域或本地环境的现有 Amazon VPC,我们建议您为 Amazon SQS、CloudWatch、Amazon S3 和 AWS KMS 添加 AWS PrivateLink 端点。有关为 Amazon MWAA 配置 AWS PrivateLink 的更多信息,请参阅创建不可互联网访问的 Amazon VPC 网络

先决条件

AWS Command Line Interface(AWS CLI)是一种开源工具,让您能够在命令行 Shell 中使用命令与 AWS 服务进行交互。要完成本节中的步骤,您需要以下满足以下条件:

开始前的准备工作

创建 Amazon VPC 网络的选项

下一节介绍可用于为环境创建 Amazon VPC 网络的选项。

注意

Amazon MWAA 不支持在美国东部(弗吉尼亚州北部)区域中使用 use1-az3 可用区(AZ)。在美国东部(弗吉尼亚州北部)区域中创建用于 Amazon MWAA 的 VPC 时,必须在 AWS CloudFormation(CFN)模板中显式分配 AvailabilityZone。分配的可用区名称不得映射到 use1-az3。您可以通过运行以下命令来检索可用区名称与其对应可用区 ID 的详细映射:

aws ec2 describe-availability-zones --region us-east-1

选项一:在 Amazon MWAA 控制台上创建 VPC 网络

下一节显示如何在 Amazon MWAA 控制台上创建 VPC 网络。此选项使用 通过互联网进行公共路由。它可用于具有私有网络公有网络访问模式的 Apache Airflow Web 服务器

下图显示了在 Amazon MWAA 控制台上哪里可以找到创建 MWAA VPC 按钮。

此图显示了在 Amazon MWAA 控制台上哪里可以找到创建 MWAA VPC。

选项二:创建访问互联网的 Amazon VPC 网络

以下 AWS CloudFormation 模板在默认 AWS 区域创建可互联网访问的 Amazon VPC 网络。此选项使用 通过互联网进行公共路由。此模板可用于具有私有网络公有网络访问模式的 Apache Airflow Web 服务器

  1. 复制以下模板的内容并将其作为 cfn-vpc-public-private.yaml 保存在本地中。您也可以使用下载模板

    Description: This template deploys a VPC, with a pair of public and private subnets spread across two Availability Zones. It deploys an internet gateway, with a default route on the public subnets. It deploys a pair of NAT gateways (one in each AZ), and default routes for them in the private subnets. Parameters: EnvironmentName: Description: An environment name that is prefixed to resource names Type: String Default: mwaa- VpcCIDR: Description: Please enter the IP range (CIDR notation) for this VPC Type: String Default: 10.192.0.0/16 PublicSubnet1CIDR: Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone Type: String Default: 10.192.10.0/24 PublicSubnet2CIDR: Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone Type: String Default: 10.192.11.0/24 PrivateSubnet1CIDR: Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone Type: String Default: 10.192.20.0/24 PrivateSubnet2CIDR: Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone Type: String Default: 10.192.21.0/24 Resources: VPC: Type: AWS::EC2::VPC Properties: CidrBlock: !Ref VpcCIDR EnableDnsSupport: true EnableDnsHostnames: true Tags: - Key: Name Value: !Ref EnvironmentName InternetGateway: Type: AWS::EC2::InternetGateway Properties: Tags: - Key: Name Value: !Ref EnvironmentName InternetGatewayAttachment: Type: AWS::EC2::VPCGatewayAttachment Properties: InternetGatewayId: !Ref InternetGateway VpcId: !Ref VPC PublicSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 0, !GetAZs '' ] CidrBlock: !Ref PublicSubnet1CIDR MapPublicIpOnLaunch: true Tags: - Key: Name Value: !Sub ${EnvironmentName} Public Subnet (AZ1) PublicSubnet2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 1, !GetAZs '' ] CidrBlock: !Ref PublicSubnet2CIDR MapPublicIpOnLaunch: true Tags: - Key: Name Value: !Sub ${EnvironmentName} Public Subnet (AZ2) PrivateSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 0, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet1CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Subnet (AZ1) PrivateSubnet2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 1, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet2CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Subnet (AZ2) NatGateway1EIP: Type: AWS::EC2::EIP DependsOn: InternetGatewayAttachment Properties: Domain: vpc NatGateway2EIP: Type: AWS::EC2::EIP DependsOn: InternetGatewayAttachment Properties: Domain: vpc NatGateway1: Type: AWS::EC2::NatGateway Properties: AllocationId: !GetAtt NatGateway1EIP.AllocationId SubnetId: !Ref PublicSubnet1 NatGateway2: Type: AWS::EC2::NatGateway Properties: AllocationId: !GetAtt NatGateway2EIP.AllocationId SubnetId: !Ref PublicSubnet2 PublicRouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub ${EnvironmentName} Public Routes DefaultPublicRoute: Type: AWS::EC2::Route DependsOn: InternetGatewayAttachment Properties: RouteTableId: !Ref PublicRouteTable DestinationCidrBlock: 0.0.0.0/0 GatewayId: !Ref InternetGateway PublicSubnet1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref PublicSubnet1 PublicSubnet2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref PublicSubnet2 PrivateRouteTable1: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Routes (AZ1) DefaultPrivateRoute1: Type: AWS::EC2::Route Properties: RouteTableId: !Ref PrivateRouteTable1 DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NatGateway1 PrivateSubnet1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTable1 SubnetId: !Ref PrivateSubnet1 PrivateRouteTable2: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub ${EnvironmentName} Private Routes (AZ2) DefaultPrivateRoute2: Type: AWS::EC2::Route Properties: RouteTableId: !Ref PrivateRouteTable2 DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NatGateway2 PrivateSubnet2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTable2 SubnetId: !Ref PrivateSubnet2 SecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupName: "mwaa-security-group" GroupDescription: "Security group with a self-referencing inbound rule." VpcId: !Ref VPC SecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref SecurityGroup IpProtocol: "-1" SourceSecurityGroupId: !Ref SecurityGroup Outputs: VPC: Description: A reference to the created VPC Value: !Ref VPC PublicSubnets: Description: A list of the public subnets Value: !Join [ ",", [ !Ref PublicSubnet1, !Ref PublicSubnet2 ]] PrivateSubnets: Description: A list of the private subnets Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]] PublicSubnet1: Description: A reference to the public subnet in the 1st Availability Zone Value: !Ref PublicSubnet1 PublicSubnet2: Description: A reference to the public subnet in the 2nd Availability Zone Value: !Ref PublicSubnet2 PrivateSubnet1: Description: A reference to the private subnet in the 1st Availability Zone Value: !Ref PrivateSubnet1 PrivateSubnet2: Description: A reference to the private subnet in the 2nd Availability Zone Value: !Ref PrivateSubnet2 SecurityGroupIngress: Description: Security group with self-referencing inbound rule Value: !Ref SecurityGroupIngress
  2. 在命令提示符下,导航到存储 cfn-vpc-public-private.yaml 的目录。例如:

    cd mwaaproject
  3. 输入 aws cloudformation create-stack 命令来使用 AWS CLI 创建堆栈。

    aws cloudformation create-stack --stack-name mwaa-environment --template-body file://cfn-vpc-public-private.yaml
    注意

    创建 Amazon VPC 基础设施需要大约 30 分钟。

选项三:创建不可互联网访问的 Amazon VPC 网络

以下 AWS CloudFormation 模板将在默认 AWS 区域创建 不可访问互联网的 Amazon VPC 网络。

此选项使用 无法访问互联网的私有路由。此模板只能用于具有私有网络访问模式的 Apache Airflow Web 服务器。它为环境使用的 AWS 服务创建所需的 VPC 端点

  1. 复制以下模板的内容并将其作为 cfn-vpc-private.yaml 保存在本地中。您也可以使用下载模板

    AWSTemplateFormatVersion: "2010-09-09" Parameters: VpcCIDR: Description: The IP range (CIDR notation) for this VPC Type: String Default: 10.192.0.0/16 PrivateSubnet1CIDR: Description: The IP range (CIDR notation) for the private subnet in the first Availability Zone Type: String Default: 10.192.10.0/24 PrivateSubnet2CIDR: Description: The IP range (CIDR notation) for the private subnet in the second Availability Zone Type: String Default: 10.192.11.0/24 Resources: VPC: Type: AWS::EC2::VPC Properties: CidrBlock: !Ref VpcCIDR EnableDnsSupport: true EnableDnsHostnames: true Tags: - Key: Name Value: !Ref AWS::StackName RouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub "${AWS::StackName}-route-table" PrivateSubnet1: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 0, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet1CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub "${AWS::StackName} Private Subnet (AZ1)" PrivateSubnet2: Type: AWS::EC2::Subnet Properties: VpcId: !Ref VPC AvailabilityZone: !Select [ 1, !GetAZs '' ] CidrBlock: !Ref PrivateSubnet2CIDR MapPublicIpOnLaunch: false Tags: - Key: Name Value: !Sub "${AWS::StackName} Private Subnet (AZ2)" PrivateSubnet1RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref RouteTable SubnetId: !Ref PrivateSubnet1 PrivateSubnet2RouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref RouteTable SubnetId: !Ref PrivateSubnet2 S3VpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3" VpcEndpointType: Gateway VpcId: !Ref VPC RouteTableIds: - !Ref RouteTable SecurityGroup: Type: AWS::EC2::SecurityGroup Properties: VpcId: !Ref VPC GroupDescription: Security Group for Amazon MWAA Environments to access VPC endpoints GroupName: !Sub "${AWS::StackName}-mwaa-vpc-endpoints" SecurityGroupIngress: Type: AWS::EC2::SecurityGroupIngress Properties: GroupId: !Ref SecurityGroup IpProtocol: "-1" SourceSecurityGroupId: !Ref SecurityGroup SqsVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.sqs" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup CloudWatchLogsVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.logs" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup CloudWatchMonitoringVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.monitoring" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup KmsVpcEndoint: Type: AWS::EC2::VPCEndpoint Properties: ServiceName: !Sub "com.amazonaws.${AWS::Region}.kms" VpcEndpointType: Interface VpcId: !Ref VPC PrivateDnsEnabled: true SubnetIds: - !Ref PrivateSubnet1 - !Ref PrivateSubnet2 SecurityGroupIds: - !Ref SecurityGroup Outputs: VPC: Description: A reference to the created VPC Value: !Ref VPC MwaaSecurityGroupId: Description: Associates the Security Group to the environment to allow access to the VPC endpoints Value: !Ref SecurityGroup PrivateSubnets: Description: A list of the private subnets Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]] PrivateSubnet1: Description: A reference to the private subnet in the 1st Availability Zone Value: !Ref PrivateSubnet1 PrivateSubnet2: Description: A reference to the private subnet in the 2nd Availability Zone Value: !Ref PrivateSubnet2
  2. 在命令提示符下,导航到存储 cfn-vpc-private.yml 的目录。例如:

    cd mwaaproject
  3. 输入 aws cloudformation create-stack 命令来使用 AWS CLI 创建堆栈。

    aws cloudformation create-stack --stack-name mwaa-private-environment --template-body file://cfn-vpc-private.yml
    注意

    创建 Amazon VPC 基础设施需要大约 30 分钟。

  4. 您需要创建一种机制,以便从计算机访问这些 VPC 端点。要了解更多信息,请参阅 在 Amazon MWAA 上管理对服务特定 Amazon VPC 端点的访问

注意

您可以在 Amazon MWAA 安全组的 CIDR 中进一步限制出站访问。例如,您可以通过添加自引用出站规则、Amazon S3 的前缀列表和 Amazon VPC 的 CIDR 来限制自身。

接下来做什么?