AWS Direct Connect  - Building a Scalable and Secure Multi-VPC AWS Network Infrastructure
MACsec security on Direct Connect connectionsAWS Direct Connect resiliency recommendationsAWS Direct Connect SiteLink

AWS Direct Connect 

While VPN over internet is a great option to get started, internet connectivity may not be reliable for production traffic. Because of this unreliability, many customers choose AWS Direct Connect. AWS Direct Connect is a networking service that provides an alternative to using the internet to connect to AWS. Using AWS Direct Connect, data that would have previously been transported over the internet is delivered through a private network connection between your facilities and AWS. In many circumstances, private network connections can reduce costs, increase bandwidth, and provide a more consistent network experience than internet-based connections. There are several ways to use AWS Direct Connect to connect to VPCs:

A diagram depicting ways to connect your on-premises data centers using AWS Direct Connect

Ways to connect your on-premises data centers using AWS Direct Connect

  • Option 1: Create a private virtual interface (VIF) to a VGW attached to a VPC — You can create 50 VIFs per Direct Connect connection, allowing you to connect to a maximum of 50 VPCs (one VIF provides connectivity to one VPC). There is one BGP peering per VPC. Connectivity in this setup is restricted to the AWS Region that the Direct Connect location is homed to. The one-to-one mapping of VIF to VPC (and lack of global access) makes this the least preferred way to access VPCs in the Landing Zone.

  • Option 2: Create a private VIF to a Direct Connect gateway associated with multiple VGWs (each VGW is attached to a VPC) — A Direct Connect gateway is a globally available resource. You can create the Direct Connect gateway in any Region and access it from all other Regions, including GovCloud (excluding China). A Direct Connect Gateway can connect to up to 20 VPCs (via VGWs) globally in any AWS account over a single private VIF. This is a great option if a Landing Zone consists of a small number of VPCs (ten or fewer VPCs) and/or you need global access. There is one BGP peering session per Direct Connect Gateway per Direct Connect connection. Direct Connect gateway is only for north/south traffic flow and does not permit VPC-to-VPC connectivity. Refer to Virtual private gateway associations in the AWS Direct Connect documentation for more details. With this option, the connectivity is not restricted to the AWS Region where the Direct Connect location is homed to. AWS Direct Connect gateway is only for north/south traffic flow and does not permit VPC-to-VPC connectivity. An exception to this rule is when a supernet is advertised across two or more VPCs that have their attached VGWs associated with the same AWS Direct Connect gateway and on the same virtual interface. In this case, VPCs can communicate with each other through the AWS Direct Connect endpoint. Refer to AWS Direct Connect gateways documentation for more details.

  • Option 3: Create a transit VIF to a Direct Connect gateway associated with Transit Gateway — You can associate a Transit Gateway instance to a Direct Connect gateway by using a Transit VIF. AWS Direct Connect now supports connections to Transit Gateway for all port speeds, providing a more cost-effective choice for Transit Gateway users when high-speed connections (greater than 1Gbps) are not required. This enables you to use Direct Connect at speeds of 50, 100, 200, 300, 400, and 500 Mbps connecting to Transit Gateway. Transit VIF allows you to connect your on-premises data center to up to six Transit Gateway instances per AWS Direct Connect gateway (which can connect to thousands of VPCs) across different AWS Regions and AWS accounts over single transit VIF and BGP peering. This is the simplest setup among the options for connecting multiple VPCs at scale, but you should be mindful of the Transit Gateway quotas. One key limit to note is that you can advertise only 200 prefixes from a Transit Gateway to on-premises router over the transit VIF. With the previous options, you pay for Direct Connect pricing. For this option, you also pay for the Transit Gateway attachment and data processing charges. For more information, refer to the Transit Gateway Associations on Direct Connect documentation.

  • Option 4: Create a VPN connection to Transit Gateway over Direct Connect public VIF — A public VIF allows you to access all AWS public services and endpoints using the public IP addresses. When you create a VPN attachment on a Transit Gateway, you get two public IP addresses for VPN endpoints at the AWS side. These public IPs are reachable over the public VIF. You can create as many VPN connections to as many Transit Gateway instances as you want over Public VIF. When you create a BGP peering over the public VIF, AWS advertises the entire AWS public IP range to your router. To ensure that you only permit certain traffic (for example, allowing traffic only to the VPN termination endpoints) you are advised to use a firewall on-premises facilities. This option can be used to encrypt your Direct Connect at the network layer.

  • Option 5: Create a VPN connection to Transit Gateway over AWS Direct Connect using Private IP VPN — Private IP VPN is a feature that provides customers the ability to deploy AWS Site-to-Site VPN connections over Direct Connect using private IP addresses. With this feature, you can encrypt traffic between your on-premises networks and AWS through Direct Connect connections without the need for public IP addresses, thus enhancing security and network privacy at the same time. Private IP VPN is deployed on top of Transit VIFs, so it allows you to use Transit Gateway for centralized management of customers’ VPCs and connections to the on-premises networks in a more secured, private, and scalable manner.

  • Option 6: Create GRE tunnels to Transit Gateway over a transit VIF – The Transit Gateway Connect attachment type supports GRE. With Transit Gateway Connect, SD-WAN infrastructure can be natively connected to AWS without having to set up IPsec VPNs between SD-WAN network virtual appliances and Transit Gateway. The GRE tunnels can be established over a transit VIF, having Transit Gateway Connect as the attachment type, providing higher bandwidth performance compared to a VPN connection. For more information, refer to the Simplify SD-WAN connectivity with AWS Transit Gateway Connect blog post.

The “transit VIF to Direct Connect gateway” option might seem to be the best option because it lets you consolidate all your on-premises connectivity for a given AWS Region at a single point (Transit Gateway) using a single BGP session per Direct Connect connection; however, some of the limits and considerations around this option might lead you to use both private and transit VIFs in conjuction for your Landing Zone connectivity requirements.

The following figure illustrates a sample setup where Transit VIF is used as a default method for connecting to VPCs and a private VIF is used for an edge use case where exceptionally large amounts of data must be transferred from an on-premises Data Center to the media VPC. Private VIF is used to avoid Transit Gateway data processing charges. As a best practice, you should have at least two connections at two different Direct Connect locations for maximum redundancy—a total of four connections. You create one VIF per connection for a total of four private VIFs and four transit VIFs. You can also create a VPN as backup connectivity to AWS Direct Connect connections.

With the “Create GRE tunnels to Transit Gateway over a transit VIF” option, you get the capability to natively connect your SD-WAN infrastructure with AWS. It eliminates the need to setup IPsec VPNs between SD-WAN network virtual appliances and Transit Gateway.

A diagram depicting a sample reference architecture for hybrid connectivity

Sample reference architecture for hybrid connectivity

Use the Network Services account for creating Direct Connect resources enabling demarcation of network administrative boundaries. The Direct Connect connections, Direct Connect gateways, and Transit Gateways can all reside in a Network Services account. To share the AWS Direct Connect connectivity with your Landing Zone, simply share the Transit Gateway through AWS RAM with other accounts.

MACsec security on Direct Connect connections

Customers can use MAC Security Standard (MACsec) encryption (IEEE 802.1AE) with their Direct Connect connections for 10 Gbps and 100 Gbps dedicated connections at select locations. With this capability, customers can secure their data on the layer 2 level, and Direct Connect delivers point-to-point encryption. To enable the Direct Connect MACsec feature, ensure that the MACsec pre-requisites are met. Because MACsec protects links on a hop-by-hop basis, your device must have a direct layer 2 adjacency with our Direct Connect device. Your last-mile provider can help you verify that your connection will work with MACsec. For more information refer to Adding MACsec security to AWS Direct Connect connections.

AWS Direct Connect resiliency recommendations

With AWS Direct Connect, customers can achieve highly resilient connectivity into their Amazon VPCs and AWS resources from their on-premises networks. It is best practice that customers connect from multiple data centers to eliminate any single point physical location failures. It is also recommended that, depending on the type of workloads, customers utilize more than one Direct Connect connection for redundancy.

AWS also offers the AWS Direct Connect Resiliency Toolkit, which provides customers with a connection wizard with multiple redundancy models; to help them determine which model works best for their service level agreement (SLA) requirements and design their hybrid connectivity using Direct Connect connections accordingly. For more information, refer to AWS Direct Connect Resiliency Recommendations.

Previously, configuring site-to-site links for your on-premises networks was only possible by using direct circuit buildout through dark fiber or other technologies, IPSEC VPNs, or by using third-party circuit providers with technologies such as MPLS, MetroEthernet, or legacy T1 circuits. With the advent of SiteLink, customers can now enable direct site-to-site connectivity for their on-premises location that terminate at an AWS Direct Connect location. Use your Direct Connect circuit to provide site-to-site connectivity without having to route traffic through your VPCs, bypassing the AWS region completely.

Now, you can create global, reliable, and pay-as-you-go connections between the offices and data centers in your global network by sending data over the fastest path between AWS Direct Connect locations.

A diagram AWS Direct Connect SiteLink

Sample reference architecture for AWS Direct Connect SiteLink

When using SiteLink, you first connect your on-premises networks to AWS at any of over 100 AWS Direct Connect locations worldwide. Then, you create virtual interfaces (VIFs) on those connections and enable SiteLink. Once all VIFs are attached to the same AWS Direct Connect gateway (DXGW), you can start sending data between them. Your data follows the shortest path between AWS Direct Connect locations to its destination, using the fast, secure, and reliable AWS global network. You don’t need to have any resources in any AWS Region to use SiteLink.

With SiteLink, the DXGW learns IPv4/IPv6 prefixes from your routers over SiteLink enabled VIFs, runs BGP best path algorithm, updates attributes such as NextHop and AS_Path, and re-advertises these BGP prefixes to the rest of your SiteLink-enabled VIFs associated with that DXGW. If you disable SiteLink on a VIF, the DXGW will not advertise the learned on-premises prefixes over this VIF to the other SiteLink-enabled VIFs. The on-premises prefixes from a SiteLink disabled VIF is only advertised to the DXGW Gateway associations, such as AWS Virtual Private Gateways (VGWs) or Transit Gateway (TGW) instances associated with the DXGW.

A diagram showing Sitelink traffic flow examples

Sitelink allows traffic flows example

SiteLink allows customers to use the AWS global network to function as a primary or secondary/backup connection between their remote locations, with high bandwidth and low latency, with dynamic routing to control which locations can communicate with each other and with your AWS regional resources.

For more information, refer to Introducing AWS Direct Connect SiteLink.