Preserve routable IP space in multi-account VPC designs for non-workload subnets
Created by Adam Spicer (AWS)
Code repository: Non routable secondary CIDRs pattern | Environment: Production | Technologies: Infrastructure; DevOps; Management & governance; Networking |
AWS services: AWS Transit Gateway; Amazon VPC; Elastic Load Balancing (ELB) |
Summary
Amazon Web Services (AWS) has published best practices that recommend using dedicated subnets in a virtual private cloud (VPC) for both transit gateway attachments and Gateway Load Balancer endpoints (to support AWS Network Firewall or third-party appliances). These subnets are used to contain elastic network interfaces for these services. If you use both AWS Transit Gateway and a Gateway Load Balancer, two subnets are created in each Availability Zone for the VPC. Because of the way VPCs are designed, these extra subnets can’t be smaller than a /28 mask and can consume precious routable IP space that could otherwise be used for routable workloads. This pattern demonstrates how you can use a secondary, non-routable Classless Inter-Domain Routing (CIDR) range for these dedicated subnets to help preserve routable IP space.
Prerequisites and limitations
Prerequisites
Multi-VPC strategy for routable IP space
A non-routable CIDR range for the services you’re using (transit gateway attachments and Gateway Load Balancer
or Network Firewall endpoints )
Architecture
Target architecture
This pattern includes two reference architectures: one architecture has subnets for transit gateway (TGW) attachments and a Gateway Load Balancer endpoint (GWLBe), and the second architecture has subnets for TGW attachments only.
Architecture 1 ‒ TGW-attached VPC with ingress routing to an appliance
The following diagram represents a reference architecture for a VPC that spans two Availability Zones. On ingress, the VPC uses an ingress routing pattern
This pattern uses a non-routable CIDR range for the TGW attachment subnet and the GWLBe subnet. In the TGW routing table, this non-routable CIDR is configured with a blackhole (static) route by using a set of more specific routes. If the routes were to get propagated to the TGW routing table, these more specific blackhole routes would apply.
In this example, the /23 routable CIDR is divided up and fully allocated to routable subnets.
Architecture 2 – TGW-attached VPC
The following diagram represents another reference architecture for a VPC that spans two Availability Zones. A TGW attachment supports outbound traffic (egress) from the private subnets to a separate VPC. It uses a non-routable CIDR range only for the TGW attachments subnet. In the TGW routing table, this non-routable CIDR is configured with a blackhole route by using a set of more specific routes. If the routes were to get propagated to the TGW routing table, these more specific blackhole routes would apply.
In this example, the /23 routable CIDR is divided up and fully allocated to routable subnets.
Tools
AWS services and resources
Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. In this pattern, VPC secondary CIDRs are used to preserve routable IP space in workload CIDRs.
Internet gateway ingress routing
(edge associations) can be used along with Gateway Load Balancer endpoints for dedicated non-routable subnets. AWS Transit Gateway is a central hub that connects VPCs and on-premises networks. In this pattern, VPCs are centrally attached to a transit gateway, and the transit gateway attachments are in a dedicated non-routable subnet.
Gateway Load Balancers help you deploy, scale, and manage virtual appliances, such as firewalls, intrusion detection and prevention systems, and deep packet inspection systems. The gateway serves as a single entry and exit point for all traffic. In this pattern, endpoints for a Gateway Load Balancer can be used in a dedicated non-routable subnet.
AWS Network Firewall is a stateful, managed, network firewall and intrusion detection and prevention service for VPCs in the AWS Cloud. In this pattern, endpoints for an firewall can be used in a dedicated non-routable subnet.
Code repository
A runbook and AWS CloudFormation templates for this pattern are available in the GitHub Non-Routable Secondary CIDR Patterns
Best practices
AWS Transit Gateway
Use a separate subnet for each transit gateway VPC attachment.
Allocate a /28 subnet from the secondary non-routable CIDR range for the transit gateway attachment subnets.
In each transit gateway routing table, add a static, more specific route for the non-routable CIDR range as a blackhole.
Gateway Load Balancer and ingress routing
Use ingress routing to direct traffic from the internet to the Gateway Load Balancer endpoints.
Use a separate subnet for each Gateway Load Balancer endpoint.
Allocate a /28 subnet from the secondary non-routable CIDR range for the Gateway Load Balancer endpoint subnets.
Epics
| Task | Description | Skills required |
|---|---|---|
Determine non-routable CIDR range. | Determine a non-routable CIDR range that will be used for the transit gateway attachment subnet and (optionally) for any Gateway Load Balancer or Network Firewall endpoint subnets. This CIDR range will be used as the secondary CIDR for the VPC. It must not be routable from the VPC’s primary CIDR range or the larger network. | Cloud architect |
Determine routable CIDR ranges for VPCs. | Determine a set of routable CIDR ranges that will be used for your VPCs. This CIDR range will be used as the primary CIDR for your VPCs. | Cloud architect |
Create VPCs. | Create your VPCs and attach them to the transit gateway. Each VPC should have a primary CIDR range that is routable and a secondary CIDR range that is non-routable, based on the ranges you determined in the previous two steps. | Cloud architect |
| Task | Description | Skills required |
|---|---|---|
Create more specific non-routable CIDRs as blackholes. | Each transit gateway routing table needs to have a set of blackhole routes created for the non-routable CIDRs. These are configured to ensure that any traffic from the secondary VPC CIDR remains non-routable and doesn't leak into the larger network. These routes should be more specific than the non-routable CIDR that is set as the secondary CIDR on the VPC. For example, if the secondary non-routable CIDR is 100.64.0.0/26, the blackhole routes in the transit gateway routing table should be 100.64.0.0/27 and 100.64.0.32/27. | Cloud architect |
Related resources
Additional information
The non-routable secondary CIDR range can also be useful when working with larger scaled container deployments that require a large set of IP addresses. You can use this pattern with a private NAT Gateway to use a non-routable subnet to host your container deployments. For more information, see the blog post How to solve Private IP exhaustion with Private NAT Solution
