Getting started with Amazon Elastic VMware Service
Use this guide to get started with Amazon Elastic VMware Service (Amazon EVS). You’ll learn how to create an Amazon EVS environment with hosts within your own Amazon Virtual Private Cloud (VPC).
After you’re finished, you’ll have an Amazon EVS environment that you can use to migrate your VMware vSphere-based workloads to the AWS Cloud.
Amazon EVS can deploy VCF for you, or you can use Self-deployed mode to install VCF yourself. For the VCF versions that Amazon EVS supports, see VCF versions and EC2 instance types provided by Amazon EVS.
For Self-deployed mode, see Creating an Amazon EVS environment with Self-deployed mode. The procedures under Create an Amazon EVS environment cover environment creation where Amazon EVS deploys VCF for you.
Important
To get started as simply and quickly as possible, this topic includes steps to create a VPC, and specifies minimum requirements for DNS server configuration and Amazon EVS environment creation.
Before creating these resources, we recommend that you plan out your IP address space and DNS record setup that meets your requirements.
You should also familiarize yourself with VCF 5.2.x requirements. See the VCF 5.2.x release notes
Important
For information about VCF versions provided by Amazon EVS, see VCF versions and EC2 instance types provided by Amazon EVS.
Topics
Prerequisites
Before getting started, you must complete the Amazon EVS prerequisite tasks. For more information, see Setting up Amazon Elastic VMware Service.
Create a VPC with subnets and route tables
Note
The VPC, subnets, and Amazon EVS environment must all be created in the same account. Amazon EVS does not support cross-account sharing of VPC subnets or Amazon EVS environments.
Example
Choose your HCX connectivity option
Select one connectivity option for your Amazon EVS environment:
-
Private connectivity: Provides high-performance network pathways for HCX, optimizing reliability and consistency. Requires use of AWS Direct Connect or Site-to-Site VPN for external network connectivity.
-
Internet connectivity: Uses the public internet to establish a flexible migration path that is quick to set up. Requires use of VPC IP Address Manager (IPAM) and Elastic IP addresses.
For detailed analysis, see HCX connectivity options.
Choose your option:
-
Option A: Private connectivity only → Continue to Configure the VPC main route table.
-
Option B: Internet connectivity → Continue to HCX internet connectivity setup.
Note
Skip this section if you chose HCX private connectivity and continue to Configure the VPC main route table.
To enable HCX internet connectivity for Amazon EVS, you must:
-
Ensure that your VPC IP Address Manager (IPAM) quota for Amazon-provided contiguous public IPv4 CIDR block netmask length is /28 or greater.
Important
Use of any Amazon-provided contiguous public IPv4 CIDR block with a netmask length smaller than /28 will result in HCX connectivity issues. For more information about increasing IPAM quotas, see Quotas for your IPAM.
-
Create an IPAM and a public IPv4 IPAM pool with a CIDR that has a a minimum netmask length of /28.
-
Allocate at least two Elastic IP addresses (EIPs) from the IPAM pool for the HCX Manager and HCX Interconnect (HCX-IX) appliances. Allocate an additional Elastic IP address for each HCX network appliance that you need to deploy.
-
Add the public IPv4 CIDR block as an additional CIDR to your VPC.
For more information about managing HCX internet connectivity after environment creation, see Configure HCX public internet connectivity.
Create an IPAM
Follow these steps to Create an IPAM.
Note
You can use IPAM Free Tier to create IPAM resources for use with Amazon EVS.
While IPAM itself is free with Free Tier, you are responsible for the costs of other AWS services used in conjunction with IPAM such as NAT gateways and any public IPv4 addresses you use that are beyond the free tier limit.
For more information about IPAM pricing, see the Amazon VPC pricing page
Note
Amazon EVS does not support private IPv6 Global Unicast Address (GUA) CIDRs at this time.
Create a public IPv4 IPAM pool
Follow these steps to create a public IPv4 pool.
Allocate Elastic IP addresses from the IPAM pool
Follow these steps to allocate Elastic IP addresses (EIPs) from the IPAM pool for HCX Service Mesh appliances.
Add the public IPv4 CIDR block from the IPAM pool to the VPC for HCX internet connnectivity
To enable HCX internet connectivity, you must add the public IPv4 CIDR block from the IPAM pool to your VPC as an additional CIDR. Amazon EVS uses this CIDR block to connect VMware HCX to your network. Follow these steps to add the CIDR block to your VPC.
Important
You must manually input the IPv4 CIDR block that you add to your VPC. Amazon EVS does not support use of an IPAM-allocated CIDR block at this time. Use of an IPAM-allocated CIDR block may result in EIP association failure.
Configure the VPC main route table
Amazon EVS VLAN subnets are implicitly associated to the VPC main route table. To enable connectivity to dependent services such as DNS or on-premises systems for successful environment deployment, you must configure the main route table to allow traffic to these systems. The main route table must include a route for the VPC’s CIDR. Use of the main route table is only required for initial Amazon EVS environment deployment. After environment deployment, you can configure your environment to use a custom route table. For more information, see Configure a custom route table for Amazon EVS subnets.
After environment deployment, you must explicitly associate each of the Amazon EVS VLAN subnets with a route table in your VPC. NSX connectivity fails if your VLAN subnets are not explicitly associated with a VPC route table. We strongly recommend that you explicitly associate your subnets with a custom route table after environment deployment. For more information, see Configure the VPC main route table.
Important
Amazon EVS supports the use of a custom route table only after the Amazon EVS environment is created. Custom route tables should not be used during Amazon EVS environment creation, as this may result in connectivity issues.
Configure DNS and NTP servers using the VPC DHCP option set
Important
Your environment deployment fails if you don’t meet these Amazon EVS requirements:
-
Include a primary DNS server IP address and a secondary DNS server IP address in the DHCP option set.
-
Include a DNS forward lookup zone with A records for each VCF management appliance and Amazon EVS host in your deployment.
-
Include a DNS reverse lookup zone with PTR records for each VCF management appliance and Amazon EVS host in your deployment.
-
Configure the VPC’s main route table to ensure a route to your DNS servers exist.
-
Ensure that your domain name registration is valid and unexpired, and no duplicate hostnames or IP addresses exist.
-
Configure your security groups and network access control lists (ACLs) to allow Amazon EVS to communicate with:
-
DNS servers over TCP/UDP port 53.
-
Host management VLAN subnet over HTTPS and SSH.
-
Management VLAN subnet over HTTPS and SSH.
-
Amazon EVS uses your VPC’s DHCP option set to retrieve the following:
-
Domain Name System (DNS) servers for host IP address resolution.
-
Domain names for DNS resolution.
-
Network Time Protocol (NTP) servers for time synchronization.
You can create a DHCP option set using the Amazon VPC console or AWS CLI. For more information, see Create a DHCP option set in the Amazon VPC User Guide.
Configure DNS servers
DNS configuration enables hostname resolution in your Amazon EVS environment. To successfully deploy an Amazon EVS environment, your VPC’s DHCP option set must have the following DNS settings:
-
A primary DNS server IP address and a secondary DNS server IP address in the DHCP option set.
-
A DNS forward lookup zone with A records for each VCF management appliance and Amazon EVS host in your deployment.
-
A reverse lookup zone with PTR records for each VCF management appliance and Amazon EVS host in your deployment. For NTP configuration, you can use the the default Amazon NTP address
169.254.169.123, or another IPv4 address that you prefer.
For more information about configuring DNS servers in a DHCP option set, see Create a DHCP option set.
Configure DNS for on-premisis connectivity
For on-premises connectivity, we recommend the use of Route 53 private hosted zones with inbound resolvers. This setup enables hybrid DNS resolution, where you can use Route 53 for internal DNS within your VPC and integrate it with your existing on-premises DNS infrastructure. This allows resources within your VPC to resolve domain names hosted on your on-premises network, and vice versa, without requiring complex configurations. If required, you can also use your own DNS server with Route 53 outbound resolvers. For steps to configure, see Creating a private hosted zone and Forwarding inbound DNS queries to your VPC in the Amazon Route 53 Developer Guide.
Note
Using both Route 53 and a custom Domain Name System (DNS) server in the DHCP option set may cause unexpected behavior.
Note
If you use custom DNS domain names defined in a private hosted zone in Route 53, or use private DNS with interface VPC endpoints (AWS PrivateLink), you must set both the enableDnsHostnames and enableDnsSupport attributes to true.
For more information, see DNS attributes for your VPC.
Troubleshoot DNS reachability issues
Amazon EVS requires a persistent connection to SDDC Manager and DNS servers in your VPC’s DHCP option set to reach DNS records. If the persistent connection to SDDC Manager becomes unavailable, Amazon EVS will no longer be able to validate environment status, and you may lose environment access. For steps to troubleshoot this issue, see Reachability check failed.
Configure NTP servers
NTP servers provide the time to your network. A consistent and accurate time reference on your Amazon EC2 instance is crucial for many VCF environment tasks and processes. Time synchronization is essential for:
-
System logging and auditing
-
Security operations
-
Distributed system management
-
Troubleshooting
You can enter the IPv4 addresses of up to four NTP servers in your VPC’s DHCP option set.
You can specify the Amazon Time Sync Service at IPv4 address 169.254.169.123.
By default, the Amazon EC2 instances that Amazon EVS deploys use the Amazon Time Sync Service at IPv4 address 169.254.169.123.
For more information about NTP servers, see RFC 2123
To configure NTP settings
-
Choose your NTP source:
-
Amazon Time Sync Service (recommended)
-
Custom NTP servers
-
-
Add NTP servers to your DHCP options set. For more information, see Create a DHCP option set in the Amazon VPC User Guide.
-
Verify time synchronization. For more information about DHCP option set configuration, see Configure your VPC’s DHCP option set.
You can configure connectivity for your on-premises data center to your AWS infrastructure using Direct Connect with an associated transit gateway, or using an AWS Site-to-Site VPN attachment to a transit gateway.
To enable connectivity to on-premises systems for successful environment deployment, you must configure the VPC’s main route table to allow traffic to these systems. For more information, see Configure the VPC main route table.
After the Amazon EVS environment is created, you must update the transit gateway route tables with the VPC CIDRs created within the Amazon EVS environment. For more information, see Configure transit gateway route tables and Direct Connect prefixes for on-premises connectivity (optional).
For more information about setting up an Direct Connect connection, see Direct Connect gateways and transit gateway associations. For more information about using AWS Site-to-Site VPN with AWS Transit Gateway, see AWS Site-to-Site VPN attachments in Amazon VPC Transit Gateways in the Amazon VPC Transit Gateway User Guide.
Note
Amazon EVS does not support connectivity via an AWS Direct Connect private virtual interface (VIF), or via an AWS Site-to-Site VPN connection that terminates directly into the underlay VPC.
Set up a VPC Route Server instance with endpoints and peers
Amazon EVS uses Amazon VPC Route Server to to enable BGP-based dynamic routing to your VPC underlay network. You must specify a route server that shares routes to at least two route server endpoints in the service access subnet. The peer ASN configured on the route server peers must match, and the peer IP addresses must be unique.
If you are configuring Route Server for HCX internet connectivity, you must configure Route Server propagations for both the service access subnet and public subnet that you created in the first step of this procedure.
Important
Your environment deployment fails if you don’t meet these Amazon EVS requirements for VPC Route Server configuration:
-
You must configure at least two route server endpoints in the service access subnet.
-
When configuring Border Gateway Protocol (BGP) for the Tier-0 gateway, the VPC Route Server peer ASN value must match the NSX Edge peer ASN value.
-
When creating the two route server peers, you must use a unique IP address from the NSX uplink VLAN for each endpoint. These two IP addresses will be assigned to the NSX edges during Amazon EVS environment deployment.
-
When enabling Route Server propagation, you must ensure that all route tables being propagated have at least one explicit subnet association. BGP route advertisement fails if propagated route tables do not have an explicit subnet association.
Note
The NSX uplink VLAN subnet does not exist yet when you create the route server peers — Amazon EVS creates it during environment creation. Choose the two peer IP addresses from the planned NSX uplink VLAN CIDR block (the value you will pass as initialVlans.nsxUplink). The two IP addresses must fall within that planned CIDR block and remain unused.
For more information about setting up VPC Route Server, see the Route Server get started tutorial.
When you follow that tutorial, use the following Amazon EVS-specific values:
-
Amazon-side ASN — the BGP ASN of the VPC Route Server. Use any private ASN (for example,
65022). The NSX Edge Tier-0 gateway uses this value as its BGP neighbor (remote) ASN in Step 5: Configure NSX networking, so note the value you choose. -
Route server endpoints — create two endpoints, both in the service access subnet.
-
Route server peers — We recommend that you create four route server peers and configure a full mesh, so that each of the two NSX Edge nodes peers with both route server endpoints. A full mesh keeps routes propagating if a route server endpoint goes into maintenance. At minimum, you must create two peers. Use a unique IP address from your planned NSX uplink VLAN CIDR for each peer, and set the peer ASN to the NSX Edge Tier-0 ASN that you configure in Step 5: Configure NSX networking (for example,
65000). -
Propagation — enable route server propagation on the route table associated with your service access subnet. That route table must have at least one explicit subnet association.
Important
When enabling Route Server propagation, ensure that all route tables being propagated have at least one explicit subnet association. BGP route advertisement fails if the route table does not have an explicit subnet association.
Note
For Route Server peer liveness detection, Amazon EVS only support the default BGP keepalive mechanism. Amazon EVS does not support multi-hop Bidirectional Forwarding Detection (BFD).
Note
We recommend that you enable persistent routes for the route server instance with a persist duration between 1-5 minutes. If enabled, routes will be preserved in the route server’s routing database even if all BGP sessions end. For more information, see Create a route server in the Amazon VPC User Guide.
Note
If you are using a NAT gateway or a transit gateway, ensure that your route server is configured correctly to propagate NSX routes to the VPC route table(s).
Troubleshooting
If you encounter issues:
-
Verify that each route table has an explicit subnet association.
-
Check that the peer ASN values entered for route server and the NSX Tier-0 gateway match.
-
Confirm that Route Server endpoint IP addresses are unique.
-
Review route propagation status in your route tables.
-
Use VPC Route Server peer logging to monitor BGP session health and troubleshoot connection issues. For more information, see Route server peer logging in the Amazon VPC User Guide.
Create a network ACL to control Amazon EVS VLAN subnet traffic
Amazon EVS uses a network access control list (ACL) to control traffic to and from Amazon EVS VLAN subnets. You can use the default network ACL for your VPC, or you can create a custom network ACL for your VPC with rules that are similar to the rules for your security groups to add a layer of security to your VPC. For more information, see Create a network ACL for your VPC in the Amazon VPC User Guide.
If you plan to configure HCX internet connectivity, ensure that the network ACL rules that you configure allow the necessary inbound and outbound connections for HCX components.
For more information about HCX port requirements, see the VMware HCX User Guide
Important
If you are connecting over the internet, associating an Elastic IP address with a VLAN provides direct internet access to all resources on that VLAN subnet. Ensure that you have appropriate network access control lists configured to restrict access as needed for your security requirements.
Important
EC2 security groups do not function on elastic network interfaces that are attached to Amazon EVS VLAN subnets. To control traffic to and from Amazon EVS VLAN subnets, you must use a network access control list.
Creating an Amazon EVS environment with Self-deployed mode
Amazon EVS supports a Self-deployed mode that provides you full control over your VCF deployment using the VCF Installer or your preferred Infrastructure as Code solutions to automate the deployment.
For example scripts that automate your VCF deployment, see the Solutions for Amazon EVS
For the VCF versions currently supported in Self-deployed mode, see VCF versions and EC2 instance types provided by Amazon EVS.
Overview
In Self-deployed mode, you create an Amazon EVS environment, add hosts, and then install and configure VCF yourself. Amazon EVS provisions the AWS networking and VLAN subnets; you deploy VCF with the VCF Installer (or your own IaC) and connect it back to Amazon EVS with connectors.
Before you begin, complete the AWS networking and account prerequisites for your environment. For more information, see Setting up Amazon Elastic VMware Service and the prerequisite checklist in Amazon EVS deployment prerequisite checklist.
Then complete these steps in order:
-
Create the environment — Amazon EVS provisions your VLAN subnets.
-
Create DNS records — Create A and PTR records for your ESX hosts and VCF management appliances.
-
Add hosts — Add bare-metal EC2 hosts to your environment.
-
Install VCF — Install VCF on your hosts using the VCF Installer.
-
Configure NSX networking — Create your overlay networks on the NSX Edges and configure routing to your VPC.
-
Create connectors — Create connectors so that Amazon EVS can monitor your deployment and report license usage.
-
Verify your environment — Confirm that your hosts, management appliances, and connectors are healthy.
Billing
After you add hosts to your environment, you accrue AWS charges for the EC2 bare-metal instances as you would for any other EC2 instance, regardless of whether you have installed VCF on them yet.
If you have created an environment in Self-deployed mode but not yet added hosts or installed VCF, AWS may reach out to you using the email address associated with your AWS account, requesting that you either complete setup or remove the environment.
To stop accruing charges for hosts that you are no longer using, delete those hosts. For more information, see Clean up an Amazon EVS environment with Self-deployed mode.
Step 1: Create the environment
In Self-deployed mode, environment creation provisions the Amazon EVS VLAN subnets that you specify. It does not deploy VCF or create hosts.
Example
When the environment reaches the CREATED state, you can proceed to Step 2.
Step 2: Create DNS records
Before you add hosts and install VCF, create forward (A record) and reverse (PTR record) DNS entries for each ESX host and for each VCF management appliance you plan to deploy. Amazon EVS performs a DNS lookup of each host’s fully qualified domain name (FQDN) during host creation, and host creation fails if the records do not already exist.
The host FQDN is <hostName>.<domain>, where <hostName> is the name you will pass to CreateEnvironmentHost and <domain> is the domain name configured in your VPC’s DHCP option set (see Configure DNS and NTP servers using the VPC DHCP option set).
Host records must:
-
Use the A record IP address within the host management (vmkManagement) VLAN CIDR that you specified in
initialVlans. Amazon EVS assigns each host its management IP from the A record that you create. -
Have a matching PTR record in your reverse lookup zone.
-
Be resolvable through DNS from the Amazon EVS service access subnet (both forward and reverse lookups must succeed).
VCF management appliance records (for vCenter Server, NSX Manager, and the other appliances for your VCF version) must:
-
Use FQDNs that match the hostnames you will configure during VCF installation.
-
Fall within the IP address range of the appropriate VLAN subnet.
-
Resolve through DNS from the Amazon EVS management VLAN and from any network from which you reach the VCF management appliances.
Important
Create the A and PTR records for every host before you run CreateEnvironmentHost for that host. If the records do not resolve, the host transitions to CREATE_FAILED.
For more information about DNS configuration for Amazon EVS, see Configure DNS and NTP servers using the VPC DHCP option set.
Step 3: Add hosts to your environment
Add enough hosts to serve your target VCF version topology.
For minimum host counts, including vSAN requirements, see the VMware Cloud Foundation documentation
All hosts in a VCF cluster must use the same instance type.
For the list of ESX versions or instance types available to your account, see VCF versions and EC2 instance types provided by Amazon EVS or run aws evs get-versions.
If you do not specify --esx-version, Amazon EVS uses the current default ESX version for Self-deployed mode, which is reported as defaultEsxVersion by aws evs get-versions.
To use a specific version such as ESX 9.0.2 or later, pass --esx-version explicitly.
Confirm that the ESX version you choose is compatible with your VCF release in the Broadcom Interoperability Matrix
The following example adds a host running ESX 9.0.2 to an Amazon EVS environment.
aws evs create-environment-host \ --environment-id env-0123456789abcdef0 \ --esx-version ESXi-9.0.2.0.25148076 \ --host '{ "hostName": "esx01", "keyName": "my-ec2-key-pair", "instanceType": "i4i.metal" }' \ --region us-west-2
Repeat this command for each host that your VCF topology requires.
Step 4: Install VCF on your hosts
After your hosts are in the CREATED state and your DNS records resolve, install VCF using the VMware Cloud Foundation Installer.
Follow the installation guidance for your target VCF version in the Broadcom VCF product documentation. See the VMware Cloud Foundation documentation
This section describes the Amazon EVS-specific configuration you provide during installation. The installation mechanics themselves (running the VCF Installer, the bringup workflow) follow Broadcom’s standard VCF process and are documented by Broadcom.
Installation overview
At a high level, installing VCF on your Amazon EVS hosts involves the following. Before you start, review the Amazon EVS network settings that you apply throughout installation.
-
Prepare your ESX hosts by setting the VM management VLAN on every host.
-
Prepare a temporary datastore for the VCF Installer appliance. The vSAN datastore does not exist until bringup completes, so the Installer needs somewhere to run first.
-
Deploy the VCF Installer appliance and download the VCF software using a Broadcom download token.
-
Run VCF bringup , which deploys the VCF management appliances and forms the vSAN datastore.
-
Reclaim the temporary datastore after VCF is fully installed and the Installer appliance is running on vSAN.
You supply Amazon EVS-specific network, storage, and credential settings during bringup. The rest of the process follows Broadcom’s standard VCF installation.
Note
Amazon EVS provides automated procedures that perform this installation end to end, including the Amazon EVS-specific configuration described in this section. For a worked example, see the Solutions for Amazon EVS
Amazon EVS network settings for VCF
Amazon EVS assigns a VLAN ID to each network function in your environment.
To find the VLAN ID for a function, open the Amazon EVS console (Environments → your environment → Networks and connectivity tab), or run aws evs list-environment-vlans and match on the function name (for example, vmManagement).
Use these VLAN IDs when you configure the distributed switch, port groups, and host networking during VCF installation.
| Network function | MTU | Used for |
|---|---|---|
|
Host management (vmkManagement) |
1500 |
ESX host management |
|
VM management (vmManagement) |
1500 |
VCF management appliances (vCenter Server, NSX Manager, and SDDC Manager or VCF Operations) |
|
vMotion |
8500 |
vMotion traffic |
|
vSAN |
8500 |
vSAN storage traffic |
|
Host overlay (vTep) |
8500 |
Host overlay (Geneve) tunnel endpoints |
|
Edge overlay (edgeVTep) |
8500 |
NSX Edge overlay tunnel endpoints |
|
NSX uplink (nsxUplink) |
1500 |
Tier-0 gateway north-south uplink |
Important
Configure jumbo frames (MTU 8500) on the vMotion, vSAN, and overlay (TEP) networks. The management and uplink networks use MTU 1500. The MTU must be consistent across the network path, or vSAN and overlay traffic will fail.
When you configure the management cluster during bringup, also apply these Amazon EVS-specific settings:
-
vSAN — Use vSAN ESA (Express Storage Architecture) with failures-to-tolerate (FTT) set to at least 1.
-
Uplink teaming — Use a failover teaming policy (active uplink with a standby uplink) for the distributed switch port groups, rather than a load-balancing policy.
-
EVC mode — Set the cluster Enhanced vMotion Compatibility (EVC) mode to match your instance type:
INTEL_ICELAKEfori4i.metal, orINTEL_SAPPHIRERAPIDSfori7i.metal-24xl.
Prepare your ESX hosts
On every ESX host in your environment, set the VM Network port group to the VM management VLAN. The VCF management appliances must run on the VM management VLAN, and the VCF Installer migrates host networking to a distributed switch only later, during bringup. You do not need to enable SSH on the hosts.
-
Find the VLAN ID of the VM management network. Amazon EVS assigns a fixed VLAN ID to each network function. Look up the ID for your environment in the Amazon EVS console (Environments → your environment → Networks and connectivity tab), or by running
aws evs list-environment-vlansand matching on thevmManagementfunction. -
On each host, using the VMware Host Client or the vSphere APIs, set the
VM Networkport group to that VM management VLAN ID. On a new ESX host this port group is untagged (VLAN ID0, the host management network).
Prepare a temporary datastore for the VCF Installer
Amazon EVS hosts have no local VMFS datastores, and the vSAN datastore does not exist until bringup completes, so the VCF Installer appliance needs a temporary datastore to run from. Choose one host to run the Installer.
-
In the Amazon EC2 console, create an encrypted Amazon EBS volume in the same Availability Zone as the host you chose. Size it to hold the VCF Installer appliance and the VCF installation bundles — at least 256 GB.
-
Attach the volume to that host.
-
Using the VMware Host Client or the vSphere APIs, create a local VMFS datastore on the attached EBS volume.
Deploy the VCF Installer and download the VCF software
-
Download the VCF Installer OVA for your target VCF version, and generate a Broadcom download token from the Broadcom Support Portal
. You use this token in the VCF Installer to enable the software depot. -
Deploy the VCF Installer OVA onto the local VMFS datastore. Attach it to the
VM Networkport group, set its management IP address to the SDDC Manager address from your DNS plan, and set the appliance password. The VCF Installer appliance becomes SDDC Manager during bringup, so it uses the SDDC Manager address. (On VCF 9.0.x and 9.1.x, VCF Operations is a separate appliance.) -
In the VCF Installer, enable the software depot using your Broadcom download token, then sync the VCF version you want. Syncing pulls that version of the VCF software into the Installer’s local depot.
Note
Enabling the depot and syncing software requires outbound internet access from the Installer. The NAT gateway in your network foundation provides this access. For more information, see Create a VPC with subnets and route tables.
Run VCF bringup
With the software synced, create your VCF deployment specification, validate it, and run the deployment.
-
In the VCF Installer, create the deployment specification for your management domain. Apply the Amazon EVS-specific network, storage, teaming, and validation settings described in this step.
-
Run validation against the specification, and resolve any errors that it reports.
-
Run the deployment.
Note
Bringup is the longest part of the installation and accounts for most of the setup time. When you use vSAN, forming the datastore and deploying the management appliances can take several hours.
Reclaim the temporary datastore
When bringup finishes, the management appliances run on the vSAN datastore, and the temporary VMFS datastore is empty. Unmount the temporary VMFS datastore from the host, then detach and delete the EBS volume to stop accruing storage charges.
VCF appliance passwords
During bringup you set passwords for the VCF management appliances. Each appliance enforces its own password complexity requirements, which are defined by VCF. If an appliance rejects a password, the validation error states the specific requirement that the password must meet.
Bringup validation settings for VCF Installer
Several VCF Installer standard validation checks do not apply to the Amazon EVS network environment and fail unless you turn them off. Adjust the following values in the VCF spec file or the corresponding VCF Installer wizard options so that validation passes:
-
Skip gateway ping validation — Set
skipGatewayPingValidationtotrue. AWS VPC gateways do not respond to ICMP, so the gateway reachability check fails on Amazon EVS even when routing is correct. -
Skip ESX thumbprint validation — Set
skipEsxThumbprintValidationtotrue. -
Distributed switch teaming — Set the NSX teaming policy to
FAILOVER_ORDER, consistent with the failover teaming described earlier in this step.
Note
When you run bringup through the VCF Installer wizard, use the wizard to identify and correct spec errors. The wizard surfaces validation problems more clearly than the API, whose errors are less descriptive.
Note
For a validated bringup spec, refer to the Solutions for Amazon EVS
Note
Amazon EVS does not support running ESX outside of a full VCF deployment. VMware workload virtual machines must be deployed onto NSX overlay networks. Attaching a large number of virtual machines directly to the underlay VLAN networks may result in stability and performance issues.
Important
In Self-deployed mode, Amazon EVS does not manage VCF installation. If you have VCF-specific requests, you can use your active VCF subscription entitlements to contact Broadcom directly through the Broadcom Support Portal. For more information about support boundaries, see Troubleshooting.
Step 5: Configure NSX networking
Create your overlay networks using Tier-0/Tier-1 routers on the NSX Edges directly, or by configuring VPCs, a centralized transit gateway, and edge clusters. After the VCF Installer completes bringup, the NSX Manager is operational, but the NSX Edge cluster and Tier-0 gateway are not fully configured for connectivity with the VPC Route Server.
Note
NSX defines its own VPC and transit gateway abstractions, which are different from Amazon VPC and AWS Transit Gateway. In this guide, "VPC" and "transit gateway" refer to the AWS resources unless prefixed with "NSX".
Before you begin, confirm that the following are in place:
-
VCF installation completed successfully (NSX Manager and your VCF management appliance — Operations Manager for VCF 9.0.x and 9.1.x or SDDC Manager for VCF 5.2.x — are accessible).
-
Your VPC Route Server is created with endpoints and peers. For more information, see Set up a VPC Route Server instance with endpoints and peers.
-
You have the two Route Server endpoint IP addresses. Both endpoints are in the service access subnet, which provides redundancy.
-
You choose two private BGP ASNs, which must match the values you configured on the VPC Route Server peers (see Set up a VPC Route Server instance with endpoints and peers):
-
NSX Edge Tier-0 local ASN (for example,
65000) -
VPC Route Server (remote) ASN (for example,
65022)Private ASNs are in the range 64512–65534 (16-bit) or 4200000000–4294967294 (32-bit).
-
Deploy the NSX Edge cluster
-
Log in to the NSX Manager UI (
https://<nsx-manager-fqdn>/). -
Navigate to System → Fabric → Nodes → Edge Transport Nodes.
-
Choose Add Edge VM and configure the following:
-
Name — for example,
edge-node-01. -
Form factor — Large (recommended for production).
-
Host switch — configure with the appropriate uplink profile and transport VLAN.
-
-
Repeat for the second Edge node (
edge-node-02). -
Navigate to System → Fabric → Nodes → Edge Clusters.
-
Choose Add Edge Cluster and add both Edge nodes as members.
Create the Tier-0 gateway
-
In NSX Manager, navigate to Networking → Tier-0 Gateways.
-
Choose Add Tier-0 Gateway and configure the following:
-
Name — for example,
evs-tier0-gw. -
HA mode — Active-Standby with failover mode set to Non-preemptive.
-
Edge cluster — select the Edge cluster that you created.
-
-
Save the Tier-0 gateway.
Note
Ensure both the Tier-0 and Tier-1 gateways have Non-preemptive failover. Non-preemptive is the NSX default and avoids an unnecessary BGP session drop when a recovered Edge node would otherwise fail back to the preferred node.
Configure BGP
-
Select the Tier-0 gateway and expand BGP.
-
Enable BGP and set the Local AS number to the NSX Edge Tier-0 ASN you chose (for example,
65000). -
Under BGP Neighbors, configure peering between the Edge nodes and the VPC Route Server endpoints. We recommend a full mesh: each of the two Edge nodes peers with both Route Server endpoints, for four BGP sessions in total. A full mesh keeps routes propagating if a Route Server endpoint goes into maintenance. At minimum, configure two sessions, with each Edge node peering with one endpoint.
For a full mesh, add four neighbors. Set the Remote AS of every neighbor to the VPC Route Server ASN (for example,
65022) and the Address family to IPv4 Unicast.Neighbor Neighbor address Source interface Edge 1 → endpoint 1
<route-server-endpoint-1-ip>Uplink interface on Edge node 1
Edge 1 → endpoint 2
<route-server-endpoint-2-ip>Uplink interface on Edge node 1
Edge 2 → endpoint 1
<route-server-endpoint-1-ip>Uplink interface on Edge node 2
Edge 2 → endpoint 2
<route-server-endpoint-2-ip>Uplink interface on Edge node 2
Note
The Edge uplink interfaces and the Route Server endpoints are in different subnets, so these are multihop BGP sessions. Set the BGP multihop limit to at least 2, and ensure the Tier-0 gateway can reach each Route Server endpoint IP address (for example, with a static route to the endpoint by way of the uplink gateway).
-
Choose Save and wait for the BGP sessions to establish.
-
On the Tier-0 gateway, expand Route Re-Distribution and enable redistribution into BGP for the route types that carry your workload networks — for example, Tier-1 Connected (workload segment subnets), Tier-1 NAT, and Tier-1 Static Routes.
Do not redistribute Tier-0 Connected or Tier-0 Static Routes. Redistributing Tier-0 Connected advertises the NSX uplink subnet to AWS, and Tier-0 Static Routes re-advertises the Route Server endpoint host routes — neither is wanted.
-
Apply an outbound route filter so that the Tier-0 gateway advertises only private (RFC 1918) networks to the VPC Route Server. Create an IP prefix list that permits
10.0.0.0/8,172.16.0.0/12, and192.168.0.0/16(including the more-specific routes within them), denies all other prefixes, and apply it as the out-filter on each BGP neighbor.
Verify BGP peering
-
In NSX Manager, navigate to Networking → Tier-0 Gateways → BGP → BGP Neighbors, and confirm that both neighbors show status Established.
-
In the AWS console, navigate to VPC → Route Server → Routes, and confirm that NSX overlay routes appear.
-
In NSX Manager, verify that VPC routes are learned under Routing → Forwarding Table.
The following table lists common BGP peering issues.
| Symptom | Likely cause | Resolution |
|---|---|---|
|
BGP session stuck in |
Firewall or network ACL blocking TCP port 179 |
Verify that your network ACL allows TCP 179 between the Edge uplink IP addresses and the Route Server endpoint IP addresses, and that the security group attached to the Route Server endpoints also allows inbound TCP 179. |
|
BGP session flaps repeatedly |
MTU mismatch on the uplink path |
Ensure that the MTU is consistent along the NSX uplink path. The NSX uplink network uses MTU 1500; the overlay (TEP) networks use MTU 8500. |
|
Routes not appearing in the VPC route table |
Route Server propagation not enabled |
Verify that Route Server propagation is enabled on the target route table. |
|
One-sided peering (only one session is up) |
Edge node connectivity issue |
Verify that both Edge nodes have reachability to both Route Server endpoints. |
After the BGP sessions are established and routes are propagating, proceed to Step 6: Create connectors.
Step 6: Create connectors
After VCF is installed and its management appliances are reachable over your VCF management network, create connectors so that Amazon EVS can monitor your deployment and report license usage. A connector is an Amazon EVS sub-resource that represents a persistent connection from Amazon EVS to a specific VCF management appliance. For more information, see Connector.
Important
Before you create a connector, store the credentials for the target VCF management appliance in AWS Secrets Manager.
Tag the secret and the AWS KMS key that encrypts it with EvsAccess=true.
Without this tag, Amazon EVS cannot access the secret and connector creation fails.
The connector type you create depends on your VCF version: Operations Manager (OPERATIONS_MANAGER) for VCF 9.0.x and 9.1.x, or SDDC Manager (SDDC_MANAGER) for VCF 5.2.x. You can also create a vCenter connector (VCENTER). For the connector types, required secret keys, and descriptions, see Create an Amazon EVS environment connector.
Step 7: Verify the environment
After you have added hosts, installed VCF, and created at least one connector, verify that:
-
Your hosts are in the
CREATEDstate. -
Your VCF management appliances are reachable from the management VLAN.
-
The connectors you created reach the
ACTIVEstate and the Environment status on the Environments page aggregates to healthy.
For guidance on interpreting environment status and connector health, see Monitor your environment's status and resources.
VLAN subnet sizing guidance
Amazon EVS VLAN subnets cannot be modified after environment creation. Size each VLAN based on the number of IP addresses your VCF components consume now and over the lifetime of the environment. Consider the following when sizing:
-
Host management (vmkManagement) VLAN — one IP per host. Plan for the maximum number of hosts you expect in this environment.
-
vMotion, vSAN, VTEP VLANs — one or more IPs per host depending on your VCF configuration.
-
Management VM (vmManagement) VLAN — IPs for the VCF management appliances you plan to deploy: vCenter, NSX Manager cluster, NSX Edge nodes, and SDDC Manager or Operations Manager.
-
Edge VTEP, HCX uplink, NSX uplink VLANs — IPs for NSX Edge uplinks and HCX appliances, if used.
-
Expansion VLANs — reserve space for future features such as NSX Federation.
As a starting point, use /24 for each VLAN unless you have a specific reason to choose otherwise.
VLAN subnets have a minimum size of /28 and a maximum of /24.
Security considerations
In Self-deployed mode, you install and operate the VCF software stack, so you are responsible for its security. AWS secures the underlying AWS infrastructure that Amazon EVS provisions. This split of responsibilities is in addition to the shared responsibility model described in Security in Amazon Elastic VMware Service.
Your side of the shared responsibility model includes:
-
Installing, patching, and upgrading VCF components including vCenter Server, NSX, SDDC Manager or Operations Manager, and ESX.
-
Configuring VCF authentication, role-based access control, and password rotation for all VCF management appliances.
-
Hardening your VCF management network in accordance with Broadcom guidance and your organization’s security requirements.
-
Rotating the secrets in AWS Secrets Manager that Amazon EVS connectors use to access your VCF management appliances.
-
Monitoring your VCF deployment for security events.
-
Maintaining valid VCF licenses in your VCF management appliance. For more information, see VCF subscriptions.
Amazon EVS is responsible for:
-
Securing the Amazon EVS control plane and Amazon EVS-provisioned AWS resources.
-
Encrypting customer credentials that you store in Secrets Manager (via AWS KMS) and restricting service access to those credentials using resource tags.
-
Monitoring the health of the connectors that you create and reporting aggregate environment health.
Troubleshooting
| Symptom | Where to get help |
|---|---|
|
|
Remove the indicated parameter from your request.
These parameters are not supported when |
|
Environment remains in the |
Open a support case with AWS Support. Include the environment ID. |
|
|
Verify the ESX version string using |
|
A host is stuck in |
Open a support case with AWS Support. Include the environment ID and host ID. |
|
VCF Installer fails during VCF deployment. |
Contact AWS Support for any Amazon EVS issue. For VCF-specific requests, you can also contact Broadcom directly using your VCF subscription entitlements. |
|
|
Confirm that your secret and its AWS KMS encryption key are both tagged with |
|
Connector reaches |
Confirm that the appliance FQDN resolves from the Amazon EVS control plane and that the stored credentials are valid. For more information, see Monitor your environment's status and resources. |
Clean up an Amazon EVS environment with Self-deployed mode
When you no longer need your Amazon EVS environment:
-
Delete all connectors. For more information, see Delete an Amazon EVS environment connector.
-
Delete all hosts. For more information, see Delete an Amazon EVS host.
-
Delete the environment. For more information, see Delete the Amazon EVS hosts and environment.
Deleting the environment removes the Amazon EVS VLAN subnets that Amazon EVS created. It does not delete the VPC, VPC Route Server, or other AWS resources that you created outside of Amazon EVS.
Create an Amazon EVS environment
Important
To get started as simply and quickly as possible, this topic includes steps to create an Amazon EVS environment with default settings. Before creating an environment, we recommend that you familiarize yourself with all settings and deploy an environment with the settings that meet your requirements. Environments can only be configured during initial environment creation. Environments cannot be modified after you’ve created them. For an overview of all possible Amazon EVS environment settings, see the Amazon EVS API Reference Guide.
Note
You environment ID will be available to Amazon EVS across all AWS Regions for VCF license compliance needs.
Note
Amazon EVS environments must be deployed into the same Region and Availability Zone as the VPC and VPC subnets.
Complete this step to create an Amazon EVS environment with hosts and VLAN subnets.
Example
Verify Amazon EVS environment creation
Example
Explicitly associate Amazon EVS VLAN subnets to a VPC route table
Explicitly associate each of the Amazon EVS VLAN subnets with a route table in your VPC. This route table is used to allow AWS resources to communicate with virtual machines on NSX network segments, running with Amazon EVS. If you’ve created a public HCX VLAN, be sure to explicitly associate the public HCX VLAN subnet with a public route table in your VPC that routes to an internet gateway.
Example
Follow these steps to associate Elastic IP address (EIPs) from the IPAM pool to the HCX public VLAN for HCX internet connectivity. You are required to associate at least two EIPs for the HCX Manager and HCX Interconnect (HCX-IX) appliances. Associate an additional EIP for each HCX network appliance that you need to deploy. You can have up to 13 EIPs from the IPAM pool associated with the HCX public VLAN.
Important
HCX public internet connectivity fails if you do not associate at least two EIPs from the IPAM pool with an HCX public VLAN subnet.
Note
Amazon EVS only supports associating EIPs with the HCX VLAN at this time.
Note
You cannot associate the first two EIPs or the last EIP from the public IPAM CIDR block with the VLAN subnet. These EIPs are reserved as network, default gateway, and broadcast addresses. Amazon EVS throws a validation error if you attempt to associate these EIPs with the VLAN subnet.
If you are configuring on-premises network connectivity using Direct Connect or AWS Site-to-Site VPN with a transit gateway, you must update the transit gateway route tables with the VPC CIDRs created within the Amazon EVS environment. For more information, see Transit gateway route tables in Amazon VPC Transit Gateways.
If you are using AWS Direct Connect, you may need to also update your Direct Connect prefixes to send and receive updated routes from the VPC. For more information, see Allows prefixes interactions for AWS Direct Connect gateways.
Retrieve VCF credentials and access VCF management appliances
Amazon EVS uses AWS Secrets Manager to create, encrypt, and store managed secrets in your account. These secrets contain the VCF credentials needed to install and access VCF management appliances such as vCenter Server, NSX, and SDDC Manager, as well as the ESX root password. For more information about retrieving secrets, see Get secrets from AWS Secrets Manager in the AWS Secrets Manager User Guide.
Note
Amazon EVS does not provide managed rotation of your secrets. We recommend that you rotate your secrets regularly on a set rotation window to ensure that secrets are not long-lived.
After you have retrieved your VCF credentials from AWS Secrets Manager, you can use them to log into your VCF management appliances.
For more information, see Log in to the SDDC Manager User Interface
By default, Amazon EVS enables the ESX Shell on newly deployed Amazon EVS hosts. This configuration allows access to the Amazon EC2 instance’s serial port through the EC2 serial console, which you can use to troubleshoot boot, network configuration, and other issues. The serial console does not require your instance to have any networking capabilities. With the serial console, you can enter commands to a running EC2 instance as if your keyboard and monitor are directly attached to the instance’s serial port.
The EC2 serial console can be accessed using the EC2 console or the AWS CLI. For more information, see EC2 Serial Console for instances in the Amazon EC2 User Guide.
Note
The EC2 serial console is the only Amazon EVS supported mechanism to access the Direct Console User Interface (DCUI) to interact with an ESX host locally.
Note
Amazon EVS disables remote SSH by default.
For more information about enabling SSH to access the remote ESX Shell, see Remote ESX Shell Access with SSH
Connect to the EC2 Serial Console
To connect to the EC2 serial console and use your chosen tool for troubleshooting, certain prerequisite tasks must be completed. For more information, see Prerequisites for the EC2 Serial Console and Connect to the EC2 Serial Console in the Amazon EC2 User Guide.
Note
To connect to the EC2 serial console, your EC2 instance state must be running.
You can’t connect to the serial console if the instance is in the pending, stopping, stopped, shutting-down, or terminated state.
For more information about instance state changes, see Amazon EC2 instance state change in the Amazon EC2 User Guide.
Configure access to the EC2 Serial Console
To configure access to the EC2 serial console, you or your administrator must grant serial console access at the account level and then configure IAM policies to grant access to your users. For Linux instances, you must also configure a password-based user on every instance so that your users can use the serial console for troubleshooting. For more information, see Configure access to the EC2 Serial Console in the Amazon EC2 User Guide.
Clean up
Follow these steps to delete the AWS resources that were created.
Delete the Amazon EVS hosts and environment
Follow these steps to delete the Amazon EVS hosts and environment. This action deletes the VMware VCF installation that runs in your Amazon EVS environment.
Note
To delete an Amazon EVS environment, you must first delete all hosts within the environment. An environment cannot be deleted if there are hosts associated with the environment.
Example
If you’ve configured HCX internet connectivity, follow these steps to delete your IPAM resources.
-
Release EIP allocations from the public IPAM pool. For more information, see Release an allocation in the VPC IP Address Manager User Guide.
-
Deprovision the public IPv4 CIDR from the IPAM pool. For more information, see Deprovision CIDRs from a pool in the VPC IP Address Manager User Guide.
-
Delete the public IPAM pool. For more information, see Delete a pool in the VPC IP Address Manager User Guide.
-
Delete the IPAM. For more information, see Delete an IPAM in the VPC IP Address Manager User Guide.
Delete the VPC Route Server components
For steps to delete the Amazon VPC Route Server components that you created, see Route Server cleanup in the Amazon VPC User Guide.
Delete the network access control list (ACL)
For steps to delete a network access control list, see Delete a network ACL for your VPC in the Amazon VPC User Guide.
Disassociate and delete subnet route tables
For steps to disassociate and delete subnet route tables, see Subnet route tables in the Amazon VPC User Guide.
Delete subnets
Delete the VPC subnets, including the service access subnet. For steps to delete VPC subnets, see Delete a subnet in the Amazon VPC User Guide.
Note
If you’re using Route 53 for DNS, remove the inbound endpoints before you attempt to delete the service access subnet. Otherwise, you will not be able to delete the service access subnet.
Note
Amazon EVS deletes the VLAN subnets on your behalf when the environment is deleted. Amazon EVS VLAN subnets can only be deleted when the environment is deleted.
Delete the VPC
For steps to delete the VPC, see Delete your VPC in the Amazon VPC User Guide.
Next steps
Migrate your workloads to Amazon EVS using VMware Hybrid Cloud Extension (VMware HCX). For more information, see Migrate workloads to Amazon EVS using VMware HCX.