# Connecting HealthOmics workflows to a VPC
With Amazon Virtual Private Cloud (Amazon VPC), you can launch AWS resources in a private, virtual network that you've defined. You can give your HealthOmics workflows access to resources in your VPC by configuring your runs to use VPC networking mode. When VPC networking is enabled, your runs can access resources within your VPC and connect to external resources over the public internet if your VPC has internet access configured.
**Note**
Every HealthOmics workflow run executes inside a VPC that is owned and managed by the HealthOmics service. These VPCs are maintained automatically and are not visible to customers. Configuring your run to access resources in your Amazon VPC has no effect on the HealthOmics-managed VPC.
## When to use VPC networking
Use VPC networking when your runs need to:
+ Access publicly available datasets over the internet (for example, NIH datasets, academic repositories)
+ Connect to third-party license servers or external APIs
+ Read or write data from Amazon S3 buckets in other AWS Regions
+ Access on-premises resources in your private network
+ Connect to AWS resources within your VPC
**Note**
When you connect a run to a VPC, it can only access resources available within that VPC. To give your run access to the internet, you must also configure your VPC for internet access. For more information, see [Internet access for VPC-connected workflows](workflows-vpc-internet.md).
**Topics**
+ [When to use VPC networking](#vpc-when-to-use)
+ [Networking modes](#vpc-networking-modes)
+ [Getting started](#vpc-getting-started)
+ [VPC requirements](#vpc-requirements)
+ [Service-linked role for VPC networking](vpc-service-linked-role.md)
+ [Configuration APIs](#vpc-configuration-apis)
+ [Running workflows with VPC networking](#vpc-running-workflows)
+ [Troubleshooting VPC networking](vpc-troubleshooting-guide.md)
+ [Best practices](#vpc-best-practices)
+ [VPC networking quotas](#vpc-quotas)
## Networking modes
HealthOmics Workflows supports two networking modes. By default, workflow runs operate in RESTRICTED mode. You can enable VPC networking on a per-run basis when you start the workflow run.
**RESTRICTED (default)**
Runs can only access Amazon S3 and Amazon ECR resources within the same AWS Region. Runs cannot access other AWS services, resources across AWS Regions, or the public internet.
**VPC**
Run traffic is routed through elastic network interfaces (ENIs) provisioned by HealthOmics in your VPC subnets. You control network routing, security groups, network ACLs, and internet access through NAT gateways. This mode enables access to:
+ Public internet resources (requires NAT Gateway configuration)
+ AWS services in other Regions
+ Private resources in your VPC
+ Access on-premises resources in your private network
You specify the networking mode when you start a workflow run using the `networkingMode` parameter in the `StartRun` API.
## Getting started
This section guides you through setting up VPC networking for HealthOmics Workflows for the first time.
### Prerequisites
Before configuring VPC networking for HealthOmics Workflows, ensure that you have the following:
+ An existing VPC with appropriate subnets and security groups. The VPC must be in the same Region as your workflows.
+ At least one subnet in an Availability Zone where HealthOmics operates in your Region.
+ Appropriate IAM permissions to create and manage HealthOmics configurations.
+ Understanding of VPC networking concepts (subnets, security groups, route tables).
+ Sufficient ENI capacity in your AWS account. HealthOmics scales and manages ENIs in your VPC using the service-linked role. The number of ENIs required depends on your workload. Monitor your ENI usage in the Amazon EC2 console to ensure you have sufficient capacity.
**Important**
Your VPC configuration must include at least one subnet in an Availability Zone where HealthOmics operates in your Region to support workflow task placement. When using VPC networking mode, you are responsible for determining whether it is safe and compliant to transfer or use data across AWS Regions.
### Step 1: Create or configure your VPC
Create a VPC with private subnets, security groups, and NAT gateways (if internet access is needed). For detailed step-by-step instructions, see [Internet access for VPC-connected workflows](workflows-vpc-internet.md).
### Step 2: Configure security groups
Create a security group that allows outbound traffic to the destinations your runs need to access. Configure security groups to allow only the minimum required outbound traffic following the principle of least privilege.
For example configurations and detailed guidance, see the security group section in [Internet access for VPC-connected workflows](workflows-vpc-internet.md).
### Step 3: Verify route tables
Ensure your private subnets have routes to a NAT Gateway for internet access. For example route table configurations, see the route table section in [Internet access for VPC-connected workflows](workflows-vpc-internet.md).
**Note**
Connecting a run to a public subnet does not give it internet access or a public IP address. Always use private subnets with NAT Gateway routes for runs requiring internet connectivity.
### Step 4: Create a configuration resource
Create a HealthOmics Configuration resource that defines your VPC networking settings:
```
aws omics create-configuration \
--name my-vpc-config \
--description "VPC configuration for genomics workflows" \
--run-configurations '{
"vpcConfig": {
"securityGroupIds": ["sg-0123456789abcdef0"],
"subnetIds": [
"subnet-0a1b2c3d4e5f6g7h8",
"subnet-1a2b3c4d5e6f7g8h9"
]
}
}' \
--region us-west-2
```
The configuration will transition from `CREATING` to `ACTIVE` status once network resources are provisioned. This takes up to 15 minutes.
### Step 5: Start a workflow run with VPC networking
Once your configuration is `ACTIVE`, start a workflow run with VPC networking enabled:
```
aws omics start-run \
--workflow-id 1234567 \
--role-arn arn:aws:iam::123456789012:role/OmicsWorkflowRole \
--output-uri s3://my-bucket/outputs/ \
--networking-mode VPC \
--configuration-name my-vpc-config \
--region us-west-2
```
### Step 6: Verify connectivity
Monitor your workflow run to verify it can access the required external resources. Check the workflow logs in CloudWatch Logs for connection success or failure messages. For detailed guidance on testing connectivity, see [Testing VPC connectivity](workflows-vpc-internet.md#vpc-testing-connectivity).
## VPC requirements
Your VPC must meet the following requirements:
### Subnet requirements
+ **Minimum:** At least one subnet in an Availability Zone where HealthOmics operates
+ **Maximum:** 16 subnets per configuration
+ **Restriction:** Maximum of one subnet per Availability Zone
+ **Recommendation:** Use private subnets with NAT Gateway routes for runs requiring internet access. While you can specify a single subnet, we recommend using multiple subnets across different Availability Zones for better availability.
### Security group requirements
+ **Minimum:** 1 security group
+ **Maximum:** 5 security groups per configuration
+ **Requirement:** All security groups must belong to the same VPC as the subnets
Security groups control inbound and outbound traffic for your runs.
**Note**
All subnets and security groups must belong to the same VPC.
### Network interface requirements
HealthOmics provisions elastic network interfaces (ENIs) in your VPC to connect runs to your network. Ensure your AWS account has sufficient ENI capacity (default limit: 5,000 ENIs per Region).
ENIs created by HealthOmics are tagged with the following tags:
```
"TagSet": [
{
"Key": "Service",
"Value": "HealthOmics"
},
{
"Key": "eniType",
"Value": "CUSTOMER"
}
]
```
**Important**
Do not modify or delete ENIs created by HealthOmics. Modifying these network interfaces can cause service delays or disruptions to your workflow runs.
# Service-linked role for VPC networking
HealthOmics uses an IAM service-linked role to create and manage elastic network interfaces (ENIs) in your VPC. This role is automatically created when you create your first VPC configuration.
## Service-linked role details
+ **Role Name**: `AWSServiceRoleForHealthOmics`
+ **Managed Policy**: `AWSHealthOmicsServiceLinkedRolePolicy`
+ **Service Principal**: `omics.amazonaws.com`
## Permissions granted
The service-linked role grants HealthOmics the following Amazon EC2 permissions to manage network interfaces in your VPC:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowEC2DescribeActions",
"Effect": "Allow",
"Action": [
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSecurityGroupRules",
"ec2:DescribeVpcs",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeAvailabilityZones"
],
"Resource": "*"
},
{
"Sid": "AllowVpcGetActions",
"Effect": "Allow",
"Action": [
"ec2:GetSecurityGroupsForVpc"
],
"Resource": "arn:aws:ec2:*:*:vpc/*"
},
{
"Sid": "AllowCreateNetworkInterfaceWithTag",
"Effect": "Allow",
"Action": "ec2:CreateNetworkInterface",
"Resource": "arn:aws:ec2:*:*:network-interface/*",
"Condition": {
"StringEquals": {
"aws:RequestTag/Service": "HealthOmics"
}
}
},
{
"Sid": "AllowCreateNetworkInterfaceSubnetSecurityGroup",
"Effect": "Allow",
"Action": "ec2:CreateNetworkInterface",
"Resource": [
"arn:aws:ec2:*:*:subnet/*",
"arn:aws:ec2:*:*:security-group/*"
]
},
{
"Sid": "AllowCreateTags",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": "arn:aws:ec2:*:*:network-interface/*",
"Condition": {
"StringEquals": {
"ec2:CreateAction": "CreateNetworkInterface"
}
}
},
{
"Sid": "AllowDeleteNetworkInterface",
"Effect": "Allow",
"Action": "ec2:DeleteNetworkInterface",
"Resource": "arn:aws:ec2:*:*:network-interface/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/Service": "HealthOmics"
}
}
},
{
"Sid": "AllowAssignUnassignPrivateIpAddresses",
"Effect": "Allow",
"Action": [
"ec2:AssignPrivateIpAddresses",
"ec2:UnassignPrivateIpAddresses"
],
"Resource": "arn:aws:ec2:*:*:network-interface/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/Service": "HealthOmics"
}
}
}
]
}
```
## Creating the service-linked role
The service-linked role is automatically created when you create your first VPC configuration using the `CreateConfiguration` API. You must have the following IAM permission:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/omics.amazonaws.com/AWSServiceRoleForHealthOmics",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": "omics.amazonaws.com"
}
}
}
]
}
```
## Deleting the service-linked role
You can delete the service-linked role only after deleting all VPC configurations in your account. To delete the role:
```
aws iam delete-service-linked-role \
--role-name AWSServiceRoleForHealthOmics
```
If you have active VPC configurations, the deletion will fail with a message indicating which configuration resources must be deleted first.
## Configuration APIs
HealthOmics provides APIs to create, manage, and delete VPC configurations. You can reuse configurations across multiple workflow runs.
**Topics**
+ [CreateConfiguration](#vpc-create-configuration)
+ [GetConfiguration](#vpc-get-configuration)
+ [ListConfigurations](#vpc-list-configurations)
+ [DeleteConfiguration](#vpc-delete-configuration)
### CreateConfiguration
Creates a new configuration resource with VPC networking settings. For a step-by-step example, see [Step 4: Create a configuration resource](#vpc-step-create-configuration).
**Request syntax:**
```
aws omics create-configuration \
--name configuration-name \
--description description \
--run-configurations '{"vpcConfig":{"securityGroupIds":["security-group-id"],"subnetIds":["subnet-id"]}}' \
--tags Key=key,Value=value \
--region region
```
**Parameters:**
+ **name** (required) — A unique name for the configuration (maximum 50 characters).
+ **description** (optional) — A description of the configuration.
+ **run-configurations** (optional) — VPC configuration settings:
+ `vpcConfig.securityGroupIds` — A list of 1–5 security group IDs.
+ `vpcConfig.subnetIds` — A list of 1–16 subnet IDs.
+ **tags** (optional) — Resource tags.
**Response:**
```
{
"arn": "arn:aws:omics:region:account-id:configuration/configuration-name",
"uuid": "configuration-uuid",
"name": "configuration-name",
"runConfigurations": {
"vpcConfig": {
"securityGroupIds": ["security-group-id"],
"subnetIds": ["subnet-id"],
"vpcId": "vpc-id"
}
},
"status": "CREATING",
"creationTime": "timestamp",
"tags": {}
}
```
**Configuration status values:**
+ **CREATING** — The configuration is being created and network resources are being provisioned (up to 15 minutes).
+ **ACTIVE** — The configuration is ready to use.
+ **DELETING** — The configuration is being deleted.
+ **DELETED** — The configuration has been deleted.
### GetConfiguration
Retrieves details of a specific configuration.
**Request syntax:**
```
aws omics get-configuration \
--name configuration-name \
--region region
```
**Response:**
```
{
"arn": "arn:aws:omics:region:account-id:configuration/configuration-name",
"uuid": "configuration-uuid",
"name": "configuration-name",
"runConfigurations": {
"vpcConfig": {
"securityGroupIds": ["security-group-id"],
"subnetIds": ["subnet-id"],
"vpcId": "vpc-id"
}
},
"status": "ACTIVE",
"creationTime": "timestamp",
"tags": {}
}
```
### ListConfigurations
Lists all configurations in your account.
**Request syntax:**
```
aws omics list-configurations \
--region region
```
**Response:**
```
{
"items": [
{
"arn": "arn:aws:omics:region:account-id:configuration/configuration-name",
"name": "configuration-name",
"description": "description",
"status": "ACTIVE",
"creationTime": "timestamp"
}
]
}
```
### DeleteConfiguration
Deletes a configuration. You cannot delete a configuration that is currently in use by active workflow runs.
**Request syntax:**
```
aws omics delete-configuration \
--name configuration-name \
--region region
```
**Note**
The configuration status changes to DELETING while network resources are being cleaned up, and then to DELETED once the process is complete.
## Running workflows with VPC networking
### Starting a run with VPC networking
To use VPC networking in a workflow run, specify the `networking-mode` parameter and the `configuration-name`:
```
aws omics start-run \
--workflow-id 1234567 \
--role-arn arn:aws:iam::123456789012:role/OmicsWorkflowRole \
--output-uri s3://my-bucket/outputs/ \
--networking-mode VPC \
--configuration-name my-vpc-config \
--region us-west-2
```
**Parameters:**
+ **networking-mode** — Set to `VPC` to enable VPC networking. The default is `RESTRICTED`.
+ **configuration-name** (required) — The name of the configuration to use.
### Viewing run network configuration
Use `GetRun` to view the networking configuration for a run:
```
aws omics get-run \
--id run-id \
--region region
```
The response includes the networking mode, configuration details, and VPC configuration. The following example shows the VPC-related fields from the response:
```
{
"arn": "arn:aws:omics:region:account-id:run/run-id",
"id": "run-id",
"status": "status",
"workflowId": "workflow-id",
"networkingMode": "VPC",
"configuration": {
"name": "configuration-name",
"arn": "arn:aws:omics:region:account-id:configuration/configuration-name",
"uuid": "configuration-uuid"
},
"vpcConfig": {
"subnets": ["subnet-id-1", "subnet-id-2"],
"securityGroupIds": ["security-group-id"],
"vpcId": "vpc-id"
}
}
```
### Configuration immutability
Workflows use a snapshot of the configuration as it existed when the run started. You can safely modify or delete configurations during run execution without affecting active runs.
### Call caching considerations
When using VPC networking with call caching, ensure your workflow engine is configured appropriately. For detailed guidance on call caching per engine, see [Engine-specific caching features](workflow-cache-per-engine.md).
**Important**
When connecting to non-deterministic or dynamic resources (for example, third-party databases on the public internet), consider using the cache task opt-out feature in your workflows to avoid caching dynamic datasets that could impact run outputs.
# Troubleshooting VPC networking
**Topics**
+ [Network monitoring and troubleshooting](#vpc-network-monitoring-troubleshooting)
+ [Configuration troubleshooting](#vpc-configuration-troubleshooting)
## Network monitoring and troubleshooting
### CloudTrail logging
All Configuration API operations and workflow runs using VPC networking are logged in CloudTrail. Use CloudTrail to audit configuration changes and track which runs use VPC networking.
### Troubleshooting with ENI flow logs
When your workflow runs access external resources over the internet, you can use VPC Flow Logs to verify connectivity and diagnose issues. HealthOmics provisions elastic network interfaces (ENIs) in your VPC subnets to route traffic from your workflow tasks. By examining flow logs on these ENIs, you can trace network traffic to and from external destinations.
**Cost management for VPC Flow Logs**
VPC Flow Logs can incur significant costs, especially at the VPC level. To minimize costs:
**Delete flow logs after troubleshooting.** Once you've resolved connectivity issues, delete the flow log to stop incurring charges.
**Use Amazon S3 instead of CloudWatch Logs for long-term storage.** Amazon S3 storage is significantly cheaper than CloudWatch Logs. Configure flow logs to publish to Amazon S3 if you need to retain logs for compliance or security analysis.
**Set CloudWatch Logs retention policies.** If using CloudWatch Logs, configure automatic log expiration (for example, 7 days) to prevent indefinite storage costs.
**Use ENI-level flow logs for troubleshooting.** For one-time debugging, create flow logs on the specific customer ENI rather than the entire VPC.
#### Setting up flow logs for troubleshooting
**Option 1: VPC-level flow logs (for ongoing monitoring)**
Enable flow logs on your VPC to automatically capture traffic from all HealthOmics workflow runs. This is best when you have many workflow runs and want comprehensive visibility without tracking individual ENIs.
1. **Enable VPC Flow Logs.** In the Amazon VPC console:
1. Choose **Your VPCs** and select the VPC used in your HealthOmics configuration
1. Choose the **Flow logs** tab
1. Choose **Create flow log**
1. Configure the flow log to capture **All** traffic (both accepted and rejected)
1. Select CloudWatch Logs as the destination for easier querying
1. **Start a workflow run.** Start a workflow run with VPC networking enabled. Note the run ID and start time for filtering flow logs later.
Query flow logs using CloudWatch Logs Insights by time window, destination IP, or traffic patterns. You don't need to identify specific ENI IDs.
**Option 2: ENI-level flow logs (for targeted troubleshooting)**
Enable flow logs on specific ENIs when you have only a few HealthOmics ENIs in your account. This is the most cost-effective approach and makes it easy to isolate traffic for specific workflow runs.
1. **Find the customer ENI.** In the Amazon EC2 console:
1. Choose **Network Interfaces**
1. Filter by tag `Service: HealthOmics` to show only ENIs created by HealthOmics
1. Optionally, further filter by the subnet ID from your HealthOmics configuration
1. Note the ENI ID and private IP address
1. **Enable flow logs on the ENI.**
1. Select the ENI and choose the **Flow logs** tab
1. Choose **Create flow log**
1. Configure the flow log to capture **All** traffic
1. Select CloudWatch Logs as the destination
**Note**
Flow logs only capture traffic from the time they are enabled. For VPC-level flow logs, enable them before running workflows. For ENI-level flow logs, once enabled on an ENI, the same flow log will capture traffic for all future workflow runs that use that ENI.
#### Understanding VPC Flow Log format
VPC Flow Logs use a space-separated format with the following fields:
```
version account_id interface_id srcaddr dstaddr srcport dstport protocol packets bytes start end action log_status
```
**Field descriptions:**
+ **version** — Flow log format version (typically 2)
+ **account\$1id** — Your AWS account ID
+ **interface\$1id** — The ENI ID (for example, eni-0e57c5476efeac402)
+ **srcaddr** — Source IP address
+ **dstaddr** — Destination IP address
+ **srcport** — Source port number
+ **dstport** — Destination port number
+ **protocol** — IANA protocol number (6=TCP, 17=UDP, 1=ICMP)
+ **packets** — Number of packets in the flow
+ **bytes** — Number of bytes in the flow
+ **start** — Flow start time (Unix timestamp)
+ **end** — Flow end time (Unix timestamp)
+ **action** — ACCEPT or REJECT
+ **log\$1status** — OK, NODATA, or SKIPDATA
**Example flow log entries:**
```
2 074296239033 eni-0e57c5476efeac402 10.0.130.58 13.226.238.96 40565 443 6 13 1502 1774338927 1774338929 ACCEPT OK
2 074296239033 eni-0e57c5476efeac402 13.226.238.96 10.0.130.58 443 40565 6 8 1024 1774338928 1774338930 ACCEPT OK
```
These entries show successful bidirectional HTTPS communication. Key IPs: **10.0.130.58** is the customer ENI created by HealthOmics in your account, and **13.226.238.96** is the external public domain your workflow is accessing. The first entry is outbound traffic, and the second is the return traffic. Both show ACCEPT, indicating the traffic was allowed by security groups.
#### Querying flow logs in CloudWatch Logs Insights
When flow logs are published to CloudWatch Logs, use CloudWatch Logs Insights to query and analyze the data.
**Find rejected traffic (start here)**
```
fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter action = "REJECT"
| sort @timestamp desc
```
If this returns results, you may have a connectivity issue. The rejected entries show which traffic is being blocked by security groups or network ACLs.
**Find traffic to a specific external IP**
First, resolve the domain to an IP address using `nslookup` or `dig`:
```
$ nslookup ftp.ncbi.nlm.nih.gov
Server: 127.53.53.53
Address: 127.53.53.53#53
Non-authoritative answer:
ftp.ncbi.nlm.nih.gov canonical name = ftp.wip.ncbi.nlm.nih.gov.
Name: ftp.wip.ncbi.nlm.nih.gov
Address: 130.14.250.10
Name: ftp.wip.ncbi.nlm.nih.gov
Address: 130.14.250.11
```
The "Server" and "Address" at the top are your DNS resolver. The addresses under "Non-authoritative answer" (130.14.250.10 and 130.14.250.11) are the actual IPs for the domain.
Query flow logs using a prefix to match any IP in that range:
```
fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter dstAddr like "130.14.250"
| sort @timestamp desc
```
This matches any IP starting with 130.14.250, capturing traffic to all IPs in that subnet.
**Find HTTPS traffic to external destinations**
```
fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter dstPort = 443 and protocol = 6
| filter not (dstAddr like /^10\./ or dstAddr like /^172\./ or dstAddr like /^192\.168\./)
| sort @timestamp desc
```
The second filter excludes private IP ranges, showing only traffic to external (public) destinations.
**Note**
Protocol numbers: 6=TCP, 17=UDP, 1=ICMP. For load-balanced services (for example, CloudFront), DNS may return different IPs, so filter by destination port instead of IP address.
#### Common flow log patterns and issues
**Outbound traffic rejected**
```
Outbound: 2 074296239033 eni-0e57c5476efeac402 10.0.130.58 13.226.238.96 40565 443 6 1 60 1774338927 1774338929 REJECT OK
```
**Cause:** Security group doesn't allow outbound traffic to the destination port or IP range.
**Solution:** Add an outbound rule to your security group:
+ For HTTPS: Allow TCP port 443 to 0.0.0.0/0
+ For HTTP: Allow TCP port 80 to 0.0.0.0/0
+ For broader access: Allow all TCP/UDP to 0.0.0.0/0
**Return traffic rejected**
```
Outbound: 2 074296239033 eni-0e57c5476efeac402 10.0.130.58 8.8.8.8 54321 53 17 1 64 1774338927 1774338929 ACCEPT OK
Return: 2 074296239033 eni-0e57c5476efeac402 8.8.8.8 10.0.130.58 53 54321 17 1 64 1774338928 1774338930 REJECT OK
```
**Cause:** Network ACL is blocking return traffic. Unlike security groups (stateful), network ACLs are stateless and require explicit rules for both directions.
**Solution:** In the VPC console, check your subnet's network ACL and verify inbound rules allow traffic on ephemeral ports (1024-65535) from external sources. Add rule if needed: Allow TCP/UDP ports 1024-65535 from 0.0.0.0/0
**Missing return traffic**
```
Outbound: 2 074296239033 eni-0e57c5476efeac402 10.0.130.58 8.8.8.8 54321 53 17 1 64 1774338927 1774338929 ACCEPT OK
```
**Cause:** NAT Gateway/Internet Gateway not configured properly, or ENI doesn't have connectivity to internet.
**Solution:**
+ Verify route table has route to NAT Gateway (0.0.0.0/0 → nat-xxxxx)
+ Verify NAT Gateway is in AVAILABLE state with an Elastic IP
+ Check NAT Gateway is in a public subnet with route to Internet Gateway
**No flow log entries for expected traffic**
**Cause:** Traffic not reaching the ENI, or flow logs not configured correctly.
**Solution:**
+ Verify flow logs are enabled and configured to capture all traffic
+ Check workflow logs in CloudWatch Logs to confirm the workflow is attempting to access the external resource
+ Verify route table has route to NAT Gateway (0.0.0.0/0 → nat-xxxxx)
+ Verify NAT Gateway is in AVAILABLE state with an Elastic IP
#### Best practices for flow log troubleshooting
1. **Enable flow logs before starting troubleshooting.** Flow logs only capture traffic from the time they are enabled. Enable them on all subnets in your HealthOmics configuration before running workflows.
1. **Use CloudWatch Logs Insights for analysis.** CloudWatch Logs Insights provides powerful querying capabilities for flow logs. Save commonly used queries for quick access.
1. **Filter by time window.** Narrow your flow log queries to the specific time window when your workflow run was active to reduce noise and improve query performance.
1. **Look for both directions of traffic.** Always verify that both outbound and return traffic show ACCEPT. A connection requires bidirectional communication.
1. **Document your findings.** When troubleshooting connectivity issues, document the customer ENI ID, IP addresses, ports, and flow log entries. This information is valuable for support cases and future troubleshooting.
1. **Test with a simple workflow first.** Before running complex workflows, test connectivity with a simple workflow that attempts to access the external resource and logs the result. This helps isolate network issues from workflow logic issues.
## Configuration troubleshooting
### Configuration stuck in CREATING status
**Cause:** Network resource provisioning can take several minutes.
**Solution:** Wait up to 10 minutes. If the status doesn't change to ACTIVE, check the following:
+ Your subnets and security groups exist and are in the same VPC.
+ You have the required IAM permissions.
+ The service-linked role was created successfully.
### Run fails to start with VPC networking
**Cause:** The configuration might not be ACTIVE, or there might be network connectivity issues.
**Solution:**
+ Verify that the configuration status is ACTIVE by using `GetConfiguration`.
+ Check that security group rules allow the required outbound traffic.
+ Ensure that subnets are in Availability Zones where HealthOmics operates.
### Cannot delete configuration
**Cause:** The configuration is in use by active workflow runs.
**Solution:** Wait for all runs using the configuration to complete, then retry the deletion.
### Cannot delete service-linked role
**Cause:** Active VPC configurations exist in your account.
**Solution:** Delete all VPC configurations first, then delete the service-linked role.
### Workflow cannot connect to external resource
**Cause:** Security group or route table misconfiguration.
**Solution:**
1. Enable VPC Flow Logs to identify rejected packets
1. Check security group outbound rules allow traffic to the destination
1. Verify route table has a route to NAT Gateway (0.0.0.0/0 → nat-xxxxxx)
1. For cross-Region AWS service access, ensure the destination Region is reachable
1. Test connectivity from an Amazon EC2 instance in the same subnet
### Network performance issues
**Symptom:** Slow data transfer or workflow timeouts.
**Cause:** Network throughput limitations or NAT Gateway saturation.
**Solution:**
+ Network throughput begins at 10 Gbps per ENI and scales up to 100 Gbps over a 60-minute period with sustained traffic
+ For workflows with immediate high-throughput requirements, please contact AWS Support
+ Monitor NAT Gateway metrics in CloudWatch to identify saturation
+ Consider deploying additional NAT Gateways in multiple Availability Zones for higher throughput
### Workflow cannot reach the internet
**Cause:** The private subnets might not have a route to a NAT gateway, or security group rules might be blocking outbound traffic.
**Solution:**
+ Verify that the route table for your private subnets includes a route to a NAT gateway (0.0.0.0/0 → nat-xxxxxxxxx).
+ Check that security group rules allow outbound traffic on the required ports.
+ Verify that the NAT gateway is in a public subnet with a route to an internet gateway.
### Workflow run fails with connectivity errors
**Cause:** Network traffic might be blocked or misconfigured.
**Solution:**
1. Verify that the configuration is still in ACTIVE status by using `GetConfiguration`.
1. Create a VPC flow log on the ENIs in your VPC to inspect traffic. For more information, see [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) in the *Amazon VPC User Guide*.
1. Check the flow log for REJECT entries. If you see rejected packets, update your security group rules to allow the required outbound traffic.
1. If the flow log does not reveal a root cause, contact AWS Support.
## Best practices
### Security
1. **Use least-privilege security groups.** Allow only the minimum required outbound traffic. Use specific destination CIDR blocks instead of 0.0.0.0/0 when possible. Document the purpose of each security group rule.
1. **Separate configurations by environment.** Create separate configurations for development, staging, and production. Use different VPCs or subnets for each environment. Apply appropriate tags to configurations for organization.
1. **Implement network monitoring.** Enable VPC Flow Logs for security analysis. Set up CloudWatch alarms for unusual traffic patterns. Regularly review CloudTrail logs for configuration changes.
1. **Use VPC endpoints for AWS services.** Configure VPC endpoints for Amazon S3, Amazon ECR, and other AWS services. This reduces NAT Gateway costs, improves performance, and provides additional security by keeping traffic within the AWS network.
### Performance
1. **Plan for network scaling.** Network throughput starts at 10 Gbps and scales to 100 Gbps over time. For immediate high-throughput needs, plan ahead and request pre-warming. Monitor network metrics to understand your workflow requirements.
1. **Deploy NAT Gateways per Availability Zone.** Use one NAT Gateway per AZ for production workloads. This improves resiliency and throughput, and reduces cross-AZ data transfer costs.
1. **Reuse configurations.** Create configurations that can be shared across multiple workflows. This reduces configuration management overhead and ensures consistent network settings.
1. **Test configurations before production use.** Validate network connectivity with test workflows. Verify security group rules allow required traffic. Test failover scenarios with multi-AZ configurations.
### Cost optimization
1. **Use VPC endpoints instead of NAT Gateway.** For AWS service access, use VPC endpoints (no data processing charges). Amazon S3 Gateway endpoints have no additional costs. Interface endpoints have hourly charges but can be more cost-effective than NAT Gateway.
1. **Monitor data transfer costs.** Data transfer in has no charge. Data transfer out to internet incurs standard AWS data transfer rates. Cross-Region data transfer has higher rates. Use AWS Cost Explorer to track VPC-related costs.
1. **Right-size NAT Gateway deployment.** For development, use one NAT Gateway for all AZs. For production, use one NAT Gateway per AZ for resiliency. Monitor NAT Gateway utilization to avoid over-provisioning.
1. **Delete unused configurations.** Regularly review and delete configurations no longer in use. Use tags to identify configuration ownership and purpose.
### Operational
1. **Use descriptive configuration names.** Include environment, purpose, and team in the name (for example, `prod-genomics-vpc`, `dev-clinical-trials-vpc`).
1. **Tag all configurations.** Use consistent tagging strategy across all resources. Include tags for Environment, Owner, CostCenter, and Purpose.
1. **Document network requirements.** Document which external services each configuration accesses. Maintain a map of security group rules and their purposes. Share network architecture diagrams with your team.
## VPC networking quotas
The following table lists the quotas for VPC networking configurations:
| Resource | Default limit | Adjustable |
| --- | --- | --- |
| Maximum configurations per account | 10 | Yes |
| Maximum security groups per configuration | 5 | No |
| Maximum subnets per configuration | 16 | No |
| Maximum subnets per Availability Zone | 1 | No |
| CreateConfiguration API TPS | 1 | Yes |
| Elastic network interfaces per Region (customer VPC) | 5,000 | Yes |
To request a quota increase, open the [Service Quotas console](https://console.aws.amazon.com/servicequotas/home), choose **AWS services**, search for **AWS HealthOmics**, select the quota you want to increase, and choose **Request quota increase**. Quota increase requests are typically processed within 1-2 business days.