Improve network performance between EC2 instances with ENA Express

Benefits of ENA Express

• Increases the maximum bandwidth a single flow can use from 5 Gbps up to 25 Gbps within the same Region, up to the aggregate instance limit.

• Reduces tail latency of network traffic between EC2 instances in the same Availability Zone, especially during periods of high network load.

• Detects and avoids congested network paths.

• Handles some tasks directly in the network layer, such as packet reordering on the receiving end, and most retransmits that are needed. This frees up the application layer for other work.

• Processes can benefit from increased maximum single flow bandwidth from 5 Gbps up to 25 Gbps within the same Region, up to the aggregate instance limit. For example, if a specific instance type supports up to 12.5 Gbps, the single flow bandwidth is also limited to 12.5 Gbps.

• Longer running processes in the same Availability Zone will experience reduced tail latency during periods of network congestion.

• Processes can benefit from a smoother and more standard distribution for network response times.

• Both sending and receiving instance types are supported. See the Supported instance types for ENA Express table for more information.

• Both sending and receiving instances must have ENA Express configured. If there are differences in the configuration, you can run into situations where traffic defaults to standard ENA transmission. The following scenario shows what can happen.

  Scenario: Differences in configuration

  Instance	ENA Express Enabled	UDP uses ENA Express
  Instance 1	Yes	Yes
  Instance 2	Yes	No

  In this case, TCP traffic between the two instances can use ENA Express, as both instances have enabled it. However, since one of the instances does not use ENA Express for UDP traffic, communication between these two instances over UDP uses standard ENA transmission.

• The sending and receiving instances must run in the same Region.

• The network path between the instances must not include middleware boxes. ENA Express doesn't currently support middleware boxes.

• (Linux instances only) To utilize full bandwidth potential, use driver version 2.2.9 or higher.

• (Linux instances only) To produce metrics, use driver version 2.8 or higher.

• MTU size – ENA Express requires a lower MTU than the default to accommodate additional AWS SRD headers. Newly established TCP connections automatically clamp the MSS to mitigate this, but UDP traffic still requires a lower MTU.

• TCP output queue size limit – Checks that the per-socket in-flight byte limit is sufficient to sustain high throughput. Environments with increased network latency require a higher limit.

• Byte queue limit – Confirms that the byte queue limit (BQL) is disabled on the network interface. BQL can restrict the amount of data queued for device-level transmission, which limits ENA Express performance.

  Note

  The ENA driver for the Amazon Linux distribution disables byte queue limits by default.

• TCP autocorking – Checks whether TCP autocorking is disabled. Disabling autocorking can reduce latency for certain ENA Express TCP traffic patterns, such as request-response workloads. This might result in a minimal increase in packet processing overhead.

• TX queue size and Large LLQ – Verifies that the transmit queue size for the network interface is large enough for optimal performance. The script also checks whether the ENA module parameter explicitly disables the Large Low Latency Queue (Large LLQ) feature, as it can reduce the available TX queue depth. For more information about Large LLQ and its impact on TX queue size, see Large Low Latency Queue (Large LLQ) on GitHub.

• RX queue size – Checks that the receive ring buffer for the network interface is large enough to handle incoming traffic efficiently and avoid packet drops under load.

• TCP and network socket buffer sizes – Validates that the TCP receive and send buffer maximum sizes, as well as the core network socket buffer defaults and maximums, are large enough to sustain high throughput. These settings are important in environments with increased network latency, where you need larger buffers to utilize the connection.

• TCP congestion control – Verifies that the TCP congestion control configuration is optimized for use with ENA Express in environments with increased network latency.