Troubleshooting TLS inspection in AWS Network Firewall

Use the information here to help you diagnose common issues that you might encounter when working with TLS inspection in Network Firewall.

Outbound TLS - Blocked connections to servers with revoked certificates

If you've enabled the certificate revocation status check for your outbound TLS workflow, and your client starts getting blocked connections to a specific target server, use one of the following procedures to determine whether the firewall has blocked the connection based on the certificate revocation check action. For more information about actions, see Checking certificate revocation status.

The check differs depending on whether you have TLS logging enabled.

Outbound TLS - Passing traffic for specific target server with revoked certificates by adjusting scope

You can pass traffic to hosts with revoked certificate by updating the scope configuration in your TLS inspection configuration. Because a service endpoint hostname might resolve and be served using more than one IP address, it's important to ensure that the list of destinations contain all of the CIDRs corresponding to the service being investigated. In the following example, the server name with the revoked certificate is revoked-rsa-dv.ssl.com with IP address 54.92.160.72/32, and is associated with the scope configuration on a TLS configuration as follows:

{
  ...
    "Scopes": [{
      "Sources": [{"AddressDefinition": "0.0.0.0/0"}],
      "Destinations": [{"AddressDefinition": "54.92.160.72/32"}],
      "SourcePorts": [{"FromPort": 0, "ToPort": 65535}],
      "DestinationPorts": [{"FromPort": 443, "ToPort": 443}],
      "Protocols": [6]
    },{
      "Sources": [{"AddressDefinition": "0.0.0.0/0"}],
      "Destinations": [{"AddressDefinition": "55.0.0.1/24"}],
      "SourcePorts": [{"FromPort": 0, "ToPort": 65535}],
      "DestinationPorts": [{"FromPort": 443, "ToPort": 443}],
      "Protocols": [6]
    }]
  ...
}

When you update the scope configuration to not send traffic for that target through TLS inspection, you can exclusively pass that traffic for the revoked target though the stateful engine without changing to a global Pass action for all revocation check failures. This keeps the traffic in the remaining scopes, flowing through the TLS inspection, without the existing Drop or Reject actions.

{
  ...
    "Scopes": [{
      "Sources": [{"AddressDefinition": "0.0.0.0/0"}],
      "Destinations": [{"AddressDefinition": "55.0.0.1/24"}],
      "SourcePorts": [{"FromPort": 0, "ToPort": 65535}],
      "DestinationPorts": [{"FromPort": 443, "ToPort": 443}],
      "Protocols": [6]
    }]
  ...
}

For information about scope configurations, see TLS inspection configuration settings in AWS Network Firewall.

You can verify your scope configuration changes using the alert or TLS logs. For information about logging, see Logging and monitoring in AWS Network Firewall.

Verify with TLS logs

Review the TLS logs to verify that the servers you've removed from the scope of TLS inspection are no longer showing up. TLS logging reports revocation check failures for outbound traffic and SNI mismatches and naming errors.

Verify with alert logs

Add an alert rule to match the SNI of the revoked certificate, like in the following example.

Example rule:

alert tls any any -> 54.92.160.72 443 (msg:"revoked cert rule match"; tls.sni; content:"revoked-rsa-dv.ssl.com"; ssl_state:client_hello; sid:1;)

Example alert log:

{
  "firewall_name": "test-network-firewall",
  ...
  "event": {
    "timestamp": "2023-10-25T15:14:11.010637+0000",
    "app_proto": "tls",
    "proto": "TCP",
    "src_ip": "10.0.2.230",
    "src_port": 55784,
    "dest_ip": "54.92.160.72",
    "dest_port": 443,
    "event_type": "alert",
    "alert": {
      "severity": 1,
      "signature_id": 1,
      "signature": "revoked cert rule match",
      ...
    },
    "tls": {
      "subject": "CN=revoked-rsa-dv.ssl.com",
      "issuerdn": "C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com Root Certification Authority RSA",
       ...
    },
    ...
  }
}

Troubleshooting connection issues with AWS service endpoints (including the AWS Systems Manager agent)

When TLS outbound inspection is enabled, depending on the firewall's VPC architecture, instances behind a firewall might be unable to connect to some public AWS service endpoints (including Systems Manager). This can occur for the following common reasons.

Troubleshooting TLS - Connections dropping or resetting

When TLS inspection is enabled, if you experience connection drops, it's possible that the cause is TLS errors. You can use the TLS logs to determine if this is the case and to troubleshoot the problems that are logged. To do this, enable TLS logs and then check the logs for TLS error entries.

In the following example TLS log entry, the error is in the client hello.

{
    "firewall_name": "firewall-tls-test",
    "availability_zone": "us-east-1d",
    "event_timestamp": 1719943304,
    "event": {
        "timestamp": "2024-07-02T18:01:44.412778Z",
        "src_ip": "10.0.2.53",
        "src_port": "59844",
        "dest_ip": "10.0.1.27",
        "dest_port": "443",
        "sni": "-",
        "tls_error": {
            "error_message": "Server name is not found in client hello."
        }
    }
} 

For information about enabling TLS logging, see Logging network traffic from AWS Network Firewall.

For more information about the logs for this type of error, see Logging for TLS inspection in AWS Network Firewall.