Supported logs and discovered fields

CloudWatch Logs Insights supports different log types. For every log that's sent to a Standard class log group in Amazon CloudWatch Logs, CloudWatch Logs Insights automatically generates five system fields:

CloudWatch Logs Insights inserts the @ symbol at the start of fields that it generates.

For many log types, CloudWatch Logs also automatically discovers the log fields contained in the logs. These automatic discovery fields are shown in the following table.

For other types of logs with fields that CloudWatch Logs Insights doesn't automatically discover, you can use the parse command to extract and create extracted fields for use in that query. For more information, see CloudWatch Logs Insights language query syntax.

If the name of a discovered log field starts with the @ character, CloudWatch Logs Insights displays it with an additional @ appended to the beginning. For example, if a log field name is @example.com, this field name is displayed as @@example.com.

Fields in JSON logs

With CloudWatch Logs Insights, you use dot notation to represent JSON fields. This section contains an example JSON event and code snippet that show how you can access JSON fields using dot notation.

Example: JSON event

{
    "eventVersion": "1.0",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "EX_PRINCIPAL_ID",
        "arn": "arn: aws: iam: : 123456789012: user/Alice",
        "accessKeyId": "EXAMPLE_KEY_ID",
        "accountId": "123456789012",
        "userName": "Alice"
    },
    "eventTime": "2014-03-06T21: 22: 54Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "StartInstances",
    "awsRegion": "us-east-2",
    "sourceIPAddress": "192.0.2.255",
    "userAgent": "ec2-api-tools1.6.12.2",
    "requestParameters": {
        "instancesSet": {
            "items": [
                {
                    "instanceId": "i-abcde123"
                }
            ]
        }
    },
    "responseElements": {
        "instancesSet": {
            "items": [
                {
                    "instanceId": "i-abcde123",
                    "currentState": {
                        "code": 0,
                        "name": "pending"
                    },
                    "previousState": {
                        "code": 80,
                        "name": "stopped"
                    }
                }
            ]
        }
    }
}

The example JSON event contains an object that's named userIdentity. userIdentity contains a field that's named type. To represent value of type using dot notation, you use userIdentity.type.

The example JSON event contains arrays that flatten to lists of nested field names and values. To represent the value of instanceId for the first item in requestParameters.instancesSet, you use requestParameters.instancesSet.items.0.instanceId. The number 0 that's placed before the field instanceID refers to the position of values for the field items. The following example contains a code snippet that shows how you can access nested JSON fields in a JSON log event.

Example: Query

fields @timestamp, @message
| filter requestParameters.instancesSet.items.0.instanceId="i-abcde123"
| sort @timestamp desc

The code snippet shows a query that uses dot notation with the filter command to access the value of the nested JSON field instanceId. The query filters on messages where the value of instanceId equals "i-abcde123" and returns all of the log events that contain the specified value.

Dot notation and JSON-encoded string fields

Dot notation traverses only fields that are stored as structurally nested JSON objects at ingest time. If a field's value is a JSON-encoded string (a string whose content happens to be valid JSON), dot notation treats it as an opaque leaf value and does not access sub-fields within it.

This commonly occurs with:

For example, if api.request.data contains {"startTime":1234,"endTime":5678} as a string value, then api.request.data.startTime returns no results because CloudWatch Logs Insights treats the entire string as a single leaf value rather than a nested object.

To access sub-fields within a JSON-encoded string, use jsonParse to convert the string into a traversable map.

Example: Query using jsonParse for a JSON-encoded string field

fields jsonParse(api.request.data).startTime as startTime

This query uses jsonParse to parse the JSON-encoded string in api.request.data into a map, and then accesses the startTime sub-field with dot notation. For more information about jsonParse, see Structure types.