OpenSearch PPL language - Amazon CloudWatch Logs

OpenSearch PPL language

This section contains a basic introduction to querying CloudWatch Logs using OpenSearch PPL. With PPL, you can retrieve, query, and analyze data using piped-together commands, making it easier to understand and compose complex queries. Its syntax is based on Unix pipes, and enables chaining of commands to transform and process data. With PPL, you can filter and aggregate data, and use a rich set of math, string, date, conditional, and other functions for analysis.

You can use OpenSearch PPL only for queries of log groups in the Standard Log Class.

For information about all OpenSearch PPL query commands supported in CloudWatch Logs and detailed information about syntax and restrictions, see Supported PPL commands in the OpenSearch Service Developer Guide.

Command or function Example query Description

fields

fields field1, field2

Displays a set of fields which needs projection.

where

where field1="success" | where field2 != "i-023fe0a90929d8822" | fields field3, field4, field5,field6 | head 1000

Filters the data based on the conditions that you specify.

stats

stats count(), count(field1), min(field1), max(field1), avg(field1) by field2 | head 1000

Performs aggregations and calculations

parse

parse field1 ".*/(?<field2>[^/]+$)" | where field2 = "requestId" | fields field1, field2 | head 1000

Extracts a regular expression (regex) pattern from a string and displays the extracted pattern. The extracted pattern can be further used to create new fields or filter data.

sort

stats count(), count(field1), min(field1) as field1Alias, max(`field1`), avg(`field1`) by field2 | sort -field1Alias | head 1000

Sort the displayed results by a field name. Use sort -FieldName to sort in descending order.

eval

eval field2 = field1 * 2 | fields field1, field2 | head 20

Modifies or processes the value of a field and stores it in a different field. This is useful to mathematically modify a column, apply string functions to a column, or apply date functions to a column.

rename

rename field2 as field1 | fields field1;

Renames one or more fields in the search result.

head

fields `@message` | head 20

Limits the displayed query results to the first N rows.

top

top 2 field1 by field2

Finds the most frequent values for a field.

dedup

dedup field1 | fields field1, field2, field3

Removes duplicate entries based on the fields that you specify.

rare

rare field1 by field2

Finds the least frequent values of all fields in the field list.

trendline

trendline sma(2, field1) as field1Alias

Calculates the moving averages of fields.

eventStats

eventstats sum(field1) by field2

Enriches your event data with calculated summary statistics. It analyzes specified fields within your events, computes various statistical measures, and then appends these results to each original event as new fields.

fieldsummary

where field1 != 200 | fieldsummary includefields= field1 nulls=true

Calculates basic statistics for each field (count, distinct count, min, max, avg, stddev, and mean).

grok

grok email '.+@%{HOSTNAME:host}' | fields email, host

Parses a text field with a grok pattern and appends the results to the search result.

String functions

eval field1Len = LENGTH(field1) | fields field1Len

Built-in functions in PPL that can manipulate and transform string and text data within PPL queries. For example, converting case, combining strings, extracting parts, and cleaning text.

Math functions

eval field2 = ACOS(field1) | fields field1

Built-in functions for performing mathematical calculations and transformations in PPL queries. For example, abs (absolute value), round (rounds numbers), sqrt (square root), pow (power calculation), and ceil (rounds up to nearest integer).

Date functions

eval newDate = ADDDATE(DATE('2020-08-26'), 1) | fields newDate

Built-in functions for handling and transforming date and timestamp data in PPL queries. For example, date_add, date_format, datediff, and current_date.

Condition functions

eval field2 = isnull(field1) | fields field2, field1, field3

Built-in functions that check for specific field conditions, and evaluate expressions conditionally. For example, if field1 is null, return field2.

Math functions

eval field2 = ACOS(field1) | fields field1

Built-in functions for performing mathematical calculations and transformations in PPL queries. For example, abs (absolute value), round (rounds numbers), sqrt (square root), pow (power calculation), and ceil (rounds up to nearest integer).

CryptoGraphic functions

eval crypto = MD5(field)| head 1000

To calculate the hash of given field