patterns command
Note
To see which AWS data source integrations support this PPL command, see Commands.
The patterns command extracts log patterns from a text field and
appends the results to the search result. Grouping logs by their patterns makes
it easier to aggregate stats from large volumes of log data for analysis and
troubleshooting.
Syntax
Use the following syntax:
patterns [new_field=<new-field-name>] [pattern=<pattern>] <field>
new-field-name
-
Optional string.
-
This is the name of the new field for extracted patterns.
-
The default is
patterns_field. -
If the name already exists, it will replace the original field.
pattern
-
Optional string.
-
This it the regex pattern of characters that should be filtered out from the text field.
-
If absent, the default pattern is alphanumeric characters (
[a-zA-Z\d]).
field
-
Mandatory.
-
The field must be a text field.
Example 1: Create the new field
The example shows how to use extract punctuations in email for each document. Parsing a null field will return an empty string.
PPL query:
os> source=accounts | patterns email | fields email, patterns_field ; fetched rows / total rows = 4/4 +-----------------------+------------------+ | email | patterns_field | |-----------------------+------------------| | jane_doe@example.com | @. | | john_doe@example.net | @. | | null | | | juan_li@example.org | @. | +-----------------------+------------------+
Example 2: Extract log patterns
The example shows how to extract punctuations from a raw log field using the default patterns.
PPL query:
os> source=apache | patterns message | fields message, patterns_field ; fetched rows / total rows = 4/4 +-----------------------------------------------------------------------------------------------------------------------------+---------------------------------+ | message | patterns_field | |-----------------------------------------------------------------------------------------------------------------------------+---------------------------------| | 177.95.8.74 - upton5450 [28/Sep/2022:10:15:57 -0700] "HEAD /e-business/mindshare HTTP/1.0" 404 19927 | ... - [//::: -] " /-/ /." | | ************ - pouros8756 [28/Sep/2022:10:15:57 -0700] "GET /architectures/convergence/niches/mindshare HTTP/1.0" 100 28722 | ... - [//::: -] " //// /." | | *************** - - [28/Sep/2022:10:15:57 -0700] "PATCH /strategize/out-of-the-box HTTP/1.0" 401 27439 | ... - - [//::: -] " //--- /." | | ************** - - [28/Sep/2022:10:15:57 -0700] "POST /users HTTP/1.1" 301 9481 | ... - - [//::: -] " / /." | +-----------------------------------------------------------------------------------------------------------------------------+---------------------------------+
Example 3: Extract log patterns with custom regex pattern
The example shows how to extract punctuations from a raw log field using user defined patterns.
PPL query:
os> source=apache | patterns new_field='no_numbers' pattern='[0-9]' message | fields message, no_numbers ; fetched rows / total rows = 4/4 +-----------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+ | message | no_numbers | |-----------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------| | 177.95.8.74 - upton5450 [28/Sep/2022:10:15:57 -0700] "HEAD /e-business/mindshare HTTP/1.0" 404 19927 | ... - upton [/Sep/::: -] "HEAD /e-business/mindshare HTTP/." | | 127.45.152.6 - pouros8756 [28/Sep/2022:10:15:57 -0700] "GET /architectures/convergence/niches/mindshare HTTP/1.0" 100 28722 | ... - pouros [/Sep/::: -] "GET /architectures/convergence/niches/mindshare HTTP/." | | *************** - - [28/Sep/2022:10:15:57 -0700] "PATCH /strategize/out-of-the-box HTTP/1.0" 401 27439 | ... - - [/Sep/::: -] "PATCH /strategize/out-of-the-box HTTP/." | | ************** - - [28/Sep/2022:10:15:57 -0700] "POST /users HTTP/1.1" 301 9481 | ... - - [/Sep/::: -] "POST /users HTTP/." | +-----------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+
Limitation
The patterns command has the same limitations as the parse command.
