aws-cdk-lib.aws_logs

Amazon CloudWatch Logs Construct Library

This library supplies constructs for working with CloudWatch Logs.

Log Groups/Streams

The basic unit of CloudWatch is a Log Group. Every log group typically has the same kind of data logged to it, in the same format. If there are multiple applications or services logging into the Log Group, each of them creates a new Log Stream.

Every log operation creates a "log event", which can consist of a simple string or a single-line JSON object. JSON objects have the advantage that they afford more filtering abilities (see below).

The only configurable attribute for log streams is the retention period, which configures after how much time the events in the log stream expire and are deleted.

The default retention period if not supplied is 2 years, but it can be set to one of the values in the RetentionDays enum to configure a different retention period (including infinite retention).

// Configure log group for short retention
const logGroup = new LogGroup(stack, 'LogGroup', {
  retention: RetentionDays.ONE_WEEK,
});
// Configure log group for infinite retention
const logGroup = new LogGroup(stack, 'LogGroup', {
  retention: Infinity,
});

# Configure log group for short retention
log_group = LogGroup(stack, "LogGroup",
    retention=RetentionDays.ONE_WEEK
)# Configure log group for infinite retention
log_group = LogGroup(stack, "LogGroup",
    retention=Infinity
)

// Configure log group for short retention
LogGroup logGroup = LogGroup.Builder.create(stack, "LogGroup")
        .retention(RetentionDays.ONE_WEEK)
        .build();// Configure log group for infinite retention
LogGroup logGroup = LogGroup.Builder.create(stack, "LogGroup")
        .retention(Infinity)
        .build();

// Configure log group for short retention
var logGroup = new LogGroup(stack, "LogGroup", new LogGroupProps {
    Retention = RetentionDays.ONE_WEEK
});// Configure log group for infinite retention
var logGroup = new LogGroup(stack, "LogGroup", new LogGroupProps {
    Retention = Infinity
});

// Configure log group for short retention
logGroup := awscdk.NewLogGroup(stack, jsii.String("LogGroup"), &LogGroupProps{
	Retention: awscdk.RetentionDays_ONE_WEEK,
})// Configure log group for infinite retention
logGroup := awscdk.NewLogGroup(stack, jsii.String("LogGroup"), &LogGroupProps{
	Retention: infinity,
})

LogRetention

The LogRetention construct is a way to control the retention period of log groups that are created outside of the CDK. The construct is usually used on log groups that are auto created by AWS services, such as AWS lambda.

This is implemented using a CloudFormation custom resource which pre-creates the log group if it doesn't exist, and sets the specified log retention period (never expire, by default).

By default, the log group will be created in the same region as the stack. The logGroupRegion property can be used to configure log groups in other regions. This is typically useful when controlling retention for log groups auto-created by global services that publish their log group to a specific region, such as AWS Chatbot creating a log group in us-east-1.

By default, the log group created by LogRetention will be retained after the stack is deleted. If the RemovalPolicy is set to DESTROY, then the log group will be deleted when the stack is deleted.

Log Group Class

CloudWatch Logs offers three classes of log groups:

1. The CloudWatch Logs Standard log class is a full-featured option for logs that require real-time monitoring or logs that you access frequently.

2. The CloudWatch Logs Infrequent Access log class is a new log class that you can use to cost-effectively consolidate your logs. This log class offers a subset of CloudWatch Logs capabilities including managed ingestion, storage, cross-account log analytics, and encryption with a lower ingestion price per GB. The Infrequent Access log class is ideal for ad-hoc querying and after-the-fact forensic analysis on infrequently accessed logs.

3. The CloudWatch Logs Delivery log class is used to deliver logs to a destination such as Amazon S3 or Amazon Data Firehose (for example, Lambda vended logs). A Delivery log group does not store log events itself; it forwards them to a destination configured through a subscription filter. Because of this, the Delivery log class does not support retention, dataProtectionPolicy, or fieldIndexPolicies; setting any of them together with LogGroupClass.DELIVERY results in a synthesis-time error.

new logs.LogGroup(this, 'DeliveryLogGroup', {
  logGroupClass: logs.LogGroupClass.DELIVERY,
});

logs.LogGroup(self, "DeliveryLogGroup",
    log_group_class=logs.LogGroupClass.DELIVERY
)

LogGroup.Builder.create(this, "DeliveryLogGroup")
        .logGroupClass(LogGroupClass.DELIVERY)
        .build();

new LogGroup(this, "DeliveryLogGroup", new LogGroupProps {
    LogGroupClass = LogGroupClass.DELIVERY
});

logs.NewLogGroup(this, jsii.String("DeliveryLogGroup"), &LogGroupProps{
	LogGroupClass: logs.LogGroupClass_DELIVERY,
})

For more details please check: log group class documentation

Resource Policy

CloudWatch Resource Policies allow other AWS services or IAM Principals to put log events into the log groups. A resource policy is automatically created when addToResourcePolicy is called on the LogGroup for the first time:

const logGroup = new logs.LogGroup(this, 'LogGroup');
logGroup.addToResourcePolicy(new iam.PolicyStatement({
    actions: ['logs:CreateLogStream', 'logs:PutLogEvents'],
    principals: [new iam.ServicePrincipal('es.amazonaws.com')],
    resources: [logGroup.logGroupArn],
}));

log_group = logs.LogGroup(self, "LogGroup")
log_group.add_to_resource_policy(iam.PolicyStatement(
    actions=["logs:CreateLogStream", "logs:PutLogEvents"],
    principals=[iam.ServicePrincipal("es.amazonaws.com")],
    resources=[log_group.log_group_arn]
))

LogGroup logGroup = new LogGroup(this, "LogGroup");
logGroup.addToResourcePolicy(PolicyStatement.Builder.create()
        .actions(List.of("logs:CreateLogStream", "logs:PutLogEvents"))
        .principals(List.of(new ServicePrincipal("es.amazonaws.com")))
        .resources(List.of(logGroup.getLogGroupArn()))
        .build());

var logGroup = new LogGroup(this, "LogGroup");
logGroup.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps {
    Actions = new [] { "logs:CreateLogStream", "logs:PutLogEvents" },
    Principals = new [] { new ServicePrincipal("es.amazonaws.com") },
    Resources = new [] { logGroup.LogGroupArn }
}));

logGroup := logs.NewLogGroup(this, jsii.String("LogGroup"))
logGroup.addToResourcePolicy(iam.NewPolicyStatement(&PolicyStatementProps{
	Actions: []*string{
		jsii.String("logs:CreateLogStream"),
		jsii.String("logs:PutLogEvents"),
	},
	Principals: []IPrincipal{
		iam.NewServicePrincipal(jsii.String("es.amazonaws.com")),
	},
	Resources: []*string{
		logGroup.logGroupArn,
	},
}))

Or more conveniently, write permissions to the log group can be granted as follows which gives same result as in the above example.

const logGroup = new logs.LogGroup(this, 'LogGroup');
logGroup.grants.write(new iam.ServicePrincipal('es.amazonaws.com'));

log_group = logs.LogGroup(self, "LogGroup")
log_group.grants.write(iam.ServicePrincipal("es.amazonaws.com"))

LogGroup logGroup = new LogGroup(this, "LogGroup");
logGroup.grants.write(new ServicePrincipal("es.amazonaws.com"));

var logGroup = new LogGroup(this, "LogGroup");
logGroup.Grants.Write(new ServicePrincipal("es.amazonaws.com"));

logGroup := logs.NewLogGroup(this, jsii.String("LogGroup"))
logGroup.grants.Write(iam.NewServicePrincipal(jsii.String("es.amazonaws.com")))

Similarly, read permissions can be granted to the log group as follows.

const logGroup = new logs.LogGroup(this, 'LogGroup');
logGroup.grants.read(new iam.ServicePrincipal('es.amazonaws.com'));

log_group = logs.LogGroup(self, "LogGroup")
log_group.grants.read(iam.ServicePrincipal("es.amazonaws.com"))

LogGroup logGroup = new LogGroup(this, "LogGroup");
logGroup.grants.read(new ServicePrincipal("es.amazonaws.com"));

var logGroup = new LogGroup(this, "LogGroup");
logGroup.Grants.Read(new ServicePrincipal("es.amazonaws.com"));

logGroup := logs.NewLogGroup(this, jsii.String("LogGroup"))
logGroup.grants.Read(iam.NewServicePrincipal(jsii.String("es.amazonaws.com")))

Be aware that any ARNs or tokenized values passed to the resource policy will be converted into AWS Account IDs. This is because CloudWatch Logs Resource Policies do not accept ARNs as principals, but they do accept Account ID strings. Non-ARN principals, like Service principals or Any principals, are accepted by CloudWatch.

Encrypting Log Groups

By default, log group data is always encrypted in CloudWatch Logs. You have the option to encrypt log group data using a AWS KMS customer master key (CMK) should you not wish to use the default AWS encryption. Keep in mind that if you decide to encrypt a log group, any service or IAM identity that needs to read the encrypted log streams in the future will require the same CMK to decrypt the data.

Here's a simple example of creating an encrypted Log Group using a KMS CMK.

import * as kms from 'aws-cdk-lib/aws-kms';

new logs.LogGroup(this, 'LogGroup', {
  encryptionKey: new kms.Key(this, 'Key'),
});

import aws_cdk.aws_kms as kms


logs.LogGroup(self, "LogGroup",
    encryption_key=kms.Key(self, "Key")
)

import software.amazon.awscdk.services.kms.*;


LogGroup.Builder.create(this, "LogGroup")
        .encryptionKey(new Key(this, "Key"))
        .build();

using Amazon.CDK.AWS.KMS;


new LogGroup(this, "LogGroup", new LogGroupProps {
    EncryptionKey = new Key(this, "Key")
});

import kms "github.com/aws/aws-cdk-go/awscdk"


logs.NewLogGroup(this, jsii.String("LogGroup"), &LogGroupProps{
	EncryptionKey: kms.NewKey(this, jsii.String("Key")),
})

See the AWS documentation for more detailed information about encrypting CloudWatch Logs.

Subscriptions and Destinations

Log events matching a particular filter can be sent to either a Lambda function or a Kinesis stream.

If the Kinesis stream lives in a different account, a CrossAccountDestination object must be explicitly created in the destination account which will act as a proxy for the remote Kinesis stream.

Note: The aws-cdk-lib/aws-logs-destinations KinesisDestination construct does not automatically create a CrossAccountDestination for cross-account scenarios.

Create a SubscriptionFilter, initialize it with an appropriate Pattern (see below) and supply the intended destination:

import * as destinations from 'aws-cdk-lib/aws-logs-destinations';

declare const fn: lambda.Function;
declare const logGroup: logs.LogGroup;

new logs.SubscriptionFilter(this, 'Subscription', {
  logGroup,
  destination: new destinations.LambdaDestination(fn),
  filterPattern: logs.FilterPattern.allTerms("ERROR", "MainThread"),
  filterName: 'ErrorInMainThread',
});

import aws_cdk.aws_logs_destinations as destinations

# fn: lambda.Function
# log_group: logs.LogGroup


logs.SubscriptionFilter(self, "Subscription",
    log_group=log_group,
    destination=destinations.LambdaDestination(fn),
    filter_pattern=logs.FilterPattern.all_terms("ERROR", "MainThread"),
    filter_name="ErrorInMainThread"
)

import software.amazon.awscdk.services.logs.destinations.*;

Function fn;
LogGroup logGroup;


SubscriptionFilter.Builder.create(this, "Subscription")
        .logGroup(logGroup)
        .destination(new LambdaDestination(fn))
        .filterPattern(FilterPattern.allTerms("ERROR", "MainThread"))
        .filterName("ErrorInMainThread")
        .build();

using Amazon.CDK.AWS.Logs.Destinations;

Function fn;
LogGroup logGroup;


new SubscriptionFilter(this, "Subscription", new SubscriptionFilterProps {
    LogGroup = logGroup,
    Destination = new LambdaDestination(fn),
    FilterPattern = FilterPattern.AllTerms("ERROR", "MainThread"),
    FilterName = "ErrorInMainThread"
});

import destinations "github.com/aws/aws-cdk-go/awscdk"

var fn Function
var logGroup LogGroup


logs.NewSubscriptionFilter(this, jsii.String("Subscription"), &SubscriptionFilterProps{
	LogGroup: LogGroup,
	Destination: destinations.NewLambdaDestination(fn),
	FilterPattern: logs.FilterPattern_AllTerms(jsii.String("ERROR"), jsii.String("MainThread")),
	FilterName: jsii.String("ErrorInMainThread"),
})

When you use KinesisDestination, you can choose the method used to distribute log data to the destination by setting the distribution property.

import * as destinations from 'aws-cdk-lib/aws-logs-destinations';
import * as kinesis from 'aws-cdk-lib/aws-kinesis';

declare const stream: kinesis.Stream;
declare const logGroup: logs.LogGroup;

new logs.SubscriptionFilter(this, 'Subscription', {
  logGroup,
  destination: new destinations.KinesisDestination(stream),
  filterPattern: logs.FilterPattern.allTerms("ERROR", "MainThread"),
  filterName: 'ErrorInMainThread',
  distribution: logs.Distribution.RANDOM,
});

import aws_cdk.aws_logs_destinations as destinations
import aws_cdk.aws_kinesis as kinesis

# stream: kinesis.Stream
# log_group: logs.LogGroup


logs.SubscriptionFilter(self, "Subscription",
    log_group=log_group,
    destination=destinations.KinesisDestination(stream),
    filter_pattern=logs.FilterPattern.all_terms("ERROR", "MainThread"),
    filter_name="ErrorInMainThread",
    distribution=logs.Distribution.RANDOM
)

import software.amazon.awscdk.services.logs.destinations.*;
import software.amazon.awscdk.services.kinesis.*;

Stream stream;
LogGroup logGroup;


SubscriptionFilter.Builder.create(this, "Subscription")
        .logGroup(logGroup)
        .destination(new KinesisDestination(stream))
        .filterPattern(FilterPattern.allTerms("ERROR", "MainThread"))
        .filterName("ErrorInMainThread")
        .distribution(Distribution.RANDOM)
        .build();

using Amazon.CDK.AWS.Logs.Destinations;
using Amazon.CDK.AWS.Kinesis;

Stream stream;
LogGroup logGroup;


new SubscriptionFilter(this, "Subscription", new SubscriptionFilterProps {
    LogGroup = logGroup,
    Destination = new KinesisDestination(stream),
    FilterPattern = FilterPattern.AllTerms("ERROR", "MainThread"),
    FilterName = "ErrorInMainThread",
    Distribution = Distribution.RANDOM
});

import destinations "github.com/aws/aws-cdk-go/awscdk"
import kinesis "github.com/aws/aws-cdk-go/awscdk"

var stream Stream
var logGroup LogGroup


logs.NewSubscriptionFilter(this, jsii.String("Subscription"), &SubscriptionFilterProps{
	LogGroup: LogGroup,
	Destination: destinations.NewKinesisDestination(stream),
	FilterPattern: logs.FilterPattern_AllTerms(jsii.String("ERROR"), jsii.String("MainThread")),
	FilterName: jsii.String("ErrorInMainThread"),
	Distribution: logs.Distribution_RANDOM,
})

When you use FirehoseDestination, you can choose the method used to distribute log data to the destination by setting the distribution property.

import * as destinations from 'aws-cdk-lib/aws-logs-destinations';
import * as firehose from 'aws-cdk-lib/aws-kinesisfirehose';

declare const deliveryStream: firehose.IDeliveryStream;
declare const logGroup: logs.LogGroup;

new logs.SubscriptionFilter(this, 'Subscription', {
  logGroup,
  destination: new destinations.FirehoseDestination(deliveryStream),
  filterPattern: logs.FilterPattern.allEvents(),
});

import aws_cdk.aws_logs_destinations as destinations
import aws_cdk.aws_kinesisfirehose as firehose

# delivery_stream: firehose.IDeliveryStream
# log_group: logs.LogGroup


logs.SubscriptionFilter(self, "Subscription",
    log_group=log_group,
    destination=destinations.FirehoseDestination(delivery_stream),
    filter_pattern=logs.FilterPattern.all_events()
)

import software.amazon.awscdk.services.logs.destinations.*;
import software.amazon.awscdk.services.kinesisfirehose.*;

IDeliveryStream deliveryStream;
LogGroup logGroup;


SubscriptionFilter.Builder.create(this, "Subscription")
        .logGroup(logGroup)
        .destination(new FirehoseDestination(deliveryStream))
        .filterPattern(FilterPattern.allEvents())
        .build();

using Amazon.CDK.AWS.Logs.Destinations;
using Amazon.CDK.AWS.KinesisFirehose;

IDeliveryStream deliveryStream;
LogGroup logGroup;


new SubscriptionFilter(this, "Subscription", new SubscriptionFilterProps {
    LogGroup = logGroup,
    Destination = new FirehoseDestination(deliveryStream),
    FilterPattern = FilterPattern.AllEvents()
});

import destinations "github.com/aws/aws-cdk-go/awscdk"
import firehose "github.com/aws/aws-cdk-go/awscdk"

var deliveryStream IDeliveryStream
var logGroup LogGroup


logs.NewSubscriptionFilter(this, jsii.String("Subscription"), &SubscriptionFilterProps{
	LogGroup: LogGroup,
	Destination: destinations.NewFirehoseDestination(deliveryStream),
	FilterPattern: logs.FilterPattern_AllEvents(),
})

Metric Filters

CloudWatch Logs can extract and emit metrics based on a textual log stream. Depending on your needs, this may be a more convenient way of generating metrics for you application than making calls to CloudWatch Metrics yourself.

A MetricFilter either emits a fixed number every time it sees a log event matching a particular pattern (see below), or extracts a number from the log event and uses that as the metric value.

Example:

new MetricFilter(this, 'MetricFilter', {
  logGroup,
  metricNamespace: 'MyApp',
  metricName: 'Latency',
  filterPattern: FilterPattern.all(
    FilterPattern.exists('$.latency'),
    FilterPattern.regexValue('$.message', '=', 'bind: address already in use'),
  ),
  metricValue: '$.latency',
});

MetricFilter(self, "MetricFilter",
    log_group=log_group,
    metric_namespace="MyApp",
    metric_name="Latency",
    filter_pattern=FilterPattern.all(
        FilterPattern.exists("$.latency"),
        FilterPattern.regex_value("$.message", "=", "bind: address already in use")),
    metric_value="$.latency"
)

MetricFilter.Builder.create(this, "MetricFilter")
        .logGroup(logGroup)
        .metricNamespace("MyApp")
        .metricName("Latency")
        .filterPattern(FilterPattern.all(FilterPattern.exists("$.latency"), FilterPattern.regexValue("$.message", "=", "bind: address already in use")))
        .metricValue("$.latency")
        .build();

new MetricFilter(this, "MetricFilter", new MetricFilterProps {
    LogGroup = logGroup,
    MetricNamespace = "MyApp",
    MetricName = "Latency",
    FilterPattern = FilterPattern.All(FilterPattern.Exists("$.latency"), FilterPattern.RegexValue("$.message", "=", "bind: address already in use")),
    MetricValue = "$.latency"
});

awscdk.NewMetricFilter(this, jsii.String("MetricFilter"), &MetricFilterProps{
	LogGroup: LogGroup,
	MetricNamespace: jsii.String("MyApp"),
	MetricName: jsii.String("Latency"),
	FilterPattern: awscdk.FilterPattern_All(awscdk.FilterPattern_Exists(jsii.String("$.latency")), awscdk.FilterPattern_RegexValue(jsii.String("$.message"), jsii.String("="), jsii.String("bind: address already in use"))),
	MetricValue: jsii.String("$.latency"),
})

Remember that if you want to use a value from the log event as the metric value, you must mention it in your pattern somewhere.

A very simple MetricFilter can be created by using the logGroup.extractMetric() helper function:

declare const logGroup: logs.LogGroup;
logGroup.extractMetric('$.jsonField', 'Namespace', 'MetricName');

# log_group: logs.LogGroup

log_group.extract_metric("$.jsonField", "Namespace", "MetricName")

LogGroup logGroup;

logGroup.extractMetric("$.jsonField", "Namespace", "MetricName");

LogGroup logGroup;

logGroup.ExtractMetric("$.jsonField", "Namespace", "MetricName");

var logGroup LogGroup

logGroup.extractMetric(jsii.String("$.jsonField"), jsii.String("Namespace"), jsii.String("MetricName"))

Will extract the value of jsonField wherever it occurs in JSON-structured log records in the LogGroup, and emit them to CloudWatch Metrics under the name Namespace/MetricName.

Exposing Metric on a Metric Filter

You can expose a metric on a metric filter by calling the MetricFilter.metric() API. This has a default of statistic = 'avg' if the statistic is not set in the props.

declare const logGroup: logs.LogGroup;
const mf = new logs.MetricFilter(this, 'MetricFilter', {
  logGroup,
  metricNamespace: 'MyApp',
  metricName: 'Latency',
  filterPattern: logs.FilterPattern.exists('$.latency'),
  metricValue: '$.latency',
  dimensions: {
    ErrorCode: '$.errorCode',
  },
  unit: cloudwatch.Unit.MILLISECONDS,
});

//expose a metric from the metric filter
const metric = mf.metric();

//you can use the metric to create a new alarm
new cloudwatch.Alarm(this, 'alarm from metric filter', {
  metric,
  threshold: 100,
  evaluationPeriods: 2,
});

# log_group: logs.LogGroup

mf = logs.MetricFilter(self, "MetricFilter",
    log_group=log_group,
    metric_namespace="MyApp",
    metric_name="Latency",
    filter_pattern=logs.FilterPattern.exists("$.latency"),
    metric_value="$.latency",
    dimensions={
        "ErrorCode": "$.errorCode"
    },
    unit=cloudwatch.Unit.MILLISECONDS
)

# expose a metric from the metric filter
metric = mf.metric()

# you can use the metric to create a new alarm
cloudwatch.Alarm(self, "alarm from metric filter",
    metric=metric,
    threshold=100,
    evaluation_periods=2
)

LogGroup logGroup;

MetricFilter mf = MetricFilter.Builder.create(this, "MetricFilter")
        .logGroup(logGroup)
        .metricNamespace("MyApp")
        .metricName("Latency")
        .filterPattern(FilterPattern.exists("$.latency"))
        .metricValue("$.latency")
        .dimensions(Map.of(
                "ErrorCode", "$.errorCode"))
        .unit(Unit.MILLISECONDS)
        .build();

//expose a metric from the metric filter
Metric metric = mf.metric();

//you can use the metric to create a new alarm
//you can use the metric to create a new alarm
Alarm.Builder.create(this, "alarm from metric filter")
        .metric(metric)
        .threshold(100)
        .evaluationPeriods(2)
        .build();

LogGroup logGroup;

var mf = new MetricFilter(this, "MetricFilter", new MetricFilterProps {
    LogGroup = logGroup,
    MetricNamespace = "MyApp",
    MetricName = "Latency",
    FilterPattern = FilterPattern.Exists("$.latency"),
    MetricValue = "$.latency",
    Dimensions = new Dictionary<string, string> {
        { "ErrorCode", "$.errorCode" }
    },
    Unit = Unit.MILLISECONDS
});

//expose a metric from the metric filter
var metric = mf.Metric();

//you can use the metric to create a new alarm
//you can use the metric to create a new alarm
new Alarm(this, "alarm from metric filter", new AlarmProps {
    Metric = metric,
    Threshold = 100,
    EvaluationPeriods = 2
});

var logGroup LogGroup

mf := logs.NewMetricFilter(this, jsii.String("MetricFilter"), &MetricFilterProps{
	LogGroup: LogGroup,
	MetricNamespace: jsii.String("MyApp"),
	MetricName: jsii.String("Latency"),
	FilterPattern: logs.FilterPattern_Exists(jsii.String("$.latency")),
	MetricValue: jsii.String("$.latency"),
	Dimensions: map[string]*string{
		"ErrorCode": jsii.String("$.errorCode"),
	},
	Unit: cloudwatch.Unit_MILLISECONDS,
})

//expose a metric from the metric filter
metric := mf.Metric()

//you can use the metric to create a new alarm
//you can use the metric to create a new alarm
cloudwatch.NewAlarm(this, jsii.String("alarm from metric filter"), &AlarmProps{
	Metric: Metric,
	Threshold: jsii.Number(100),
	EvaluationPeriods: jsii.Number(2),
})

Metrics for IncomingLogs and IncomingBytes

Metric methods have been defined for IncomingLogs and IncomingBytes within LogGroups. These metrics allow for the creation of alarms on log ingestion, ensuring that the log ingestion process is functioning correctly.

To define an alarm based on these metrics, you can use the following template:

const logGroup = new logs.LogGroup(this, 'MyLogGroup');
const incomingEventsMetric = logGroup.metricIncomingLogEvents();
new cloudwatch.Alarm(this, 'HighLogVolumeAlarm', {
  metric: incomingEventsMetric,
  threshold: 1000,
  evaluationPeriods: 1,
});

log_group = logs.LogGroup(self, "MyLogGroup")
incoming_events_metric = log_group.metric_incoming_log_events()
cloudwatch.Alarm(self, "HighLogVolumeAlarm",
    metric=incoming_events_metric,
    threshold=1000,
    evaluation_periods=1
)

LogGroup logGroup = new LogGroup(this, "MyLogGroup");
Metric incomingEventsMetric = logGroup.metricIncomingLogEvents();
Alarm.Builder.create(this, "HighLogVolumeAlarm")
        .metric(incomingEventsMetric)
        .threshold(1000)
        .evaluationPeriods(1)
        .build();

var logGroup = new LogGroup(this, "MyLogGroup");
var incomingEventsMetric = logGroup.MetricIncomingLogEvents();
new Alarm(this, "HighLogVolumeAlarm", new AlarmProps {
    Metric = incomingEventsMetric,
    Threshold = 1000,
    EvaluationPeriods = 1
});

logGroup := logs.NewLogGroup(this, jsii.String("MyLogGroup"))
incomingEventsMetric := logGroup.metricIncomingLogEvents()
cloudwatch.NewAlarm(this, jsii.String("HighLogVolumeAlarm"), &AlarmProps{
	Metric: incomingEventsMetric,
	Threshold: jsii.Number(1000),
	EvaluationPeriods: jsii.Number(1),
})

const logGroup = new logs.LogGroup(this, 'MyLogGroup');
const incomingBytesMetric = logGroup.metricIncomingBytes();
new cloudwatch.Alarm(this, 'HighDataVolumeAlarm', {
  metric: incomingBytesMetric,
  threshold: 5000000,  // 5 MB
  evaluationPeriods: 1,
});

log_group = logs.LogGroup(self, "MyLogGroup")
incoming_bytes_metric = log_group.metric_incoming_bytes()
cloudwatch.Alarm(self, "HighDataVolumeAlarm",
    metric=incoming_bytes_metric,
    threshold=5000000,  # 5 MB
    evaluation_periods=1
)

LogGroup logGroup = new LogGroup(this, "MyLogGroup");
Metric incomingBytesMetric = logGroup.metricIncomingBytes();
Alarm.Builder.create(this, "HighDataVolumeAlarm")
        .metric(incomingBytesMetric)
        .threshold(5000000) // 5 MB
        .evaluationPeriods(1)
        .build();

var logGroup = new LogGroup(this, "MyLogGroup");
var incomingBytesMetric = logGroup.MetricIncomingBytes();
new Alarm(this, "HighDataVolumeAlarm", new AlarmProps {
    Metric = incomingBytesMetric,
    Threshold = 5000000,  // 5 MB
    EvaluationPeriods = 1
});

logGroup := logs.NewLogGroup(this, jsii.String("MyLogGroup"))
incomingBytesMetric := logGroup.metricIncomingBytes()
cloudwatch.NewAlarm(this, jsii.String("HighDataVolumeAlarm"), &AlarmProps{
	Metric: incomingBytesMetric,
	Threshold: jsii.Number(5000000),
	 // 5 MB
	EvaluationPeriods: jsii.Number(1),
})

Patterns

Patterns describe which log events match a subscription or metric filter. There are three types of patterns:

• Text patterns
• JSON patterns
• Space-delimited table patterns

All patterns are constructed by using static functions on the FilterPattern class.

In addition to the patterns above, the following special patterns exist:

• FilterPattern.allEvents(): matches all log events.
• FilterPattern.literal(string): if you already know what pattern expression to use, this function takes a string and will use that as the log pattern. For more information, see the Filter and Pattern Syntax.

Text Patterns

Text patterns match if the literal strings appear in the text form of the log line.

• FilterPattern.allTerms(term, term, ...): matches if all of the given terms (substrings) appear in the log event.
• FilterPattern.anyTerm(term, term, ...): matches if all of the given terms (substrings) appear in the log event.
• FilterPattern.anyTermGroup([term, term, ...], [term, term, ...], ...): matches if all of the terms in any of the groups (specified as arrays) matches. This is an OR match.

Examples:

// Search for lines that contain both "ERROR" and "MainThread"
const pattern1 = logs.FilterPattern.allTerms('ERROR', 'MainThread');

// Search for lines that either contain both "ERROR" and "MainThread", or
// both "WARN" and "Deadlock".
const pattern2 = logs.FilterPattern.anyTermGroup(
  ['ERROR', 'MainThread'],
  ['WARN', 'Deadlock'],
);

# Search for lines that contain both "ERROR" and "MainThread"
pattern1 = logs.FilterPattern.all_terms("ERROR", "MainThread")

# Search for lines that either contain both "ERROR" and "MainThread", or
# both "WARN" and "Deadlock".
pattern2 = logs.FilterPattern.any_term_group(["ERROR", "MainThread"], ["WARN", "Deadlock"])

// Search for lines that contain both "ERROR" and "MainThread"
IFilterPattern pattern1 = FilterPattern.allTerms("ERROR", "MainThread");

// Search for lines that either contain both "ERROR" and "MainThread", or
// both "WARN" and "Deadlock".
IFilterPattern pattern2 = FilterPattern.anyTermGroup(List.of("ERROR", "MainThread"), List.of("WARN", "Deadlock"));

// Search for lines that contain both "ERROR" and "MainThread"
var pattern1 = FilterPattern.AllTerms("ERROR", "MainThread");

// Search for lines that either contain both "ERROR" and "MainThread", or
// both "WARN" and "Deadlock".
var pattern2 = FilterPattern.AnyTermGroup(new [] { "ERROR", "MainThread" }, new [] { "WARN", "Deadlock" });

// Search for lines that contain both "ERROR" and "MainThread"
pattern1 := logs.FilterPattern_AllTerms(jsii.String("ERROR"), jsii.String("MainThread"))

// Search for lines that either contain both "ERROR" and "MainThread", or
// both "WARN" and "Deadlock".
pattern2 := logs.FilterPattern_AnyTermGroup([]*string{
	jsii.String("ERROR"),
	jsii.String("MainThread"),
}, []*string{
	jsii.String("WARN"),
	jsii.String("Deadlock"),
})

JSON Patterns

JSON patterns apply if the log event is the JSON representation of an object (without any other characters, so it cannot include a prefix such as timestamp or log level). JSON patterns can make comparisons on the values inside the fields.

• Strings: the comparison operators allowed for strings are = and !=. String values can start or end with a * wildcard.
• Numbers: the comparison operators allowed for numbers are =, !=, <, <=, >, >=.

Fields in the JSON structure are identified by identifier the complete object as $ and then descending into it, such as $.field or $.list[0].field.

• FilterPattern.stringValue(field, comparison, string): matches if the given field compares as indicated with the given string value.
• FilterPattern.regexValue(field, comparison, string): matches if the given field compares as indicated with the given regex pattern.
• FilterPattern.numberValue(field, comparison, number): matches if the given field compares as indicated with the given numerical value.
• FilterPattern.isNull(field): matches if the given field exists and has the value null.
• FilterPattern.notExists(field): matches if the given field is not in the JSON structure.
• FilterPattern.exists(field): matches if the given field is in the JSON structure.
• FilterPattern.booleanValue(field, boolean): matches if the given field is exactly the given boolean value.
• FilterPattern.all(jsonPattern, jsonPattern, ...): matches if all of the given JSON patterns match. This makes an AND combination of the given patterns.
• FilterPattern.any(jsonPattern, jsonPattern, ...): matches if any of the given JSON patterns match. This makes an OR combination of the given patterns.

Example:

// Search for all events where the component field is equal to
// "HttpServer" and either error is true or the latency is higher
// than 1000.
const pattern = logs.FilterPattern.all(
  logs.FilterPattern.stringValue('$.component', '=', 'HttpServer'),
  logs.FilterPattern.any(
    logs.FilterPattern.booleanValue('$.error', true),
    logs.FilterPattern.numberValue('$.latency', '>', 1000),
  ),
  logs.FilterPattern.regexValue('$.message', '=', 'bind address already in use'),
);

# Search for all events where the component field is equal to
# "HttpServer" and either error is true or the latency is higher
# than 1000.
pattern = logs.FilterPattern.all(
    logs.FilterPattern.string_value("$.component", "=", "HttpServer"),
    logs.FilterPattern.any(
        logs.FilterPattern.boolean_value("$.error", True),
        logs.FilterPattern.number_value("$.latency", ">", 1000)),
    logs.FilterPattern.regex_value("$.message", "=", "bind address already in use"))

// Search for all events where the component field is equal to
// "HttpServer" and either error is true or the latency is higher
// than 1000.
JsonPattern pattern = FilterPattern.all(FilterPattern.stringValue("$.component", "=", "HttpServer"), FilterPattern.any(FilterPattern.booleanValue("$.error", true), FilterPattern.numberValue("$.latency", ">", 1000)), FilterPattern.regexValue("$.message", "=", "bind address already in use"));

// Search for all events where the component field is equal to
// "HttpServer" and either error is true or the latency is higher
// than 1000.
var pattern = FilterPattern.All(FilterPattern.StringValue("$.component", "=", "HttpServer"), FilterPattern.Any(FilterPattern.BooleanValue("$.error", true), FilterPattern.NumberValue("$.latency", ">", 1000)), FilterPattern.RegexValue("$.message", "=", "bind address already in use"));

// Search for all events where the component field is equal to
// "HttpServer" and either error is true or the latency is higher
// than 1000.
pattern := logs.FilterPattern_All(logs.FilterPattern_StringValue(jsii.String("$.component"), jsii.String("="), jsii.String("HttpServer")), logs.FilterPattern_Any(logs.FilterPattern_BooleanValue(jsii.String("$.error"), jsii.Boolean(true)), logs.FilterPattern_NumberValue(jsii.String("$.latency"), jsii.String(">"), jsii.Number(1000))), logs.FilterPattern_RegexValue(jsii.String("$.message"), jsii.String("="), jsii.String("bind address already in use")))

Space-delimited table patterns

If the log events are rows of a space-delimited table, this pattern can be used to identify the columns in that structure and add conditions on any of them. The canonical example where you would apply this type of pattern is Apache server logs.

Text that is surrounded by "..." quotes or [...] square brackets will be treated as one column.

• FilterPattern.spaceDelimited(column, column, ...): construct a SpaceDelimitedTextPattern object with the indicated columns. The columns map one-by-one the columns found in the log event. The string "..." may be used to specify an arbitrary number of unnamed columns anywhere in the name list (but may only be specified once).

After constructing a SpaceDelimitedTextPattern, you can use the following two members to add restrictions:

• pattern.whereString(field, comparison, string): add a string condition. The rules are the same as for JSON patterns.
• pattern.whereNumber(field, comparison, number): add a numerical condition. The rules are the same as for JSON patterns.

Multiple restrictions can be added on the same column; they must all apply.

Example:

// Search for all events where the component is "HttpServer" and the
// result code is not equal to 200.
const pattern = logs.FilterPattern.spaceDelimited('time', 'component', '...', 'result_code', 'latency')
  .whereString('component', '=', 'HttpServer')
  .whereNumber('result_code', '!=', 200);

# Search for all events where the component is "HttpServer" and the
# result code is not equal to 200.
pattern = logs.FilterPattern.space_delimited("time", "component", "...", "result_code", "latency").where_string("component", "=", "HttpServer").where_number("result_code", "!=", 200)

// Search for all events where the component is "HttpServer" and the
// result code is not equal to 200.
SpaceDelimitedTextPattern pattern = FilterPattern.spaceDelimited("time", "component", "...", "result_code", "latency").whereString("component", "=", "HttpServer").whereNumber("result_code", "!=", 200);

// Search for all events where the component is "HttpServer" and the
// result code is not equal to 200.
var pattern = FilterPattern.SpaceDelimited("time", "component", "...", "result_code", "latency").WhereString("component", "=", "HttpServer").WhereNumber("result_code", "!=", 200);

// Search for all events where the component is "HttpServer" and the
// result code is not equal to 200.
pattern := logs.FilterPattern_SpaceDelimited(jsii.String("time"), jsii.String("component"), jsii.String("..."), jsii.String("result_code"), jsii.String("latency")).WhereString(jsii.String("component"), jsii.String("="), jsii.String("HttpServer")).WhereNumber(jsii.String("result_code"), jsii.String("!="), jsii.Number(200))

Logs Insights Query Definition

Creates a query definition for CloudWatch Logs Insights.

Example:

new logs.QueryDefinition(this, 'QueryDefinition', {
  queryDefinitionName: 'MyQuery',
  queryString: new logs.QueryString({
    fields: ['@timestamp', '@message'],
    parseStatements: [
      '@message "[*] *" as loggingType, loggingMessage',
      '@message "<*>: *" as differentLoggingType, differentLoggingMessage',
    ],
    filterStatements: [
      'loggingType = "ERROR"',
      'loggingMessage = "A very strange error occurred!"',
    ],
    statsStatements: [
      'count(loggingMessage) as loggingErrors',
      'count(differentLoggingMessage) as differentLoggingErrors',
    ],
    sort: '@timestamp desc',
    limit: 20,
  }),
});

logs.QueryDefinition(self, "QueryDefinition",
    query_definition_name="MyQuery",
    query_string=logs.QueryString(
        fields=["@timestamp", "@message"],
        parse_statements=["@message \"[*] *\" as loggingType, loggingMessage", "@message \"<*>: *\" as differentLoggingType, differentLoggingMessage"
        ],
        filter_statements=["loggingType = \"ERROR\"", "loggingMessage = \"A very strange error occurred!\""
        ],
        stats_statements=["count(loggingMessage) as loggingErrors", "count(differentLoggingMessage) as differentLoggingErrors"
        ],
        sort="@timestamp desc",
        limit=20
    )
)

QueryDefinition.Builder.create(this, "QueryDefinition")
        .queryDefinitionName("MyQuery")
        .queryString(QueryString.Builder.create()
                .fields(List.of("@timestamp", "@message"))
                .parseStatements(List.of("@message \"[*] *\" as loggingType, loggingMessage", "@message \"<*>: *\" as differentLoggingType, differentLoggingMessage"))
                .filterStatements(List.of("loggingType = \"ERROR\"", "loggingMessage = \"A very strange error occurred!\""))
                .statsStatements(List.of("count(loggingMessage) as loggingErrors", "count(differentLoggingMessage) as differentLoggingErrors"))
                .sort("@timestamp desc")
                .limit(20)
                .build())
        .build();

new QueryDefinition(this, "QueryDefinition", new QueryDefinitionProps {
    QueryDefinitionName = "MyQuery",
    QueryString = new QueryString(new QueryStringProps {
        Fields = new [] { "@timestamp", "@message" },
        ParseStatements = new [] { "@message \"[*] *\" as loggingType, loggingMessage", "@message \"<*>: *\" as differentLoggingType, differentLoggingMessage" },
        FilterStatements = new [] { "loggingType = \"ERROR\"", "loggingMessage = \"A very strange error occurred!\"" },
        StatsStatements = new [] { "count(loggingMessage) as loggingErrors", "count(differentLoggingMessage) as differentLoggingErrors" },
        Sort = "@timestamp desc",
        Limit = 20
    })
});

logs.NewQueryDefinition(this, jsii.String("QueryDefinition"), &QueryDefinitionProps{
	QueryDefinitionName: jsii.String("MyQuery"),
	QueryString: logs.NewQueryString(&QueryStringProps{
		Fields: []*string{
			jsii.String("@timestamp"),
			jsii.String("@message"),
		},
		ParseStatements: []*string{
			jsii.String("@message \"[*] *\" as loggingType, loggingMessage"),
			jsii.String("@message \"<*>: *\" as differentLoggingType, differentLoggingMessage"),
		},
		FilterStatements: []*string{
			jsii.String("loggingType = \"ERROR\""),
			jsii.String("loggingMessage = \"A very strange error occurred!\""),
		},
		StatsStatements: []*string{
			jsii.String("count(loggingMessage) as loggingErrors"),
			jsii.String("count(differentLoggingMessage) as differentLoggingErrors"),
		},
		Sort: jsii.String("@timestamp desc"),
		Limit: jsii.Number(20),
	}),
})

Data Protection Policy

Creates a data protection policy and assigns it to the log group. A data protection policy can help safeguard sensitive data that's ingested by the log group by auditing and masking the sensitive log data. When a user who does not have permission to view masked data views a log event that includes masked data, the sensitive data is replaced by asterisks.

For more information, see Protect sensitive log data with masking.

For a list of types of managed identifiers that can be audited and masked, see Types of data that you can protect.

If a new identifier is supported but not yet in the DataIdentifiers enum, the name of the identifier can be supplied as name in the constructor instead.

To add a custom data identifier, supply a custom name and regex to the CustomDataIdentifiers constructor. For more information on custom data identifiers, see Custom data identifiers.

Each policy may consist of a log group, S3 bucket, and/or Firehose delivery stream audit destination.

Example:

import * as firehose from 'aws-cdk-lib/aws-kinesisfirehose';

const logGroupDestination = new logs.LogGroup(this, 'LogGroupLambdaAudit', {
  logGroupName: 'auditDestinationForCDK',
});

const bucket = new s3.Bucket(this, 'audit-bucket');
const s3Destination = new firehose.S3Bucket(bucket);

const deliveryStream = new firehose.DeliveryStream(this, 'Delivery Stream', {
  destination: s3Destination,
});

const dataProtectionPolicy = new logs.DataProtectionPolicy({
  name: 'data protection policy',
  description: 'policy description',
  identifiers: [
    logs.DataIdentifier.DRIVERSLICENSE_US, // managed data identifier
    new logs.DataIdentifier('EmailAddress'), // forward compatibility for new managed data identifiers
    new logs.CustomDataIdentifier('EmployeeId', 'EmployeeId-\\d{9}')], // custom data identifier
  logGroupAuditDestination: logGroupDestination,
  s3BucketAuditDestination: bucket,
  deliveryStreamNameAuditDestination: deliveryStream.deliveryStreamName,
});

new logs.LogGroup(this, 'LogGroupLambda', {
  logGroupName: 'cdkIntegLogGroup',
  dataProtectionPolicy: dataProtectionPolicy,
});

import aws_cdk.aws_kinesisfirehose as firehose


log_group_destination = logs.LogGroup(self, "LogGroupLambdaAudit",
    log_group_name="auditDestinationForCDK"
)

bucket = s3.Bucket(self, "audit-bucket")
s3_destination = firehose.S3Bucket(bucket)

delivery_stream = firehose.DeliveryStream(self, "Delivery Stream",
    destination=s3_destination
)

data_protection_policy = logs.DataProtectionPolicy(
    name="data protection policy",
    description="policy description",
    identifiers=[logs.DataIdentifier.DRIVERSLICENSE_US,  # managed data identifier
        logs.DataIdentifier("EmailAddress"),  # forward compatibility for new managed data identifiers
        logs.CustomDataIdentifier("EmployeeId", "EmployeeId-\\d{9}")
    ],  # custom data identifier
    log_group_audit_destination=log_group_destination,
    s3_bucket_audit_destination=bucket,
    delivery_stream_name_audit_destination=delivery_stream.delivery_stream_name
)

logs.LogGroup(self, "LogGroupLambda",
    log_group_name="cdkIntegLogGroup",
    data_protection_policy=data_protection_policy
)

import software.amazon.awscdk.services.kinesisfirehose.*;


LogGroup logGroupDestination = LogGroup.Builder.create(this, "LogGroupLambdaAudit")
        .logGroupName("auditDestinationForCDK")
        .build();

Bucket bucket = new Bucket(this, "audit-bucket");
S3Bucket s3Destination = new S3Bucket(bucket);

DeliveryStream deliveryStream = DeliveryStream.Builder.create(this, "Delivery Stream")
        .destination(s3Destination)
        .build();

DataProtectionPolicy dataProtectionPolicy = DataProtectionPolicy.Builder.create()
        .name("data protection policy")
        .description("policy description")
        .identifiers(List.of(DataIdentifier.DRIVERSLICENSE_US,  // managed data identifier
            new DataIdentifier("EmailAddress"),  // forward compatibility for new managed data identifiers
            new CustomDataIdentifier("EmployeeId", "EmployeeId-\\d{9}"))) // custom data identifier
        .logGroupAuditDestination(logGroupDestination)
        .s3BucketAuditDestination(bucket)
        .deliveryStreamNameAuditDestination(deliveryStream.getDeliveryStreamName())
        .build();

LogGroup.Builder.create(this, "LogGroupLambda")
        .logGroupName("cdkIntegLogGroup")
        .dataProtectionPolicy(dataProtectionPolicy)
        .build();

using Amazon.CDK.AWS.KinesisFirehose;


var logGroupDestination = new LogGroup(this, "LogGroupLambdaAudit", new LogGroupProps {
    LogGroupName = "auditDestinationForCDK"
});

var bucket = new Bucket(this, "audit-bucket");
var s3Destination = new S3Bucket(bucket);

var deliveryStream = new DeliveryStream(this, "Delivery Stream", new DeliveryStreamProps {
    Destination = s3Destination
});

var dataProtectionPolicy = new DataProtectionPolicy(new DataProtectionPolicyProps {
    Name = "data protection policy",
    Description = "policy description",
    Identifiers = new [] { DataIdentifier.DRIVERSLICENSE_US,  // managed data identifier
        new DataIdentifier("EmailAddress"),  // forward compatibility for new managed data identifiers
        new CustomDataIdentifier("EmployeeId", "EmployeeId-\\d{9}") },  // custom data identifier
    LogGroupAuditDestination = logGroupDestination,
    S3BucketAuditDestination = bucket,
    DeliveryStreamNameAuditDestination = deliveryStream.DeliveryStreamName
});

new LogGroup(this, "LogGroupLambda", new LogGroupProps {
    LogGroupName = "cdkIntegLogGroup",
    DataProtectionPolicy = dataProtectionPolicy
});

import "github.com/aws/aws-cdk-go/awscdk"


logGroupDestination := logs.NewLogGroup(this, jsii.String("LogGroupLambdaAudit"), &LogGroupProps{
	LogGroupName: jsii.String("auditDestinationForCDK"),
})

bucket := s3.NewBucket(this, jsii.String("audit-bucket"))
s3Destination := firehose.NewS3Bucket(bucket)

deliveryStream := firehose.NewDeliveryStream(this, jsii.String("Delivery Stream"), &DeliveryStreamProps{
	Destination: s3Destination,
})

dataProtectionPolicy := logs.NewDataProtectionPolicy(&DataProtectionPolicyProps{
	Name: jsii.String("data protection policy"),
	Description: jsii.String("policy description"),
	Identifiers: []DataIdentifier{
		logs.DataIdentifier_DRIVERSLICENSE_US(),
		 // managed data identifier
		logs.NewDataIdentifier(jsii.String("EmailAddress")),
		 // forward compatibility for new managed data identifiers
		logs.NewCustomDataIdentifier(jsii.String("EmployeeId"), jsii.String("EmployeeId-\\d{9}")),
	},
	 // custom data identifier
	LogGroupAuditDestination: logGroupDestination,
	S3BucketAuditDestination: bucket,
	DeliveryStreamNameAuditDestination: deliveryStream.deliveryStreamName,
})

logs.NewLogGroup(this, jsii.String("LogGroupLambda"), &LogGroupProps{
	LogGroupName: jsii.String("cdkIntegLogGroup"),
	DataProtectionPolicy: dataProtectionPolicy,
})

Configure Deletion Protection

Indicates whether deletion protection is enabled for this log group. When enabled, deletion protection blocks all deletion operations until it is explicitly disabled.

For more information, see Protecting log groups from deletion.

new logs.LogGroup(this, 'LogGroup', {
  deletionProtectionEnabled: true,
});

logs.LogGroup(self, "LogGroup",
    deletion_protection_enabled=True
)

LogGroup.Builder.create(this, "LogGroup")
        .deletionProtectionEnabled(true)
        .build();

new LogGroup(this, "LogGroup", new LogGroupProps {
    DeletionProtectionEnabled = true
});

logs.NewLogGroup(this, jsii.String("LogGroup"), &LogGroupProps{
	DeletionProtectionEnabled: jsii.Boolean(true),
})

Field Index Policies

Creates or updates a field index policy for the specified log group. You can use field index policies to create field indexes on fields found in log events in the log group. Creating field indexes lowers the costs for CloudWatch Logs Insights queries that reference those field indexes, because these queries attempt to skip the processing of log events that are known to not match the indexed field. Good fields to index are fields that you often need to query for and fields that have high cardinality of values.

For more information, see Create field indexes to improve query performance and reduce costs.

Only log groups in the Standard log class support field index policies. Currently, this array supports only one field index policy object.

Example:

const fieldIndexPolicy = new logs.FieldIndexPolicy({
  fields: ['Operation', 'RequestId'],
});

new logs.LogGroup(this, 'LogGroup', {
  logGroupName: 'cdkIntegLogGroup',
  fieldIndexPolicies: [fieldIndexPolicy],
});

field_index_policy = logs.FieldIndexPolicy(
    fields=["Operation", "RequestId"]
)

logs.LogGroup(self, "LogGroup",
    log_group_name="cdkIntegLogGroup",
    field_index_policies=[field_index_policy]
)

FieldIndexPolicy fieldIndexPolicy = FieldIndexPolicy.Builder.create()
        .fields(List.of("Operation", "RequestId"))
        .build();

LogGroup.Builder.create(this, "LogGroup")
        .logGroupName("cdkIntegLogGroup")
        .fieldIndexPolicies(List.of(fieldIndexPolicy))
        .build();

var fieldIndexPolicy = new FieldIndexPolicy(new FieldIndexPolicyProps {
    Fields = new [] { "Operation", "RequestId" }
});

new LogGroup(this, "LogGroup", new LogGroupProps {
    LogGroupName = "cdkIntegLogGroup",
    FieldIndexPolicies = new [] { fieldIndexPolicy }
});

fieldIndexPolicy := logs.NewFieldIndexPolicy(&FieldIndexPolicyProps{
	Fields: []*string{
		jsii.String("Operation"),
		jsii.String("RequestId"),
	},
})

logs.NewLogGroup(this, jsii.String("LogGroup"), &LogGroupProps{
	LogGroupName: jsii.String("cdkIntegLogGroup"),
	FieldIndexPolicies: []FieldIndexPolicy{
		fieldIndexPolicy,
	},
})

Transformer

A log transformer enables transforming log events into a different format, making them easier to process and analyze. You can transform logs from different sources into standardized formats that contain relevant, source-specific information. Transformations are performed at the time of log ingestion. Transformers support several types of processors which can be chained into a processing pipeline (subject to some restrictions, see Usage Limits).

Processor Types

1. Parser Processors: Parse string log events into structured log events. These are configurable parsers created using ParserProcessor, and support conversion to a format like Json, extracting fields from CSV input, converting vended sources to OCSF format, regex parsing using Grok patterns or key-value parsing. Refer configurable parsers for more examples.

2. Vended Log Parsers: Parse log events from vended sources into structured log events. These are created using VendedLogParser, and support conversion from sources such as AWS WAF, PostGres, Route53, CloudFront and VPC. These parsers are not configurable, meaning these can be added to the pipeline but do not accept any properties or configurations. Refer vended log parsers for more examples.

3. String Mutators: Perform operations on string values in a field of a log event and are created using StringMutatorProcessor. These can be used to format string values in the log event such as changing case, removing trailing whitespaces or extracting values from a string field by splitting the string or regex backreferences. Refer string mutators for more examples.

4. JSON Mutators: Perform operation on JSON log events and are created using JsonMutatorProcessor. These processors can be used to enrich log events by adding new fields, deleting, moving, renaming fields, copying values to other fields or converting a list of key-value pairs to a map. Refer JSON mutators for more examples.

5. Data Converters: Convert the data into different formats and are created using DataConverterProcessor. These can be used to convert values in a field to datatypes such as integers, string, double and boolean or to convert dates and times to different formats. Refer datatype processors for more examples.

Usage Limits

• A transformer can have a maximum of 20 processors
• At least one parser-type processor is required
• Maximum of 5 parser-type processors allowed
• AWS vended log parser (if used) must be the first processor
• Only one parseToOcsf processor, one grok processor, one addKeys processor, and one copyValue processor allowed per transformer
• Transformers can only be used with log groups in the Standard log class

Example:

// Create a log group
const logGroup = new logs.LogGroup(this, 'MyLogGroup');

// Create a JSON parser processor
const jsonParser = new logs.ParserProcessor({
  type: logs.ParserProcessorType.JSON
});

// Create a processor to add keys
const addKeysProcessor = new logs.JsonMutatorProcessor({
  type: logs.JsonMutatorType.ADD_KEYS,
  addKeysOptions: {
    entries: [{
      key: 'metadata.transformed_in',
      value: 'CloudWatchLogs'
    }]
  }
});

// Create a transformer with these processors
new logs.Transformer(this, 'Transformer', {
  transformerName: 'MyTransformer',
  logGroup: logGroup,
  transformerConfig: [jsonParser, addKeysProcessor]
});

# Create a log group
log_group = logs.LogGroup(self, "MyLogGroup")

# Create a JSON parser processor
json_parser = logs.ParserProcessor(
    type=logs.ParserProcessorType.JSON
)

# Create a processor to add keys
add_keys_processor = logs.JsonMutatorProcessor(
    type=logs.JsonMutatorType.ADD_KEYS,
    add_keys_options=logs.AddKeysProperty(
        entries=[logs.AddKeyEntryProperty(
            key="metadata.transformed_in",
            value="CloudWatchLogs"
        )]
    )
)

# Create a transformer with these processors
logs.Transformer(self, "Transformer",
    transformer_name="MyTransformer",
    log_group=log_group,
    transformer_config=[json_parser, add_keys_processor]
)

// Create a log group
LogGroup logGroup = new LogGroup(this, "MyLogGroup");

// Create a JSON parser processor
ParserProcessor jsonParser = ParserProcessor.Builder.create()
        .type(ParserProcessorType.JSON)
        .build();

// Create a processor to add keys
JsonMutatorProcessor addKeysProcessor = JsonMutatorProcessor.Builder.create()
        .type(JsonMutatorType.ADD_KEYS)
        .addKeysOptions(AddKeysProperty.builder()
                .entries(List.of(AddKeyEntryProperty.builder()
                        .key("metadata.transformed_in")
                        .value("CloudWatchLogs")
                        .build()))
                .build())
        .build();

// Create a transformer with these processors
// Create a transformer with these processors
Transformer.Builder.create(this, "Transformer")
        .transformerName("MyTransformer")
        .logGroup(logGroup)
        .transformerConfig(List.of(jsonParser, addKeysProcessor))
        .build();

// Create a log group
var logGroup = new LogGroup(this, "MyLogGroup");

// Create a JSON parser processor
var jsonParser = new ParserProcessor(new ParserProcessorProps {
    Type = ParserProcessorType.JSON
});

// Create a processor to add keys
var addKeysProcessor = new JsonMutatorProcessor(new JsonMutatorProps {
    Type = JsonMutatorType.ADD_KEYS,
    AddKeysOptions = new AddKeysProperty {
        Entries = new [] { new AddKeyEntryProperty {
            Key = "metadata.transformed_in",
            Value = "CloudWatchLogs"
        } }
    }
});

// Create a transformer with these processors
// Create a transformer with these processors
new Transformer(this, "Transformer", new TransformerProps {
    TransformerName = "MyTransformer",
    LogGroup = logGroup,
    TransformerConfig = new [] { jsonParser, addKeysProcessor }
});

// Create a log group
logGroup := logs.NewLogGroup(this, jsii.String("MyLogGroup"))

// Create a JSON parser processor
jsonParser := logs.NewParserProcessor(&ParserProcessorProps{
	Type: logs.ParserProcessorType_JSON,
})

// Create a processor to add keys
addKeysProcessor := logs.NewJsonMutatorProcessor(&JsonMutatorProps{
	Type: logs.JsonMutatorType_ADD_KEYS,
	AddKeysOptions: &AddKeysProperty{
		Entries: []AddKeyEntryProperty{
			&AddKeyEntryProperty{
				Key: jsii.String("metadata.transformed_in"),
				Value: jsii.String("CloudWatchLogs"),
			},
		},
	},
})

// Create a transformer with these processors
// Create a transformer with these processors
logs.NewTransformer(this, jsii.String("Transformer"), &TransformerProps{
	TransformerName: jsii.String("MyTransformer"),
	LogGroup: logGroup,
	TransformerConfig: []IProcessor{
		jsonParser,
		addKeysProcessor,
	},
})

For more details on CloudWatch Logs transformation processors, refer to the AWS documentation.

Usage of metric filters on transformed logs

In order to use the transformed logs as search pattern, set the parameter applyOnTransformedLogs: true in the MetricFilterProps.

Notes

Be aware that Log Group ARNs will always have the string :* appended to them, to match the behavior of the CloudFormation AWS::Logs::LogGroup resource.

Types

Name	Kind	Summary
CrossAccountDestination	Construct (L2)	A new CloudWatch Logs Destination for use in cross-account scenarios.
LogGroup	Construct (L2)	Define a CloudWatch Log Group.
LogStream	Construct (L2)	Define a Log Stream in a Log Group.
MetricFilter	Construct (L2)	A filter that extracts information from CloudWatch Logs and emits to CloudWatch Metrics.
QueryDefinition	Construct (L2)	Define a query definition for CloudWatch Logs Insights.
ResourcePolicy	Construct (L2)	Resource Policy for CloudWatch Log Groups.
SubscriptionFilter	Construct (L2)	A new Subscription on a CloudWatch log group.
Transformer	Construct (L2)	Represent the L2 construct for the AWS::Logs::Transformer CloudFormation resource.
CfnAccountPolicy	Construct (L1)	Creates or updates an account-level data protection policy or subscription filter policy that applies to all log groups or a subset of log groups in the account.
CfnDelivery	Construct (L1)	This structure contains information about one delivery in your account.
CfnDeliveryDestination	Construct (L1)	This structure contains information about one delivery destination in your account.
CfnDeliverySource	Construct (L1)	Creates or updates one delivery source in your account.
CfnDestination	Construct (L1)	The AWS::Logs::Destination resource specifies a CloudWatch Logs destination.
CfnIntegration	Construct (L1)	Creates an integration between CloudWatch Logs and another service in this account.
CfnLogAnomalyDetector	Construct (L1)	Creates or updates an anomaly detector that regularly scans one or more log groups and look for patterns and anomalies in the logs.
CfnLogGroup	Construct (L1)	The AWS::Logs::LogGroup resource specifies a log group.
CfnLogStream	Construct (L1)	The AWS::Logs::LogStream resource specifies an Amazon CloudWatch Logs log stream in a specific log group.
CfnMetricFilter	Construct (L1)	The AWS::Logs::MetricFilter resource specifies a metric filter that describes how CloudWatch Logs extracts information from logs and transforms it into Amazon CloudWatch metrics.
CfnQueryDefinition	Construct (L1)	Creates a query definition for CloudWatch Logs Insights.
CfnResourcePolicy	Construct (L1)	Creates or updates a resource policy that allows other AWS services to put log events to this account.
CfnScheduledQuery	Construct (L1)	Creates a new Scheduled Query that allows you to define a Logs Insights query that will run on a schedule and configure actions to take with the query results.
CfnStorageTierPolicy	Construct (L1)	Resource Type definition for AWS::Logs::StorageTierPolicy.
CfnSubscriptionFilter	Construct (L1)	The AWS::Logs::SubscriptionFilter resource specifies a subscription filter and associates it with the specified log group.
CfnTransformer	Construct (L1)	Creates or updates a log transformer for a single log group.
AddKeyEntryProperty	Struct	This object defines one key that will be added with the addKeys processor.
AddKeysProperty	Struct	This processor adds new key-value pairs to the log event.
BaseProcessorProps	Struct	Base properties for all processor types.
CfnAccountPolicyProps	Struct	Properties for defining a CfnAccountPolicy.
CfnDeliveryDestination.DestinationPolicyProperty	Struct	An IAM policy that grants permissions to CloudWatch Logs to deliver logs cross-account to a specified destination in this account.
CfnDeliveryDestinationProps	Struct	Properties for defining a CfnDeliveryDestination.
CfnDeliveryProps	Struct	Properties for defining a CfnDelivery.
CfnDeliverySourceProps	Struct	Properties for defining a CfnDeliverySource.
CfnDestinationProps	Struct	Properties for defining a CfnDestination.
CfnIntegration.OpenSearchResourceConfigProperty	Struct	This structure contains configuration details about an integration between CloudWatch Logs and OpenSearch Service.
CfnIntegration.ResourceConfigProperty	Struct	This structure contains configuration details about an integration between CloudWatch Logs and another entity.
CfnIntegrationProps	Struct	Properties for defining a CfnIntegration.
CfnLogAnomalyDetectorProps	Struct	Properties for defining a CfnLogAnomalyDetector.
CfnLogGroupProps	Struct	Properties for defining a CfnLogGroup.
CfnLogStreamProps	Struct	Properties for defining a CfnLogStream.
CfnMetricFilter.DimensionProperty	Struct	Specifies the CloudWatch metric dimensions to publish with this metric.
CfnMetricFilter.MetricTransformationProperty	Struct	MetricTransformation is a property of the AWS::Logs::MetricFilter resource that describes how to transform log streams into a CloudWatch metric.
CfnMetricFilterProps	Struct	Properties for defining a CfnMetricFilter.
CfnQueryDefinition.QueryParameterProperty	Struct	This structure defines a query parameter for a saved CloudWatch Logs Insights query definition.
CfnQueryDefinitionProps	Struct	Properties for defining a CfnQueryDefinition.
CfnResourcePolicyProps	Struct	Properties for defining a CfnResourcePolicy.
CfnScheduledQuery.DestinationConfigurationProperty	Struct
CfnScheduledQuery.S3ConfigurationProperty	Struct
CfnScheduledQuery.TagsItemsProperty	Struct
CfnScheduledQueryProps	Struct	Properties for defining a CfnScheduledQuery.
CfnStorageTierPolicyProps	Struct	Properties for defining a CfnStorageTierPolicy.
CfnSubscriptionFilterProps	Struct	Properties for defining a CfnSubscriptionFilter.
CfnTransformer.AddKeyEntryProperty	Struct	This object defines one key that will be added with the addKeys processor.
CfnTransformer.AddKeysProperty	Struct	This processor adds new key-value pairs to the log event.
CfnTransformer.CopyValueEntryProperty	Struct	This object defines one value to be copied with the copyValue processor.
CfnTransformer.CopyValueProperty	Struct	This processor copies values within a log event.
CfnTransformer.CsvProperty	Struct	The CSV processor parses comma-separated values (CSV) from the log events into columns.
CfnTransformer.DateTimeConverterProperty	Struct	This processor converts a datetime string into a format that you specify.
CfnTransformer.DeleteKeysProperty	Struct	This processor deletes entries from a log event. These entries are key-value pairs.
CfnTransformer.GrokProperty	Struct	This processor uses pattern matching to parse and structure unstructured data.
CfnTransformer.ListToMapProperty	Struct	This processor takes a list of objects that contain key fields, and converts them into a map of target keys.
CfnTransformer.LowerCaseStringProperty	Struct	This processor converts a string to lowercase.
CfnTransformer.MoveKeyEntryProperty	Struct	This object defines one key that will be moved with the moveKey processor.
CfnTransformer.MoveKeysProperty	Struct	This processor moves a key from one field to another. The original key is deleted.
CfnTransformer.ParseCloudfrontProperty	Struct	This processor parses CloudFront vended logs, extract fields, and convert them into JSON format.
CfnTransformer.ParseJSONProperty	Struct	This processor parses log events that are in JSON format.
CfnTransformer.ParseKeyValueProperty	Struct	This processor parses a specified field in the original log event into key-value pairs.
CfnTransformer.ParsePostgresProperty	Struct	Use this processor to parse RDS for PostgreSQL vended logs, extract fields, and and convert them into a JSON format.
CfnTransformer.ParseRoute53Property	Struct	Use this processor to parse Route 53 vended logs, extract fields, and and convert them into a JSON format.
CfnTransformer.ParseToOCSFProperty	Struct	This processor converts logs into Open Cybersecurity Schema Framework (OCSF) events.
CfnTransformer.ParseVPCProperty	Struct	Use this processor to parse Amazon VPC vended logs, extract fields, and and convert them into a JSON format.
CfnTransformer.ParseWAFProperty	Struct	Use this processor to parse AWS WAF vended logs, extract fields, and and convert them into a JSON format.
CfnTransformer.ProcessorProperty	Struct	This structure contains the information about one processor in a log transformer.
CfnTransformer.RenameKeyEntryProperty	Struct	This object defines one key that will be renamed with the renameKey processor.
CfnTransformer.RenameKeysProperty	Struct	Use this processor to rename keys in a log event.
CfnTransformer.SplitStringEntryProperty	Struct	This object defines one log field that will be split with the splitString processor.
CfnTransformer.SplitStringProperty	Struct	Use this processor to split a field into an array of strings using a delimiting character.
CfnTransformer.SubstituteStringEntryProperty	Struct	This object defines one log field key that will be replaced using the substituteString processor.
CfnTransformer.SubstituteStringProperty	Struct	This processor matches a key’s value against a regular expression and replaces all matches with a replacement string.
CfnTransformer.TrimStringProperty	Struct	Use this processor to remove leading and trailing whitespace.
CfnTransformer.TypeConverterEntryProperty	Struct	This object defines one value type that will be converted using the typeConverter processor.
CfnTransformer.TypeConverterProperty	Struct	Use this processor to convert a value type associated with the specified key to the specified type.
CfnTransformer.UpperCaseStringProperty	Struct	This processor converts a string field to uppercase.
CfnTransformerProps	Struct	Properties for defining a CfnTransformer.
ColumnRestriction	Struct
CopyValueEntryProperty	Struct	This object defines one value to be copied with the copyValue processor.
CopyValueProperty	Struct	Copy Value processor, copies values from source to target for each entry.
CrossAccountDestinationProps	Struct	Properties for a CrossAccountDestination.
CsvProperty	Struct	The CSV processor parses comma-separated values (CSV) from the log events into columns.
CustomDataIdentifier	Class	A custom data identifier.
DataConverterProcessor	Class	Processor for data conversion operations.
DataConverterProps	Struct	Properties for creating data converter processors.
DataConverterType	Enum	Types of data conversion operations.
DataIdentifier	Class	A data protection identifier.
DataProtectionPolicy	Class	Creates a data protection policy for CloudWatch Logs log groups.
DataProtectionPolicyProps	Struct	Properties for creating a data protection policy.
DateTimeConverterProperty	Struct	This processor converts a datetime string into a format that you specify.
DateTimeFormat	Enum	Standard datetime formats for the DateTimeConverter processor.
DelimiterCharacter	Enum	Valid delimiter characters for CSV processor.
Distribution	Enum	The method used to distribute log data to the destination.
FieldIndexPolicy	Class	Creates a field index policy for CloudWatch Logs log groups.
FieldIndexPolicyProps	Struct	Properties for creating field index policies.
FilterPattern	Class	A collection of static methods to generate appropriate ILogPatterns.
GrokProperty	Struct	This processor uses pattern matching to parse and structure unstructured data.
IFilterPattern	Interface	Interface for objects that can render themselves to log patterns.
ILogGroup	Interface
ILogStream	Interface
ILogSubscriptionDestination	Interface	Interface for classes that can be the destination of a log Subscription.
IProcessor	Interface	Interface representing a single processor in a CloudWatch Logs transformer.
JsonMutatorProcessor	Class	Processor for JSON mutation operations.
JsonMutatorProps	Struct	Properties for creating JSON mutator processors.
JsonMutatorType	Enum	Types of JSON mutation operations.
JsonPattern	Class	Base class for patterns that only match JSON log events.
KeyValueDelimiter	Enum	Valid key-value delimiters for ParseKeyValue processor.
KeyValuePairDelimiter	Enum	Valid field delimiters for ParseKeyValue processor.
ListToMapProperty	Struct	This processor takes a list of objects that contain key fields, and converts them into a map of target keys.
LogGroupClass	Enum	Class of Log Group.
LogGroupGrants	Facade	Collection of grant methods for a ILogGroupRef.
LogGroupProps	Struct	Properties for a LogGroup.
LogRetention	Construct	Creates a custom resource to control the retention policy of a CloudWatch Logs log group.
LogRetentionProps	Struct	Construction properties for a LogRetention.
LogRetentionRetryOptions	Struct	Retry options for all AWS API calls.
LogStreamAttributes	Struct	Attributes for importing a LogStream.
LogStreamProps	Struct	Properties for a LogStream.
LogSubscriptionDestinationConfig	Struct	Properties returned by a Subscription destination.
MetricFilterOptions	Struct	Properties for a MetricFilter created from a LogGroup.
MetricFilterProps	Struct	Properties for a MetricFilter.
MoveKeyEntryProperty	Struct	This object defines one key that will be moved with the moveKey processor.
MoveKeysProperty	Struct	This processor moves a key from one field to another.
OCSFSourceType	Enum	Types of event sources supported to convert to OCSF format.
OCSFVersion	Enum	OCSF Schema versions supported by transformers.
ParseJSONProperty	Struct	This processor parses log events that are in JSON format.
ParseKeyValueProperty	Struct	This processor parses a specified field in the original log event into key-value pairs.
ParserProcessor	Class	Parser processor for common data formats.
ParserProcessorProps	Struct	Properties for creating configurable parser processors.
ParserProcessorType	Enum	Types of configurable parser processors.
ParseToOCSFProperty	Struct	Processor to parse events from CloudTrail, Route53Resolver, VPCFlow, EKSAudit and AWSWAF into OCSF V1.1 format.
ProcessorDeleteKeysProperty	Struct	This processor adds new key-value pairs to the log event.
QueryDefinitionProps	Struct	Properties for a QueryDefinition.
QueryString	Class	Define a QueryString.
QueryStringProps	Struct	Properties for a QueryString.
QuoteCharacter	Enum	Valid quote characters for CSV processor.
RenameKeyEntryProperty	Struct	This object defines one key that will be renamed with the renameKey processor.
RenameKeysProperty	Struct	Use this processor to rename keys in a log event.
ResourcePolicyProps	Struct	Properties to define Cloudwatch log group resource policy.
RetentionDays	Enum	How long, in days, the log contents will be retained.
SpaceDelimitedTextPattern	Class	Space delimited text pattern.
SplitStringEntryProperty	Struct	This object defines one log field that will be split with the splitString processor.
SplitStringProperty	Struct	Use this processor to split a field into an array of strings using a delimiting character.
StreamOptions	Struct	Properties for a new LogStream created from a LogGroup.
StringMutatorProcessor	Class	Processor for string mutation operations.
StringMutatorProps	Struct	Properties for creating string mutator processors.
StringMutatorType	Enum	Types of string mutation operations.
SubscriptionFilterOptions	Struct	Properties for a new SubscriptionFilter created from a LogGroup.
SubscriptionFilterProps	Struct	Properties for a SubscriptionFilter.
SubstituteStringEntryProperty	Struct	This object defines one log field key that will be replaced using the substituteString processor.
SubstituteStringProperty	Struct	This processor matches a key's value against a regular expression and replaces all matches with a replacement string.
TransformerOptions	Struct	Properties for Transformer created from LogGroup.
TransformerProps	Struct	The Resource properties for AWS::Logs::Transformer resource.
TypeConverterEntryProperty	Struct	This object defines one value type that will be converted using the typeConverter processor.
TypeConverterProperty	Struct	Use this processor to convert a value type associated with the specified key to the specified type.
TypeConverterType	Enum	Valid data types for type conversion in the TypeConverter processor.
VendedLogParser	Class	Parser processor for AWS vended logs.
VendedLogParserProps	Struct	Properties for creating AWS vended log parsers.
VendedLogType	Enum	Types of AWS vended logs with built-in parsers.