# CloudWatch pipelines CloudWatch pipelines is a fully managed data collector that ingests, transforms, and routes log data from AWS services, third-party applications, and custom sources to CloudWatch. Built-in processors let you enrich, filter, and standardize logs into formats like OCSF—without managing infrastructure or third-party tools. CloudWatch pipelines is fully integrated with the [logs management experience](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html), enabling you to consistently process and enrich log data across related log groups via data-source and data-type specification. This unlocks use cases such as: + **Automatic log categorization** – Logs processed through pipelines are automatically tagged with data source information, enabling service-centric discovery and querying across your infrastructure + **Expanding third-party support** – Aggregate and normalize logs from a growing library of third-party sources for unified analytics and compliance Output from pipelines is fully compatible with CloudWatch Logs features including Logs Insights queries, Logs Anomaly Detection, and Live Tail. CloudWatch pipelines works with both Standard and Infrequent Access log classes and is backwards compatible with Log Transformers. To get started with CloudWatch pipelines, visit [pipelines within the CloudWatch ingestion page](https://console.aws.amazon.com/cloudwatch/home?#/telemetry-config:pipelines?useCase=All) in the CloudWatch console. **Note** Be aware of the following limits that apply to CloudWatch pipelines Maximum number of pipelines per account: 330 Up to 300 pipelines for collecting data from CloudWatch Logs Up to 30 pipelines for collecting data from other sources ## Pipeline components Each pipeline consists of the following components: + **Source** – Defines where data originates from (Amazon S3 buckets, CloudWatch Logs, third-party integration). Each pipeline must have exactly one source. + **Processors** (optional) – Transform, parse, and enrich log data as it flows through the pipeline. Processors are applied sequentially in the order they are defined. + **Sink** – Defines the destination where processed log data is sent. Each pipeline must have exactly one sink. + **Extensions** (optional) – Provide additional functionality such as AWS Secrets Manager integration for credential management. Throughout the entire pipeline, your data remains protected with transport layer encryption, ensuring security and compliance requirements are met. **Note** Pipeline definitions are not encrypted and should never include sensitive data, such as personally identifiable information (PII). **Note** Adding processors leads to mutation of the log events and original (raw) logs are not retained. ## Pricing CloudWatch pipelines is included with CloudWatch Logs at no additional cost. Standard log ingestion rates based on log class (vended or custom) and storage class (Standard or Infrequent Access) still apply. Metering occurs at time of first ingestion into CloudWatch. CloudWatch Logs sources are metered before pipeline processing. Third-party and S3 bucket sources are classified as Custom logs and metered after processing. For pricing details, see [CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). ## Region availability CloudWatch pipelines is available in the following AWS Regions: + US East (N. Virginia) + US East (Ohio) + US West (N. California) + US West (Oregon) + Africa (Cape Town) + Asia Pacific (Hong Kong) + Asia Pacific (Hyderabad) + Asia Pacific (Jakarta) + Asia Pacific (Malaysia) + Asia Pacific (Melbourne) + Asia Pacific (Mumbai) + Asia Pacific (New Zealand) + Asia Pacific (Osaka) + Asia Pacific (Seoul) + Asia Pacific (Singapore) + Asia Pacific (Sydney) + Asia Pacific (Taipei) + Asia Pacific (Thailand) + Asia Pacific (Tokyo) + Canada (Central) + Canada West (Calgary) + Europe (Frankfurt) + Europe (Ireland) + Europe (London) + Europe (Milan) + Europe (Paris) + Europe (Spain) + Europe (Stockholm) + Europe (Zurich) + Israel (Tel Aviv) + Mexico (Central) + South America (São Paulo) **Note** Third-party data source collection is available in regions where [OpenSearch Ingestion has API endpoints](https://docs.aws.amazon.com/general/latest/gr/opensearch-service.html). For more details, see [Amazon CloudWatch endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/cw_region.html) in the *AWS General Reference*. **Topics** # Creating pipelines The pipeline configuration wizard guides you through creating your data pipeline. 1. Under **General settings**, provide the data source details including source name and type. You can also specify pipeline tags and the name of your pipeline. 1. Under **Destination**, specify the destination details. CloudWatch Logs is the default destination. 1. Under **Processor**, add the desired processors and parsers. A parser is a required first step for certain data types. You can perform custom parsing using processors like Grok or CSV. Processors that are not supported by the data type are disabled. You can also add conditional processing rules to supported processors using the `when` parameter. Conditional processing lets you control which log entries a processor acts on. For the expression syntax and supported processors, see [Expression syntax for conditional processing](conditional-processing.md). To preserve unmodified copies of your log data for audit or compliance purposes, enable the **Keep original log** toggle. When enabled, CloudWatch pipelines automatically stores a copy of each raw log event before any transformation takes place. This ensures that original data is always available for audits or investigations, even after processors modify the log events. The **Keep original log** toggle is only available for pipelines with a CloudWatch vended log source. 1. Under **Review and create**, review the pipeline configuration. If you're satisfied with the configuration, choose **Create pipeline** to start deployment and creation of pipeline resources. Pipeline creation completion takes up to 5 minutes depending on the source type. Upon completion, you'll be taken to the Pipelines tab in the Ingestion Console. **Important** Pipeline processor configurations are logged in AWS CloudTrail events for auditing and compliance purposes. To protect sensitive information, do not include passwords, API keys, or other sensitive information in processor configurations. # Managing pipelines After creating pipelines, you can monitor their performance and manage their configuration through the Pipelines tab. **Pipeline status monitoring** Each pipeline displays real-time status information including: + Processing status (Active, Inactive, Error) + Data throughput metrics + Error rates and failure details **Pipeline operations** You can perform the following operations on existing pipelines: + **View details** – Review pipeline configuration and status + **Edit pipeline** – Edit pipeline configuration to add additional processing or parsing + **Delete pipeline** – Remove pipelines that are no longer needed # Supported data sources for CloudWatch CloudWatch collects and processes telemetry data from a wide range of sources to provide unified observability and security insights. Data sources fall into three categories: AWS native services, third-party platforms, and custom sources. You can automate the enablement of AWS data sources using telemetry configuration. For more information, see [Telemetry discovery and enablement](telemetry-config-cloudwatch.md). # AWS service data sources CloudWatch provides native integration with 90\$1 AWS services for automatic data collection. When an AWS service source is selected, CloudWatch pipelines intercepts logs ingested into CloudWatch Logs for processing. To get started, enable logging for the supported AWS services using the service's console, then select the data source and type in the CloudWatch pipelines creation wizard. The following table highlights key AWS service data sources. For the complete list of 90\$1 supported services, see [Supported AWS services for data sources](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/supported-aws-services-data-sources.html). **Key AWS service data sources** | AWS service | Data type | Description | | --- | --- | --- | | Amazon Amazon VPC | Flow Logs | Network traffic metadata for Amazon VPCs, subnets, and network interfaces | | Amazon Amazon EKS | Control Plane Logs | Kubernetes API server, audit, authenticator, controller manager, and scheduler logs | | AWS WAF | Web ACL Logs | Web request inspection logs including rule match details and actions taken | | Amazon Route 53 | Resolver Query Logs | DNS query logs for Amazon VPC resources routed through Route 53 Resolver | | CloudTrail | Management and Data Events | API activity and resource-level operations across AWS services | | Amazon Amazon EC2 | Detailed Metrics | Instance-level performance metrics at 1-minute granularity | | AWS Security Hub | CSPM Findings | Cloud security posture management findings from AWS and third-party providers | | Amazon Bedrock AgentCore | Runtime, Browser, CodeInterpreter, Gateway, Memory | Agent runtime execution, browser interaction, code execution, gateway, and memory operation logs | | Amazon CloudFront | Distribution Logs | CDN access logs for content delivery distributions | | Network Load Balancer | Access Logs | Network Load Balancer connection and TLS negotiation logs | For more information about CloudWatch Logs data sources, see [Data source discovery and management](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html). # Third-party data sources CloudWatch extends monitoring capabilities beyond AWS with direct integrations for 13 third-party security, identity, and endpoint platforms. These integrations consolidate security events, audit logs, and telemetry data from external sources into CloudWatch Logs for unified analysis. The following table lists the supported direct third-party integrations: **Direct third-party integrations** | Source | Integration pattern | Category | | --- | --- | --- | | CrowdStrike Falcon | S3 Delivery | Endpoint security | | SentinelOne | S3 Delivery | Endpoint security | | Microsoft Office 365 | API | Productivity and audit logs | | Microsoft Entra ID | API | Identity and access management | | Microsoft Windows Event Logs | API | Operating system events | | Okta Auth0 | API | Identity and access management | | Okta SSO | API | Identity and access management | | Palo Alto Networks NGFW | API | Network security | | Wiz CNAPP | API | Cloud security | | Zscaler ZIA/ZPA | S3 Delivery | Network security | | GitHub | API | Source code and audit logs | | ServiceNow CMDB | API | IT service management | | Cisco Umbrella | S3 Delivery | DNS and network security | For detailed setup procedures, prerequisites, and configuration steps for each integration, see [Third-party data sources integration](third-party-integration-setup.md). **Additional third-party sources through Security Hub CSPM** Beyond the 13 direct integrations, 49\$1 additional third-party sources are available through AWS Security Hub CSPM integration. Security Hub partner providers that send findings to Security Hub are automatically available as data sources. For the full list of supported partners, see the [Security Hub partner providers](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-partner-providers.html) documentation. # Custom data sources For logs that are not covered by AWS service or third-party integrations, CloudWatch pipelines can process custom logs stored in CloudWatch Logs or Amazon S3 buckets. Custom sources accommodate unique organizational requirements: + **Application-specific logs** – Custom application telemetry from Amazon EC2 instances with specialized logging formats + **File-based ingestion** – Amazon S3-based log files from legacy systems or batch processing workflows + **Serverless integration** – Lambda function logs and custom serverless application telemetry For more details, see [Custom log data from CloudWatch Logs or an Amazon S3 bucket](ingestion-custom-data-sources.md). **Topics** + [ # AWS service data sources ](data-sources-aws-services.md) + [ # Third-party data sources ](data-sources-third-party.md) + [ # Custom data sources ](data-sources-custom.md) + [ # Third-party data sources integration ](third-party-integration-setup.md) + [ # Custom log data from CloudWatch Logs or an Amazon S3 bucket ](ingestion-custom-data-sources.md) + [ # Configuring Custom S3 Bucket Sources ](configuring-custom-s3-bucket-sources.md) + [ # AWS service logs from CloudWatch Logs ](aws-service-logs-from-cwl.md) # Third-party data sources integration Integrating CloudWatch pipelines with your third-party data source let you connect external security tools, identity providers, and monitoring platforms to CloudWatch pipelines for centralized data analysis. This integration consolidates security events, audit logs, and telemetry data from multiple sources. **Note** Data collected from third-party sources is mutated to adhere to the required schema when it is collected by CloudWatch pipelines. The original data source is not retained by CloudWatch. Third-party data can be collected using two methods: 1. **Direct API Integration** – Some sources offer Event Stream APIs where you only need to provide API credentials to configure the connector 1. **S3 Bucket Integration** – Data from sources can be ingested into a customer-managed S3 bucket for CloudWatch pipelines to collect The following table identified the integrations methods used by the supported third-party data platforms: | Source | Integration Pattern | Requires S3 bucket | Requires SQS Queue | Uses Secrets Manager extension | Required IAM Policies | | --- | --- | --- | --- | --- | --- | | CrowdStrike Falcon | S3 Delivery | Yes | Yes | No | [Source-specific IAM policies](pipeline-iam-reference.md#source-specific-iam-policies) | | Microsoft Office 365 | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Okta Auth0 | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Microsoft Entra ID | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Palo Alto Networks Next Generation Firewall | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Microsoft Windows Event Logs | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Wiz CNAPP | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Zscaler ZIA/ZPA | S3 Delivery | Yes | Yes | No | [Source-specific IAM policies](pipeline-iam-reference.md#source-specific-iam-policies) | | Okta SSO | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | SentinelOne | S3 Delivery | Yes | Yes | No | [Source-specific IAM policies](pipeline-iam-reference.md#source-specific-iam-policies) | | GitHub | API | No | No | Yes (optional) | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | ServiceNow CMDB | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Cisco Umbrella | S3 Delivery | Yes | Yes | No | [Source-specific IAM policies](pipeline-iam-reference.md#source-specific-iam-policies) | | PingIdentity PingOne | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | OneLogin Identity | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Entrust IDaaS | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | | Drupal Core | API | No | No | Yes | [API caller permissions](pipeline-iam-reference.md#api-caller-permissions) | Third-party integration sent through Security Hub CSPM is also supported. For comprehensive information about Security Hub's third-party integrations, including supported partners and integration configurations with "Sends findings" direction, refer to the [Security Hub Third-Party Integration documentation](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-partner-providers.html). **Data transformation and standardization** Third-party integrations support data transformation to standardized formats for consistent analysis: + **Open Cybersecurity Schema Framework (OCSF)** – Converts security events from different vendors into a common schema for unified threat detection and analysis. Because OCSF is only for certain event classes, not all raw events are mapped to OCSF. + **Custom transformations** – Pipeline processors that normalize data formats, enrich events with additional context, and filter relevant information. + **Field mapping** – Automatic mapping of vendor-specific fields to standardized field names for consistent querying and analysis. **Note** Storing telemetry data from third-party sources in OCSF is an optional feature that might not be available for all data sources. **Log group** Third-party data is ingested into a CloudWatch log group. If you are using the AWS Management Console to configure CloudWatch pipelines if the log group does not exist, it is created automatically through the wizard process. **Authentication and security** Third-party integrations use secure authentication methods to protect data in transit: + **OAuth 2.0 and application registration** – Secure token-based authentication for cloud platforms like Microsoft and Okta. + **API keys and certificates** – Encrypted authentication credentials for direct API access. + **IAM roles and policies** – AWS Identity and Access Management integration for secure S3 bucket access and cross-account data sharing. **Note** Data collected from third-party sources is mutated to adhere to the required schema when it is collected by CloudWatch pipelines. The original data source is not retained by CloudWatch. Each integration requires platform-specific configuration to establish secure data delivery to your AWS environment. The following sections provide detailed setup procedures for supported third-party integrations. Each integration includes prerequisites, configuration steps, and validation procedures to ensure proper data flow. # CrowdStrike integration configuration To integrate CrowdStrike Falcon Data Replicator with CloudWatch Logs, you must configure both the source and the pipeline. First, set up your CrowdStrike source by configuring Amazon S3 and Amazon SQS to receive FDR data. Then, configure the CloudWatch pipeline to ingest the data from your source into CloudWatch Logs. **Topics** + [ # Source configuration for CrowdStrike ](crowdstrike-source-setup.md) + [ # CloudWatch pipelines configuration for CrowdStrike ](crowdstrike-pipeline-setup.md) # Source configuration for CrowdStrike ## Integrating with CrowdStrike Falcon CrowdStrike Falcon Data Replicator (FDR) delivers and enriches endpoint, cloud workload and identity data with the CrowdStrike Security Cloud and world-class artificial intelligence (AI), enabling your team to derive actionable insights to improve security operations center (SOC) performance. Amazon CloudWatch Logs enables you to collect this data in CloudWatch Logs. ## Instructions to setup Amazon S3 and Amazon SQS Configuring CrowdStrike FDR to send logs to an Amazon S3 bucket involves several steps, primarily focused on setting up the Amazon S3 bucket, Amazon SQS queue, IAM roles, and then configuring the Amazon Telemetry Pipeline. + Ensure CrowdStrike FDR is enabled within your CrowdStrike Falcon environment. This typically requires a specific license and may involve working with CrowdStrike support. + Amazon S3 bucket that stores the CrowdStrike logs should reside in the same AWS region where the FDR is enabled. + Configure the Amazon S3 bucket to create event notifications, specifically for "Object Create" events. These notifications should be sent to an Amazon SQS queue. + Create an Amazon SQS queue in the same AWS region as your Amazon S3 bucket. This queue will receive notifications when new log files are added to the Amazon S3 bucket. ## Configuring the CloudWatch Pipeline When configuring the pipeline to read data from CrowdStrike FDR, choose CrowdStrike as the data source. After filling in the required information and you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and the CrowdStrike FDR actions that maps to Detection Findings (2004) and Process Activity (1007). ### Detection Findings Detection Findings contains the following actions: + CloudAssociateTreeIdWithRoot + CustomIOADomainNameDetectionInfoEvent + TemplateDetectAnalysis ### Process Activity Process Activity contains the following actions: + ActiveDirectoryIncomingPsExecExecution2 + AndroidIntentSentIPC + AssociateTreeIdWithRoot + AutoRunProcessInfo + BamRegAppRunTime + BlockThreadFailed + BrowserInjectedThread + CidMigrationConfirmation + CodeSigningAltered + CommandHistory + CreateProcessArgs + CreateThreadNoStartImage + CriticalEnvironmentVariableChanged + CsUmProcessCrashAuxiliaryEvent + CsUmProcessCrashSummaryEvent + CustomIOABasicProcessDetectionInfoEvent + DebuggableFlagTurnedOn + DebuggedState + DllInjection + DocumentProgramInjectedThread + EarlyExploitPivotDetect + EndOfProcess + EnvironmentVariablesChanged + FalconProcessHandleOpDetectInfo + FlashThreadCreateProcess + IdpWatchdogRemediationActionTaken + InjectedThread + InjectedThreadFromUnsignedModule + IPCDetectInfo + JavaInjectedThread + KillProcessError + LsassHandleFromUnsignedModule + MacKnowledgeActivityEnd + MacKnowledgeActivityStart + NamespaceChanged + PcaAppLaunchEntry + PcaGeneralDbEntry + PrivilegedProcessHandle + PrivilegedProcessHandleFromUnsignedModule + ProcessActivitySummary + ProcessBlocked + ProcessControl + ProcessDataUsage + ProcessExecOnPackedExecutable + ProcessHandleOpDetectInfo + ProcessHandleOpDowngraded + ProcessInjection + ProcessPatternTelemetry + ProcessRollup + ProcessRollup2 + ProcessRollup2Stats + ProcessSelfDeleted + ProcessSessionCreated + ProcessSubstituteUser + ProcessTokenStolen + ProcessTrace + ProcessTreeCompositionPatternTelemetry + PtTelemetry + PtyCreated + QueueApcEtw + ReflectiveDllOpenProcess + RegisterRawInputDevicesEtw + RemediationActionKillProcess + RemediationMonitorKillProcess + RuntimeEnvironmentVariable + ScriptControlDotNetMetadata + ScriptControlErrorEvent + ServiceStarted + SessionPatternTelemetry + SetThreadCtxEtw + SetWindowsHook + SetWindowsHookExEtw + SetWinEventHookEtw + ShellCommandLineInfo + SruApplicationTimelineProvider + SudoCommandAttempt + SuspectCreateThreadStack + SuspendProcessError + SuspiciousPrivilegedProcessHandle + SuspiciousUserFontLoad + SuspiciousUserRemoteAPCAttempt + SyntheticPR2Stats + SyntheticProcessRollup2 + SyntheticProcessTrace + SystemTokenStolen + TerminateProcess + ThreadBlocked + UACAxisElevation + UACCOMElevation + UACExeElevation + UACMSIElevation + UmppcBypassSuspected + UnexpectedEnvironmentVariable + UserAssistAppLaunchInfo + UserSetProcessBreakOnTermination + WmiCreateProcess + WmiFilterConsumerBindingEtw # CloudWatch pipelines configuration for CrowdStrike The CrowdStrike setup on AWS reads log data from Amazon S3 buckets using Amazon SQS notifications for new object events. Configure the S3 source with the following parameters: ``` source: s3: aws: region: "us-east-1" sts_role_arn: "arn:aws:iam:::role/" compression: "gzip" codec: ndjson: data_source_name: "crowdstrike_falcon" default_bucket_owner: "123456789012" bucket_owners: my-bucket: "123456789012" disable_bucket_ownership_validation: false notification_type: "sqs" sqs: queue_url: "https://sqs.region.amazonaws.com//" on_error: "retain_messages" ```Parameters `notification_type` (required) Specifies the notification mechanism. Must be "sqs" to use SQS for S3 event notifications. `data_source_name` (required) Identifies the data source. This can be any string value that represents your data source. Example: "crowdstrike\$1falcon". `aws.region` (required) The AWS region where the S3 bucket and SQS queue are located. `aws.sts_role_arn` (required) The ARN of the IAM role to assume for accessing S3 and SQS resources. `codec` (required) Codec configuration for parsing S3 objects. Supports csv, json, ndjson codecs. `compression` (optional) Compression type of the S3 objects. Valid values are "none", "gzip", "automatic". Defaults to "none". `sqs.queue_url` (required for SQS) The complete SQS queue URL that receives S3 bucket notifications when new objects are created. `on_error` (optional) Determines how to handle errors in Amazon SQS. Can be either retain\$1messages or delete\$1messages. Default is retain\$1messages. # SentinelOne Singularity Endpoint integration configuration To integrate SentinelOne Singularity Endpoint with CloudWatch Logs, you must configure both the source and the pipeline. First, set up your SentinelOne source by configuring Amazon S3 and Amazon SQS to receive endpoint logs. Then, configure the CloudWatch pipeline to ingest the data from your source into CloudWatch Logs. **Topics** + [ # Source configuration for SentinelOne ](sentinelone-source-setup.md) + [ # CloudWatch pipelines configuration for SentinelOne ](sentinelone-pipeline-setup.md) # Source configuration for SentinelOne ## Integrating with SentinelOne Singularity Endpoint SentinelOne Singularity Endpoint is an AI-powered endpoint security platform that provides real-time protection against malware, ransomware, and zero-day attacks. It uses behavioral analysis and machine learning to detect and stop threats autonomously. The platform supports automated response, rollback, and threat remediation. It gives centralized visibility and control across all endpoints. CloudWatch pipelines enables you to collect this data in CloudWatch Logs. ## Instructions to setup Amazon S3 and Amazon SQS Configuring SentinelOne Singularity Endpoint to send logs to an Amazon S3 bucket involves several steps, primarily focused on setting up the Amazon S3 bucket, Amazon SQS queue, IAM roles, and then configuring the Amazon Telemetry Pipeline. + Create Amazon S3 bucket that stores SentinelOne Singularity Endpoint logs. + Configure Singularity Cloud Funnel or intermediate Syslog server with Amazon S3 bucket details to push logs. + Configure the Amazon S3 bucket to create event notifications, specifically for "Object Create" events. These notifications should be sent to an Amazon SQS queue. + Create an Amazon SQS queue in the same AWS region as your Amazon S3 bucket. This queue will receive notifications when new log files are added to the Amazon S3 bucket. ## Configuring the CloudWatch Pipeline To configure the pipeline to read logs, choose SentinelOne Singularity Endpoint as the data source. After filling in the required information and you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and the SentinelOne Singularity Endpoint events that maps to File System Activity (1001), Process Activity (1007), HTTP Activity (4002) and DNS Activity (4003). **File System Activity** contains the following events: + MALICIOUSFILE + FILECREATION + FILEDELETION + FILEMODIFICATION + FILERENAME + FILESCAN **Process Activity** contains the following events: + PROCESSCREATION + PROCESSTERMINATION + DUPLICATETHREAD + REMOTETHREAD + PROCESSMODIFICATION + DUPLICATEPROCESS + OPENPROCESS + PROCESSINJECTION + PROCESSMODIFIER + PROCESSEXIT + OPENPRIVILEGEDPROCESSFROMKERNEL **HTTP Activity** contains the following events: + HTTP **DNS Activity** contains the following events: + DNS # CloudWatch pipelines configuration for SentinelOne The SentinelOne setup on AWS reads log data from Amazon S3 buckets using Amazon SQS notifications for new object events. Configure the Zscalar source using the following parameters: ``` source: s3: aws: region: "us-east-1" sts_role_arn: "arn:aws:iam:::role/" compression: "gzip" codec: ndjson: data_source_name: "sentinelone_endpointsecurity" default_bucket_owner: "123456789012" bucket_owners: my-bucket: "123456789012" disable_bucket_ownership_validation: false notification_type: "sqs" sqs: queue_url: "https://sqs.region.amazonaws.com//" on_error: "retain_messages" ```Parameters `notification_type` (required) Specifies the notification mechanism. Must be "sqs" to use SQS for S3 event notifications. `data_source_name` (required) Identifies the data source. This can be any string value that represents your data source. Example: "sentinelone\$1endpointsecurity". `aws.region` (required) The AWS region where the S3 bucket and SQS queue are located. `aws.sts_role_arn` (required) The ARN of the IAM role to assume for accessing S3 and SQS resources. `codec` (required) Codec configuration for parsing S3 objects. Supports csv, json, ndjson codecs. `compression` (optional) Compression type of the S3 objects. Valid values are "none", "gzip", "automatic". Defaults to "none". `sqs.queue_url` (required for SQS) The complete SQS queue URL that receives S3 bucket notifications when new objects are created. `on_error` (optional) Determines how to handle errors in Amazon SQS. Can be either retain\$1messages or delete\$1messages. Default is retain\$1messages. # WIZ CNAPP integration configuration To integrate WIZ CNAPP with CloudWatch Logs, you must configure both the source and the pipeline. First, set up your WIZ source by authenticating with OAuth2 and configuring API access. Then, configure the CloudWatch pipeline to ingest the data from your source into CloudWatch Logs. **Topics** + [ # Source configuration for WIZ CNAPP ](wizcnapp-source-setup.md) + [ # CloudWatch pipelines configuration for WIZ ](wizcnapp-pipeline-setup.md) # Source configuration for WIZ CNAPP ## Integrating with Wiz CNAPP Wiz is a cloud-native application protection platform (CNAPP) that provides comprehensive visibility and security across multi-cloud environments. CloudWatch Pipeline uses the Wiz GraphQL API to retrieve information about security posture, vulnerabilities, misconfigurations, threats, and audit activities from your cloud infrastructure. The Wiz GraphQL API enables access to security data through flexible GraphQL queries, allowing retrieval of audit logs, issues, vulnerability findings, configuration findings, and detections from the Wiz platform. ## Authenticating with Wiz CNAPP To read Wiz Cnapp audit Logs, the pipeline needs to authenticate with your account. The plugin supports OAuth2 Authentication. Follow these instructions to get started. + Create a Service Account in Wiz with appropriate permissions. You must be logged in as a Wiz user with Write (W) permission on service accounts. + Configure the Service Account and get newly created Client ID and Client Secret. + In the AWS Secrets Manager, create a secret and store the Application (client) ID under the key `client_id` and the client secret under the key `client_secret`. + Configure API permissions (scopes) for your Service Account. Required scope: `read:issues`, `read:detections`, `read:cloud_events_cloud`, `read:cloud_events_sensor`, `read:security_scans`, `read:vulnerabilities`, `read:cloud_configuration`, `admin:audit` + Identify your GraphQL API endpoint: To find your specific endpoint check Tenant Info in the Wiz portal. The Wiz GraphQL API endpoint is `https://api..app.wiz.io/graphql`, where `` corresponds to your Wiz tenant's datacenter (e.g., us1, us2, eu1, eu2). ## Configuring the CloudWatch Pipeline When configuring the pipeline to read audit logs from Wiz, choose Wiz CNAPP as the data source. Fill in the required information like Region. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and events that map to Detection Finding (2004), Vulnerability Finding (2002), Compliance Finding (2003), Authentication (3002), and API Activity (6003). **Detection Finding** contains all events from following sources: + Issues + Detections **Vulnerability Finding** contains all events from following sources: + Vulnerability Findings **Compliance Finding** contains all events from following sources: + Cloud Configuration Findings **Authentication** contains events from following sources and given actions: + Audit logs + DeviceLogin + Login **API Activity** contains events from following sources and given actions: + Audit logs + AddSecurityScan + AddSupportTicketContext + AiAssistantSendMessage + ApproveCopyResourceForensicsSettings... + AssociateServiceTicket + CancelReportRun + ClearUIUserPreferences + CompleteAuthMigration + ConvertGitHubAppRegistrationCode + CopyResourceForensicsToExternalAccount + CreateActionTemplate + CreateApplicationServiceDiscoveryRule + CreateAutomationRule + CreateCICDScanPolicy + CreateCloudConfigurationFindingNote + CreateCloudConfigurationRule + CreateCloudConfigurationRules + CreateCloudEventRule + CreateComputeGroupTagsSet + CreateConnector + CreateControl + CreateCustomIPRange + CreateDashboard + CreateDashboardWidget + CreateDataClassifier + CreateDigitalTrustCustomDomain + CreateFileIntegrityMonitoringExclusion + CreateHostConfigurationAssessmentNote + CreateHostConfigurationRule + CreateIgnoreRule + CreateImageIntegrityValidator + CreateIntegration + CreateIssueNote + CreateMalwareExclusion + CreateMonitoredMetric + CreateOutpost + CreateOutpostCluster + CreatePolicyPackage + CreatePortalView + CreateProject + CreateRemediationAndResponseDeployment + CreateRemediationPullRequest + CreateReport + CreateRuntimeResponsePolicy + CreateSAMLIdentityProvider + CreateSAMLUser + CreateSavedCloudEventFilter + CreateSavedGraphQuery + CreateScannerAPIRateLimit + CreateSecurityFramework + CreateServiceAccount + CreateSupportTicket + CreateTestNode + CreateUser + CreateUserRole + CreateVulnerabilityFindingNote + DeleteActionTemplate + DeleteApplicationServiceDiscoveryRule + DeleteAutomationRule + DeleteCICDScan + DeleteCICDScanPolicy + DeleteCloudConfigurationFindingNote + DeleteCloudConfigurationRule + DeleteCloudEventRule + DeleteComputeGroupTagsSet + DeleteConnector + DeleteControl + DeleteCustomIPRange + DeleteDashboard + DeleteDashboardWidget + DeleteDataClassifier + DeleteDigitalTrustCustomDomain + DeleteFileIntegrityMonitoringExclusion + DeleteHostConfigurationAssessmentNote + DeleteHostConfigurationRule + DeleteIgnoreRule + DeleteImageIntegrityValidator + DeleteIntegration + DeleteIssueNote + DeleteMalwareExclusion + DeleteMonitoredMetric + DeleteOutpost + DeleteOutpostCluster + DeletePolicyPackage + DeletePortalView + DeleteProject + DeleteRemediationAndResponseDeployment + DeleteReport + DeleteRuntimeResponsePolicy + DeleteSAMLIdentityProvider + DeleteSavedCloudEventFilter + DeleteSavedGraphQuery + DeleteScannerAPIRateLimit + DeleteSecurityFramework + DeleteSecurityScan + DeleteServiceAccount + DeleteTestNode + DeleteUser + DeleteUserRole + DeleteVulnerabilityFindingNote + DisassociateServiceTicket + DuplicateDashboard + DuplicateDataClassifier + DuplicateHostConfigurationRule + DuplicateSecurityFramework + DuplicateUserRole + FinalizeCICDScan + FinalizeCICDScanTelemetry + GenerateWizContainerRegistryToken + GraphSearch + InitiateCICDScanTelemetry + InitiateDiskScanContainerImage + InitiateDiskScanDirectory + InitiateDiskScanVirtualMachine + InitiateDiskScanVirtualMachineImage + InitiateIACScan + InvokeOutpostClusterUpdate + LegalConsent + MergeDiscoveredApplicationService + MigrateUsers + ModifySAMLIdentityProviderGroupMappings + ModifySAMLIdentityProviderPortalView... + PromoteDiscoveredApplicationService + ProvideAiFeedback + ProvideAiGraphQueryExample + ProvideAiGraphQueryFeedback + ProvideIssueFeedback + ReassessIssue + RefreshResponseActions + RegisterAgent + ReportIDEActivityHeartbeat + ReportIDEAnalytics + RequestConnectorEntityScan + RequestConnectorScan + RerunReport + ResetUserPassword + RevokeSessions + RevokeUserSessions + RotateServiceAccountSecret + RunAllControls + RunCloudConfigurationRule + RunControl + RunControlsIntegrationAction + RunIssuesIntegrationAction + RunOutpostClusterUpdate + RunResponseAction + SAMLUserInitialProvision + SendUserEmailInvite + TagCICDScan + TokenDeviceRefresh + TokenRefresh + UninstallOutpost + UpdateAiSettings + UpdateApplicationServiceDiscoveryRule + UpdateAutomationRule + UpdateBasicAuthSettings + UpdateCICDScanPolicy + UpdateChampionCenterJourneyItem + UpdateCloudConfigurationFinding + UpdateCloudConfigurationRule + UpdateCloudConfigurationRules + UpdateCloudCostSettings + UpdateCloudEventRule + UpdateCloudEventRules + UpdateCloudEventSettings + UpdateComputeGroupTagsSet + UpdateConnector + UpdateContainerRegistryCustomScannin... + UpdateContainerRegistryGlobalScannin... + UpdateControl + UpdateControls + UpdateCopyResourceForensicsSettings + UpdateCustomIPRange + UpdateCustomIPRangesSettings + UpdateCustomUserRolesSettings + UpdateDashboard + UpdateDashboardSettings + UpdateDashboardWidget + UpdateDataClassifier + UpdateDataFinding + UpdateDataScannerSettings + UpdateDigitalTrustCustomDomain + UpdateDigitalTrustDashboardSettings + UpdateDigitalTrustSAMLIdentityProvider + UpdateDiscoveredApplicationServices + UpdateEventTriggeredScanningSettings + UpdateExternalExposureScannerSettings + UpdateExternalExposureSettings + UpdateFileIntegrityMonitoringExclusion + UpdateFileIntegrityMonitoringSettings + UpdateForensicsPackageSettings + UpdateGraphEntity + UpdateHostConfigurationRule + UpdateHostConfigurationRuleAssessment + UpdateHostConfigurationRules + UpdateIPRestrictions + UpdateIgnoreRule + UpdateImageIntegrityValidator + UpdateIntegration + UpdateInternalExposureSettings + UpdateIssue + UpdateIssueNote + UpdateIssueSettings + UpdateIssues + UpdateKubernetesGlobalScanningConfig... + UpdateLoginSettings + UpdateMalwareExclusion + UpdateMonitoredMetric + UpdateMonitoredMetricSettings + UpdateNode + UpdateNonOSDiskScanningSettings + UpdateNotificationSettings + UpdateOutpost + UpdateOutpostCluster + UpdatePolicyPackage + UpdatePortalInactivityTimeoutSettings + UpdatePortalSettings + UpdatePortalView + UpdatePreviewHubItem + UpdateProject + UpdateRemediationAndResponseDeployment + UpdateReport + UpdateReportSettings + UpdateRepositorySettings + UpdateResponseAction + UpdateResponseActions + UpdateRuntimeResponsePolicy + UpdateSAMLIdentityProvider + UpdateSavedCloudEventFilter + UpdateSavedGraphQuery + UpdateScannerAPIRateLimit + UpdateScannerExclusionSettingsConstr... + UpdateScannerExclusionSettingsTimeLi... + UpdateScannerExclusionSizeLimits + UpdateScannerExclusionTags + UpdateScannerResourceTagSettings + UpdateScannerResourceTags + UpdateScannerSettings + UpdateSecretInstance + UpdateSecurityFramework + UpdateSecurityScan + UpdateServiceAccount + UpdateSessionLifetimeSettings + UpdateSupportContactList + UpdateSystemHealthIssue + UpdateSystemHealthIssues + UpdateTechnology + UpdateTenantNewsletterSettings + UpdateUIUserPreferences + UpdateUser + UpdateUserRole + UpdateUserSelectedPortalView + UpdateVersionControlOrganizationSett... + UpdateVersionControlRepositorySettings + UpdateViewerPreferences + UpdateVulnerability + UpdateVulnerabilityAssessmentSettings + UpdateVulnerabilityFinding + UpdateVulnerabilityFindingStatus + UpsertAgentTelemetry # CloudWatch pipelines configuration for WIZ Collects cloud-native application protection platform (CNAPP) data from Wiz using OAuth2 authentication. Configure the Wiz CNAPP source with the following parameters: ``` source: wiz_cnapp: region: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `region` (required) Wiz region for your organization. `authentication.oauth2.client_id` (required) OAuth2 client ID for Wiz API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Wiz API authentication. # Okta SSO integration configuration CloudWatch pipelines enables you to collect logs from Okta SSO. Okta SSO is a leading cloud-based identity and access management solution that provides secure, centralized authentication for users and applications. **Topics** + [ # Source configuration for Okta SSO ](okta-sso-source-setup.md) + [ # CloudWatch pipelines configuration for Okta SSO ](okta-sso-pipeline-setup.md) # Source configuration for Okta SSO ## Integrating with Okta SSO CloudWatch Pipeline uses the Okta System Log API to retrieve Authentication, API Activity, Detection Finding and Entity Management events from your Okta SSO tenant. ## Authenticating with Okta SSO To read the logs, the pipeline needs to authenticate with your Okta SSO tenant. For Okta SSO, authentication is performed using the OAuth 2.0 Client Credentials (JWT Assertion) flow through an Okta API Services application. **Generate the private/public key pair for authentication** + Sign in to the Okta Admin Console using an administrator account. + Navigate to Applications → Applications. + Select an existing API Services Application or create a new one. + Under General → Client Credentials, upload a public key or generate a new key. This key pair will be used to authenticate using a signed JWT assertion. + Ensure the application has the required OAuth scopes assigned, specifically: `okta.logs.read` + Admin Roles → Edit assignments → Role(Select Read-only Administrator) + Copy the Client ID of the application. + Store the client\$1id and client\$1secret(private key) in AWS Secrets Manager: `client_id` and `client_secret(private_key)` (the RSA private key used to sign the JWT assertion) + Identify your Okta Organization URL and configure in the pipeline (for example: `https://yourdomain.okta.com`). Once configured, the pipeline can authenticate using Okta's OAuth 2.0 Client Credentials (JWT Assertion) flow and begin retrieving audit log events from the Okta System Log API. ## Configuring the CloudWatch Pipeline To configure the pipeline to read logs, choose Okta SSO as the data source. Fill in the required information like Okta Domain name. Once you create and activate the pipeline, audit log data from Okta SSO will begin flowing into the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and Okta events that map to Authentication (3002), API Activity (6003), Detection Finding (2004), and Entity Management (3004). **Authentication** contains the following events: + user.authentication.auth + user.authentication.auth\$1via\$1AD\$1agent + user.authentication.auth\$1via\$1IDP + user.authentication.auth\$1via\$1LDAP\$1agent + user.authentication.auth\$1via\$1inbound\$1SAML + user.authentication.auth\$1via\$1inbound\$1delauth + user.authentication.auth\$1via\$1iwa + user.authentication.auth\$1via\$1mfa + user.authentication.auth\$1via\$1radius + user.authentication.auth\$1via\$1richclient + user.authentication.auth\$1via\$1social + user.authentication.authenticate + user.authentication.sso + user.session.start + user.session.impersonation.grant + app.oauth2.signon + user.session.impersonation.initiate + user.authentication.universal\$1logout + user.session.clear + user.session.end + user.authentication.slo + user.authentication.universal\$1logout.scheduled + user.session.expire + user.session.impersonation.end + user.authentication.verify + policy.evaluate\$1sign\$1on + user.mfa.attempt\$1bypass + user.mfa.okta\$1verify + user.mfa.okta\$1verify.deny\$1push + user.mfa.okta\$1verify.deny\$1push\$1upgrade\$1needed + user.mfa.factor.activate + user.mfa.factor.deactivate + user.mfa.factor.reset\$1all + user.mfa.factor.suspend + user.mfa.factor.unsuspend + user.mfa.factor.update + user.session.impersonation.extend + user.session.impersonation.revoke + user.session.access\$1admin\$1app + user.session.context.change + application.policy.sign\$1on.deny\$1access + user.authentication.auth\$1unconfigured\$1identifier + user.authentication.dsso\$1via\$1non\$1priority\$1source + app.oauth2.invalid\$1client\$1credentials + policy.auth\$1reevaluate.fail **API Activity** contains the following events: + oauth2.claim.created + oauth2.scope.created + security.trusted\$1origin.create + system.api\$1token.create + workflows.user.table.view + app.oauth2.as.key.rollover + app.saml.sensitive.attribute.update + system.api\$1token.update + oauth2.claim.updated + oauth2.scope.updated + security.events.provider.deactivate + system.api\$1token.revoke + oauth2.claim.deleted + oauth2.scope.deleted **Detection Finding** contains the following events: + security.attack.start + security.breached\$1credential.detected + security.request.blocked + security.threat.detected + security.zone.make\$1blacklist + system.rate\$1limit.violation + user.account.report\$1suspicious\$1activity\$1by\$1enduser + user.risk.change + user.risk.detect + zone.make\$1blacklist + security.attack.end **Entity Management** contains the following events: + iam.role.create + system.idp.lifecycle.create + application.lifecycle.create + group.lifecycle.create + user.lifecycle.create + policy.lifecycle.create + zone.create + oauth2.as.created + event\$1hook.created + inline\$1hook.created + pam.security\$1policy.create + iam.resourceset.create + pam.secret.create + analytics.reports.export.download + app.audit\$1report.download + system.idp.lifecycle.read\$1client\$1secret + app.oauth2.client.read\$1client\$1secret + pam.secret.reveal + pam.service\$1account.password.reveal + support.org.update + system.idp.lifecycle.update + application.lifecycle.update + policy.lifecycle.update + user.account.update\$1profile + user.account.update\$1password + user.account.reset\$1password + group.profile.update + zone.update + group.privilege.grant + group.privilege.revoke + iam.resourceset.bindings.add + user.account.privilege.grant + user.account.privilege.revoke + pki.cert.lifecycle.revoke + iam.resourceset.update + iam.role.update + pam.security\$1policy.update + oauth2.as.updated + event\$1hook.updated + inline\$1hook.updated + pam.secret.update + iam.resourceset.bindings.delete + iam.role.delete + pam.security\$1policy.delete + policy.lifecycle.delete + user.lifecycle.delete.initiated + application.lifecycle.delete + group.lifecycle.delete + zone.delete + oauth2.as.deleted + event\$1hook.deleted + inline\$1hook.deleted + iam.resourceset.delete + pam.secret.delete + device.enrollment.create + credential.register + credential.revoke + policy.lifecycle.activate + system.feature.enable + event\$1hook.activated + inline\$1hook.activated + system.feature.disable + application.lifecycle.activate + user.lifecycle.activate + zone.activate + oauth2.as.activated + system.log\$1stream.lifecycle.activate + policy.lifecycle.deactivate + security.authenticator.lifecycle.deactivate + application.lifecycle.deactivate + user.lifecycle.deactivate + zone.deactivate + event\$1hook.deactivated + inline\$1hook.deactivated + system.log\$1stream.lifecycle.deactivate + oauth2.as.deactivated + user.account.lock + user.account.lock.limit + user.lifecycle.suspend + device.lifecycle.suspend + user.account.unlock + user.lifecycle.unsuspend + device.lifecycle.unsuspend + user.lifecycle.reactivate # CloudWatch pipelines configuration for Okta SSO Collects log data from Okta SSO using OAuth2 authentication. Configure the Okta SSO source with the following parameters: ``` source: okta_sso: domain: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `domain` (required) The Okta domain name for your organization. `authentication.oauth2.client_id` (required) OAuth2 client ID for Okta SSO API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Okta SSO API authentication. # Zscaler Internet Access CloudWatch pipelines enables you to collect logs from Zscaler Internet Access (ZIA). ZIA is a cloud-based secure web gateway that protects users connecting to the internet. **Topics** + [ # Source configuration for Zscaler Internet Access ](zscaler-zia-source-setup.md) + [ # Pipeline configuration for Zscaler Internet Access ](zscaler-zia-pipeline-setup.md) # Source configuration for Zscaler Internet Access ## Integrating with Zscaler Internet Access Zscaler Internet Access (ZIA) is a cloud-based secure web gateway that protects users connecting to the internet. It inspects all internet traffic to block malware, phishing, and data leaks using advanced threat detection and SSL inspection. ZIA enforces security policies in real time without requiring on-premises hardware. It ensures safe and compliant internet access for users anywhere. CloudWatch pipelines enables you to collect this data in CloudWatch Logs. ## Instructions to setup Amazon S3 and Amazon SQS Configuring ZIA to send logs to an Amazon S3 bucket involves several steps, primarily focused on setting up the Amazon S3 bucket, Amazon SQS queue, IAM roles, and then configuring the Amazon Telemetry Pipeline. + Create Amazon S3 bucket that stores ZIA logs and create separate folders for each log type. Create IAM user and grant s3 write permission, console access not needed only CLI and create Access key and Secret key for this account. + Configure NSS feeds with Amazon S3 bucket details to push logs. + Configure the Amazon S3 bucket to create event notifications, specifically for "Object Create" events. These notifications should be sent to an Amazon SQS queue. + Create an Amazon SQS queue in the same AWS region as your Amazon S3 bucket. This queue will receive notifications when new log files are added to the Amazon S3 bucket. ## Configuring the CloudWatch Pipeline When configuring the pipeline to read data from Zscaler Internet Access, choose Zscaler Internet Access (ZIA) as the data source. After filling in the required information and you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and events that map to DNS Activity (4003), HTTP Activity (4002), Network Activity (4001), and Authentication (3002). Each event comes from a source as mentioned below. **DNS Activity** covers all events from source: + DNS Logs **HTTP Activity** covers all events from source: + Web Logs **Network Activity** covers all events from source: + Firewall Logs **Authentication** covers events from source: + Admin Audit Logs - Event actions: SIGN\$1IN, SIGN\$1OUT # Pipeline configuration for Zscaler Internet Access The Zscaler setup on AWS reads log data from Amazon S3 buckets using Amazon SQS notifications for new object events. Configure the Zscalar source using the following parameters: ``` source: s3: aws: region: "us-east-1" sts_role_arn: "arn:aws:iam:::role/" compression: "gzip" codec: ndjson: data_source_name: "zscaler_internetaccess" default_bucket_owner: "123456789012" bucket_owners: my-bucket: "123456789012" disable_bucket_ownership_validation: false notification_type: "sqs" sqs: queue_url: "https://sqs.region.amazonaws.com//" on_error: "retain_messages" ```Parameters `notification_type` (required) Specifies the notification mechanism. Must be "sqs" to use SQS for S3 event notifications. `data_source_name` (required) Identifies the data source. This can be any string value that represents your data source. Example: "zscaler\$1internetaccess". `aws.region` (required) The AWS region where the S3 bucket and SQS queue are located. `aws.sts_role_arn` (required) The ARN of the IAM role to assume for accessing S3 and SQS resources. `codec` (required) Codec configuration for parsing S3 objects. Supports csv, json, ndjson codecs. `compression` (optional) Compression type of the S3 objects. Valid values are "none", "gzip", "automatic". Defaults to "none". `sqs.queue_url` (required for SQS) The complete SQS queue URL that receives S3 bucket notifications when new objects are created. `on_error` (optional) Determines how to handle errors in Amazon SQS. Can be either retain\$1messages or delete\$1messages. Default is retain\$1messages. # Okta Auth0 CloudWatch pipelines enables you to collect logs from Okta Auth0. Okta Auth0 is a flexible identity platform designed for modern application authentication and authorization. **Topics** + [ # Source configuration for Okta Auth0 ](auth0-source-setup.md) + [ # Pipeline configuration for Okta Auth0 ](auth0-pipeline-setup.md) # Source configuration for Okta Auth0 ## Integrating with Okta Auth0 Okta Auth0 is a flexible identity platform designed for modern application authentication and authorization. Auth0 provides developers with powerful tools to integrate secure login, user management, and access control into applications while maintaining scalability and customization. CloudWatch Pipeline uses the Auth0 Management API to retrieve Authentication (successful and failed logins), and API Activity logs from Auth0 log events. ## Authenticating with Okta Auth0 To read logs, the pipeline needs to authenticate with your Okta Auth0 tenant. Auth0 Management API access requires a Client ID and Client Secret belonging to a Machine-to-Machine (M2M) application. **Generate Client Credentials** see API Settings for more details. + Sign in to the Auth0 Dashboard using an admin account. + Navigate to Applications → Applications. + Select an existing Machine-to-Machine Application or create a new one. + Ensure the application has the required scope permissions for the Management API, specifically: `read:logs` + In the AWS Secrets Manager, create a secret and store the Client ID under the key `client_id` and the Client Secret under the key `client_secret` + Identify your Auth0 Tenant Domain (for example: `yourtenant.us.auth0.com`) and give it in pipeline. Once configured, the pipeline can authenticate using the Client Credentials flow and retrieve log events from Auth0. ## Configuring the CloudWatch Pipeline To configure the pipeline to read logs, choose Okta Auth0 as the data source. Select the Source Type as Tenant and provide the required details such as your Auth0 Tenant Domain and Client Credentials. Once you create the pipeline, log data from Okta Auth0 will be collected and made available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and the Auth0 events that maps to Authentication (3002) and API Activity (6003) **Authentication** contains the following events: + f + fu + fp + feccft + fepft + feacft + fc + fco + fcoa + fd + ferrt + fertft + fsa + limit\$1wc + limit\$1sul + limit\$1mu + pwd\$1leak + reset\$1pwd\$1leak + signup\$1pwd\$1leak + gd\$1auth\$1fail\$1email\$1verification + gd\$1auth\$1failed + gd\$1auth\$1rejected + gd\$1otp\$1rate\$1limit\$1exceed + gd\$1recovery\$1failed + gd\$1recovery\$1rate\$1limit\$1exceed + gd\$1webauthn\$1challenge\$1failed + passkey\$1challenge\$1failed + scp + sv + ss + s + fi + fv + feoobft + feotpft + fercft + ss\$1sso\$1failure + fepotpft + fvr + flo **API Activity** contains the following events: + api\$1limit + limit\$1delegation + mgmt\$1api\$1read + sapi + api\$1limit\$1warning # Pipeline configuration for Okta Auth0 Collects log data from Okta Auth0 using OAuth2 authentication. Configure the Okta Auth0 source with the following parameters: ``` source: okta_auth0: domain: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `domain` (required) The Okta Auth0 domain name for your organization. `authentication.oauth2.client_id` (required) OAuth2 client ID for Okta Auth0 API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Okta Auth0 API authentication. # ServiceNow CMDB Audit Log ServiceNow is an enterprise platform that provides IT service management (ITSM) and configuration management database (CMDB) capabilities for tracking and managing IT assets, configurations, and changes across organizations. **Topics** + [ # Source configuration for ServiceNow CMDB Audit Log ](servicenow-cmdb-source-setup.md) + [ # CloudWatch pipelines configuration for ServiceNow CMDB Audit Log ](servicenow-cmdb-pipeline-setup.md) # Source configuration for ServiceNow CMDB Audit Log ## Integrating with ServiceNow CMDB ServiceNow is an enterprise platform that provides IT service management (ITSM) and configuration management database (CMDB) capabilities for tracking and managing IT assets, configurations, and changes across organizations. CloudWatch Pipeline uses the ServiceNow Table API to retrieve information about sys\$1audit, syslog, sysevent, and syslog\$1transactions from your ServiceNow instance. ## Authenticating with ServiceNow CMDB To read the logs, the pipeline needs to authenticate with your ServiceNow instance. The ServiceNow Table API supports OAuth 2.0. + Ensure the REST API is enabled on your ServiceNow instance. + Enable OAuth 2.0 Client Credentials grant type in your ServiceNow instance + Create an OAuth Application Registry for external client authentication + In the AWS Secrets Manager, create a secret and store the Application (client) ID under the key `client_id` and the client secret under the key `client_secret`. + Configure OAuth Application User and assign required roles ## Configuring the CloudWatch Pipeline When configuring the pipeline to read audit logs from ServiceNow choose ServiceNow CMDB as the data source. Fill in the required information like `instance_url` and the secret where `client_id` and `client_secret` are stored. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and events that map to Entity Management (3004), API Activity (6003), and Datastore Activity (6005). These events are from specific tables and filtered for CMDB reference. **Entity Management** contains events from following tables: + sys\$1audit **API Activity** contains events from following tables: + sysevent + syslog **Datastore Activity** contains events from following tables: + syslog\$1transactions # CloudWatch pipelines configuration for ServiceNow CMDB Audit Log Collects configuration management database (CMDB) data from ServiceNow using OAuth2 authentication. Configure the ServiceNow CMDB source with the following parameters: ``` source: servicenow_cmdb: instance_url: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `instance_url` (required) ServiceNow instance URL. `authentication.oauth2.client_id` (required) OAuth2 client ID for ServiceNow API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for ServiceNow API authentication. # Microsoft Entra ID integration configuration Microsoft Entra ID (formerly Azure Active Directory) is Microsoft's cloud-based identity and access management service that helps organizations manage user identities and secure access to resources. **Topics** + [ # Source configuration for Microsoft Entra ID ](entraid-source-setup.md) + [ # CloudWatch pipelines configuration for Microsoft Entra ID ](entraid-pipeline-setup.md) # Source configuration for Microsoft Entra ID ## Integrating with Microsoft Entra ID Microsoft Entra ID (formerly Azure Active Directory) is Microsoft's cloud-based identity and access management service that helps organizations manage user identities and secure access to resources. CloudWatch Pipeline uses the Microsoft Graph API to retrieve comprehensive identity and security information from Microsoft Entra ID audit logs. The Microsoft Graph API provides access to three primary log types: Directory Audit Logs (tracking directory-level changes and administrative actions), Sign-In Logs (capturing user authentication events and activities), and Provisioning Logs (monitoring user and group provisioning operations). ## Authenticating with Microsoft Entra ID To retrieve the Audit Logs EntraID, pipelines needs to authenticate with your account. The plugin supports OAuth2 Authentication. Follow the instructions in Microsoft Graph APIs and should have the Microsoft Entra ID P1 or P2 license. + Register an application in Azure with Supported account types, Accounts in this organizational directory only (Single tenant). After registration is complete, note down the Application (client) ID and Directory (tenant) ID. + Generate a new key for your application. Key is also known as client secret, which are used when exchanging an authorization code for an access token. + In the AWS Secrets Manager, create a secret and store the Application (client) ID under the key `client_id` and the client secret under the key `client_secret` + Specify the permissions your application requires to access the Microsoft Graph APIs. The permissions you need are: + AuditLog.Read.All: Required to read audit logs, sign-in logs, and provisioning logs + Directory.Read.All: Required to read directory data ## Configuring the CloudWatch Pipeline When configuring the pipeline to read Audit Logs from Microsoft EntraID, choose Microsoft EntraID as the data source. Fill in the required information like Tenant Id using Directory (tenant) ID. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and Entra ID events that map to Authentication (3002), Account Change (3001), User Access Management (3005), and Entity Management (3004). **Authentication** contains the following events with type in brackets: + Invalid Username or Password (Sign-in) + User Strong Auth ClientAuthN Required Interrupt (Sign-in) + Pass Through User Mfa Error (Sign-in) + Authentication Failed During Strong Auth (Sign-in) **Account Change** contains the following events with type in brackets: + Add user (Audit) + Update user (Audit) + Delete user (Audit) + Hard delete user (Audit) + Reset password (Audit) + User changed default security info (Audit) + Enable Strong Authentication (Audit) + Disable Strong Authentication (Audit) **User Access Management** contains the following events with type in brackets: + Add eligible member to role (Audit) + Remove eligible member from role (Audit) + Add eligible member to role in PIM completed (Audit) + Remove eligible member from role in PIM completed (Audit) + Add member to role (Audit) + Remove member from role (Audit) + Remove permanent direct role assignment (Audit) + Add permanent direct role assignment (Audit) + Triggered PIM alert (Audit) + Add delegated permission grant (Audit) + Remove delegated permission grant (Audit) **Entity Management** contains the following events with type in brackets: + Create (Provisioning) + Update (Provisioning) + Add app role assignment to service principal (Audit) + Remove app role assignment to service principal (Audit) + Add service principal credentials (Audit) + Remove service principal credentials (Audit) + Update service principal (Audit) + Add service principal (Audit) + Hard delete service principal (Audit) + Remove service principal (Audit) + Consent to application (Audit) + Add application (Audit) + Add owner to application (Audit) + Hard Delete application (Audit) + Delete application (Audit) + Update application (Audit) + Update application – Certificates and secrets management (Audit) + Add device (Audit) + Update device (Audit) + Delete device (Audit) + Hard delete device (Audit) # CloudWatch pipelines configuration for Microsoft Entra ID Collects log data from Microsoft Entra ID (formerly Azure Active Directory) using OAuth2 authentication. Configure the Microsoft Entra ID source with the following parameters: ``` source: microsoft_entra_id: tenant_id: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `tenant_id` (required) The Microsoft Entra ID tenant ID for your organization. `authentication.oauth2.client_id` (required) OAuth2 client ID for Microsoft Graph API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Microsoft Graph API authentication. # Microsoft Windows Events integration configuration Microsoft Windows Event Logs provide a comprehensive logging system that records system, security, and application events on Windows operating systems. **Topics** + [ # Source configuration for Microsoft Windows Events ](windows-events-source-setup.md) + [ # CloudWatch pipelines configuration for Microsoft Windows Events ](windows-events-pipeline-setup.md) # Source configuration for Microsoft Windows Events ## Integrating with Windows Event Microsoft Windows Event Logs provide a comprehensive logging system that records system, security, and application events on Windows operating systems. CloudWatch Pipeline uses the Log Analytics API to retrieve information about system operations, security events, user activities, and application behaviors from Windows servers and workstations. The Log Analytics API enables access to event data through KQL (Kusto Query Language) queries, allowing retrieval of Windows Event logs from Log Analytics workspaces. ## Authenticating with Windows Event To read Windows Event audit Logs, the pipeline needs to authenticate with your account. The plugin supports OAuth2 Authentication. Follow these instructions to get started with Microsoft Windows Event: Log Analytics APIs. + Register an application in Azure with Supported account types, Accounts in this organizational directory only (Single tenant). After registration is complete, note down the Application (client) ID and Directory (tenant) ID. + Generate a new client secret for your application. The client secret is used when exchanging an authorization code for an access token. Copy the secret value immediately as it won't be shown again. + In the AWS Secrets Manager, create a secret and store the Application (client) ID under the key `client_id` and the client secret under the key `client_secret`. + Specify the API permissions your application requires to access the Log Analytics API. The permission you need is: Data.Read: Required to execute KQL queries and read log data from Log Analytics workspaces, including Windows Event logs. + Create and configure a Log Analytics Workspace: Create a workspace in Azure portal (Monitor → Log Analytics workspaces). Create a Data Collection Rule (DCR) to specify which Windows Event Logs to collect (System, Application, Security). Connect your Windows servers/VMs to the workspace through the DCR. Note down your Workspace ID from the workspace Overview page (required for API queries) + Grant workspace access to your application: Navigate to your Log Analytics workspace → Access control (IAM). Assign the Log Analytics Reader role to your registered application. This RBAC role works together with the API permission to provide secure access: OAuth confirms API usage rights, while IAM confirms workspace data access rights. ## Configuring the CloudWatch Pipeline When configuring the pipeline to read logs, choose Microsoft Windows Events as the data source. Fill in the required information like Tenant Id using Directory (tenant) ID and Workspace Id (workspace\$1id). Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and Windows audit events that map to Account Change (3001), Authentication (3002), Entity Management (3004), Event Log Activity (1008), File System Activity (1001), Group Management (3006), and Kernel Activity (1003). **Account Change** contains the following events: + 4740 **Authentication** contains the following events: + 4624 + 4625 + 4634 + 4647 + 4648 + 4649 + 4672 **Entity Management** contains the following events: + 4616 + 4907 + 4719 + 4902 **Event Log Activity** contains the following events: + 1100 + 1102 + 1104 + 1105 **File System Activity** contains the following events: + 4608 + 4660 + 4688 + 4696 + 4826 + 5024 + 5033 + 5058 + 5059 + 5061 + 5382 + 5379 **Group Management** contains the following events: + 4732 + 4798 + 4799 + 4733 + 4731 + 4734 + 4735 **Kernel Activity** contains the following events: + 4674 # CloudWatch pipelines configuration for Microsoft Windows Events Collects log data from Microsoft Windows Event logs using OAuth2 authentication. Configure the Microsoft Windows Event source with the following parameters: ``` source: microsoft_windows_event: tenant_id: "" workspace_id: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `tenant_id` (required) The Microsoft tenant ID for your organization. `workspace_id` (required) The Microsoft Log Analytics workspace ID. `authentication.oauth2.client_id` (required) OAuth2 client ID for Log Analytics workspace API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Log Analytics workspace API authentication. # Palo Alto Networks Next-Generation Firewalls integration configuration CloudWatch pipelines enables you to collect network security logs from Palo Alto Networks Next-Generation Firewalls. Palo Alto Networks provides network security solutions, including firewall, VPN, and threat detection services. **Topics** + [ # Source configuration for Palo Alto Networks Next-Generation Firewalls ](paloalto-ngfw-source-setup.md) + [ # CloudWatch pipelines configuration for Palo Alto Networks Next-Generation Firewalls ](paloalto-ngfw-pipeline-setup.md) # Source configuration for Palo Alto Networks Next-Generation Firewalls ## Integrating with Palo Alto Networks Next-Generation Firewalls CloudWatch Pipeline integrates with Palo Alto Networks NGFW using the PAN-OS XML API to retrieve security, authentication, network activity, process activity, detection finding and threat activity. The PAN-OS XML API enables structured access, allowing the retrieval of System Logs, GlobalProtect, Traffic Logs, Threat Logs and URL Filtering Log. ## Authenticating with Palo Alto NGFW To read network security logs, the pipeline needs to authenticate with your Palo Alto Networks NGFW login device interface. The plugin supports Basic Authentication. + Create and Manage Users on a Palo Alto Networks NGFW Firewall via CLI + Login firewall using with hostname using user admin and your password + Store this username and password in a secret in AWS Secrets Manager under the keys `username` and `password`. + Identify and note down your PAN-OS hostname. Once configured, the pipeline can authenticate using the username and password and retrieve log activity from PAN-OS. ## Configuring the CloudWatch Pipeline When configuring the pipeline to read logs from Palo Alto Networks NGFW, choose Palo Alto Networks Next-Generation Firewalls as the data source. Fill in the required information like `hostname`. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and events that map to Authentication (3002), Network Activity (4001), Process Activity (1007), and Detection Finding (2004). **Authentication** contains the following type and subtypes: + GlobalProtect + data + file + flood + packet + scan + spyware + url + virus + vulnerability + wildfire + wildfire-virus + System Logs + auth **Network Activity** contains the following types and subtypes: + Traffic Logs + start + end + drop + deny + System Logs + vpn + url-filtering + app-cloud-engine + dhcp + ssh + dnsproxy + dns-security + wildfire + wildfire-appliance + ntpd + userid **Process Activity** contains the following type and subtypes: + System Logs + general + satd + ras + sslmgr + hw + iot + ctd-agent + routing + port + device-telemetry **Detection Finding** contains the following type and subtypes: + Threat Logs + data + file + flood + packet + scan + spyware + url + ml-virus + virus + vulnerability + wildfire + wildfire-virus + URL Filtering Log # CloudWatch pipelines configuration for Palo Alto Networks Next-Generation Firewalls Collects log data from Palo Alto Next-Generation Firewall using basic authentication. Configure the Palo Alto NGFW source with the following parameters: ``` source: palo_alto_ngfw: hostname: "" authentication: basic: username: "${{aws_secrets::username}}" password: "${{aws_secrets::password}}" ```Parameters `hostname` (required) The Palo Alto NGFW hostname for your firewall. `authentication.basic.username` (required) Basic authentication username for Palo Alto NGFW API authentication. `authentication.basic.password` (required) Basic authentication password for Palo Alto NGFW API authentication. # GitHub Audit Log integration configuration Amazon Telemetry Pipelines enables you to collect audit logs from [ GitHub Enterprise ](https://github.com/enterprise) Cloud. GitHub Enterprise is an enterprise-grade software development platform designed for the complex workflows of modern development. **Topics** + [ # Source configuration for GitHub Audit Log ](github-audit-log-source-setup.md) + [ # CloudWatch pipelines configuration for GitHub Audit Log ](github-audit-log-pipeline-setup.md) # Source configuration for GitHub Audit Log **Note** Important: GitHub Enterprise accounts are required to use this connector. GitHub Personal or Organization accounts are not supported. ## Integrating with GitHub Amazon Telemetry Pipelines enables you to collect audit logs from GitHub Enterprise Cloud. GitHub Enterprise is an enterprise-grade software development platform designed for the complex workflows of modern development. GitHub Enterprise Cloud is the cloud-based solution of GitHub Enterprise, hosted on GitHub's servers. ## Authenticating with GitHub To read the audit logs, pipeline needs to authenticate with your GitHub account. For Enterprise [ scope ](https://docs.github.com/en/enterprise-cloud@latest/rest/enterprise-admin/audit-log?apiVersion=2022-11-28#get-the-audit-log-for-an-enterprise), you can use Personal Access Token and for Organization [ scope ](https://docs.github.com/en/enterprise-cloud@latest/rest/orgs/orgs?apiVersion=2022-11-28#get-the-audit-log-for-an-organization), you can either use Personal Access Token or GitHub App. **Generate the token to authenticate as Personal Access Token:** + Sign in to [ GitHub ](https://github.com/dashboard) using credentials for the GitHub account + The authenticated user must be an enterprise admin to use this endpoint + Open the GitHub Personal access tokens (classic) page, locate the Generate new token (classic) and then follow the GitHub procedure to generate a token with `read:audit_log` scope and No expiration + Store this new token in a secret in the AWS Secrets Manager under the key `personal_access_token` **Generate the private key to authenticate as GitHub App:** + Sign in to [ GitHub ](https://github.com/dashboard) using credentials for the GitHub account + Ensure the GitHub App has the "Administration" organization [ permissions ](https://docs.github.com/en/enterprise-cloud@latest/apps/creating-github-apps/registering-a-github-app/choosing-permissions-for-a-github-app) (read) permission + Follow the instructions in [ Managing private keys for GitHub Apps ](https://docs.github.com/en/enterprise-cloud@latest/apps/creating-github-apps/authenticating-with-a-github-app/managing-private-keys-for-github-apps) and generate the private key + Store this private key in a secret in the AWS Secrets Manager under the key `private_key` and the GitHub App name under the key `app_id` ## Configuring the CloudWatch Pipeline When configuring the pipeline to read audit logs from GitHub Enterprise Cloud, choose GitHub Audit Logs as the data source. Select the Source Type as Enterprise or Organization based on the scope of your integration and fill in the required information like Enterprise or Organization name according to the selected scope. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and the [ GitHub actions ](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/security-log-events) that maps to Account Change (3001), API Activity (6003) and Entity Management (3004). **Account Change** contains the following actions: + org.enable\$1two\$1factor\$1requirement + org.disable\$1two\$1factor\$1requirement + two\$1factor\$1authentication.add\$1factor + two\$1factor\$1authentication.enabled + two\$1factor\$1authentication.disabled + two\$1factor\$1authentication.remove\$1factor + org.disable\$1saml + org.enable\$1saml + personal\$1access\$1token.access\$1restriction\$1disabled + personal\$1access\$1token.access\$1restriction\$1enabled + personal\$1access\$1token.expiration\$1limit\$1set + personal\$1access\$1token.expiration\$1limit\$1unset **API Activity** contains the following actions: + repository\$1secret\$1scanning\$1custom\$1pa....create + repository\$1secret\$1scanning\$1custom\$1pa....update + repository\$1secret\$1scanning\$1custom\$1pa....delete + repository\$1secret\$1scanning\$1custom\$1pa....publish + repository\$1secret\$1scanning\$1custom\$1p....enabled + repository\$1secret\$1scanning\$1custom\$1p....disabled + repository\$1secret\$1scanning\$1non\$1provi....enabled + repository\$1secret\$1scanning\$1non\$1provi....disabled + repository\$1secret\$1scanning\$1generic\$1s....enabled + repository\$1secret\$1scanning\$1generic\$1s....disabled + business\$1secret\$1scanning\$1custom\$1pattern.create + business\$1secret\$1scanning\$1custom\$1pattern.update + business\$1secret\$1scanning\$1custom\$1pattern.delete + business\$1secret\$1scanning\$1custom\$1pattern.publish + business\$1secret\$1scanning\$1custom\$1patt....enabled + business\$1secret\$1scanning\$1custom\$1patt....disabled + business\$1secret\$1scanning\$1generic\$1secrets.enabled + business\$1secret\$1scanning\$1generic\$1secrets.disabled + business\$1secret\$1scanning\$1non\$1provide....enabled + business\$1secret\$1scanning\$1non\$1provide....disabled + org\$1secret\$1scanning\$1non\$1provider\$1patt....enabled + org\$1secret\$1scanning\$1non\$1provider\$1patt....disabled + org\$1secret\$1scanning\$1generic\$1secrets.enabled + org\$1secret\$1scanning\$1generic\$1secrets.disabled + org\$1secret\$1scanning\$1custom\$1pattern.create + org\$1secret\$1scanning\$1custom\$1pattern.update + org\$1secret\$1scanning\$1custom\$1pattern.delete + org\$1secret\$1scanning\$1custom\$1pattern.publish **Entity Management** contains the following actions: + oauth\$1application.destroy + oauth\$1application.generate\$1client\$1secret + oauth\$1application.remove\$1client\$1secret + oauth\$1application.revoke\$1all\$1tokens + oauth\$1application.revoke\$1tokens + oauth\$1application.transfer + personal\$1access\$1token.auto\$1approve\$1grant\$1requests\$1enabled + personal\$1access\$1token.auto\$1approve\$1grant\$1requests\$1disabled + ip\$1allow\$1list.disable + ip\$1allow\$1list.enable\$1for\$1installed\$1apps + ip\$1allow\$1list.disable\$1for\$1installed\$1apps + ip\$1allow\$1list\$1entry.create + ip\$1allow\$1list\$1entry.update + ip\$1allow\$1list\$1entry.destroy + repository\$1secret\$1scanning.disable + repository\$1secret\$1scanning\$1automatic....disabled + repository\$1secret\$1scanning\$1push\$1prot....disable + repository\$1secret\$1scanning\$1push\$1prot....enable + oauth\$1application.create + oauth\$1application.reset\$1secret + auto\$1approve\$1personal\$1access\$1token\$1req....enabled + auto\$1approve\$1personal\$1access\$1token\$1req....disabled + ip\$1allow\$1list.enable + ip\$1allow\$1list.disable\$1user\$1level\$1enforcement + ip\$1allow\$1list.enable\$1user\$1level\$1enforcement + repository\$1secret\$1scanning.enable + repository\$1secret\$1scanning\$1automatic....enabled + repository\$1secret\$1scanning\$1push\$1prot....enable + repository\$1secret\$1scanning\$1push\$1prot....add + repository\$1secret\$1scanning\$1push\$1prot....remove + repository\$1secret\$1scanning\$1push\$1prot....disable + secret\$1scanning.enable + secret\$1scanning.disable + secret\$1scanning\$1new\$1repos.enable + org\$1secret\$1scanning\$1automatic\$1validi....enabled + org\$1secret\$1scanning\$1automatic\$1validi....disabled + org\$1secret\$1scanning\$1push\$1protection\$1b....add + org\$1secret\$1scanning\$1push\$1protection\$1b....remove + org\$1secret\$1scanning\$1push\$1protection\$1b....disable + org\$1secret\$1scanning\$1push\$1protection\$1b....enable + business\$1secret\$1scanning\$1automatic\$1va....enabled + business\$1secret\$1scanning\$1automatic\$1va....disabled + business\$1secret\$1scanning\$1push\$1protection.enable + business\$1secret\$1scanning\$1push\$1protection.disable + business\$1secret\$1scanning\$1push\$1protection.enabled\$1for\$1new\$1repos + business\$1secret\$1scanning\$1push\$1protection.disabled\$1for\$1new\$1repos + business\$1secret\$1scanning\$1push\$1prote....enable + business\$1secret\$1scanning\$1push\$1prote....update + business\$1secret\$1scanning\$1push\$1prote....disable # CloudWatch pipelines configuration for GitHub Audit Log **Note** Important: GitHub Enterprise accounts are required to use this connector. GitHub Personal or Organization accounts are not supported. Collects audit log data from GitHub organizations or enterprises using personal access tokens or GitHub App authentication. Configure the GitHub Audit Log source with the following parameters: ``` source: github_auditlog: scope: "ORGANIZATION" organization: "" authentication: personal_access_token: "${{aws_secrets::token}}" ```Parameters `scope` (required) Scope of audit logs to collect. Must be "ORGANIZATION" or "ENTERPRISE". `organization` (required when scope is ORGANIZATION) GitHub organization name. `enterprise` (required when scope is ENTERPRISE) GitHub enterprise name. `authentication.personal_access_token` (required for PAT auth) Personal access token for GitHub API authentication. # Microsoft 365 integration configuration This section describes how to configure Microsoft 365 as a data source for CloudWatch Pipeline. **Topics** + [ # Source configuration for Microsoft 365 ](microsoft365-source-setup.md) + [ # CloudWatch pipelines configuration for Microsoft 365 ](microsoft365-pipeline-setup.md) # Source configuration for Microsoft 365 ## Integrating with Microsoft 365 Microsoft 365 is a product family of productivity software, collaboration, and cloud-based services owned by Microsoft. CloudWatch Pipeline uses the Office 365 Management Activity API to retrieve information about user, admin, system, and policy actions and events from Office 365 and Microsoft Entra activity logs. Office 365 Management Activity API (also known as the Unified Auditing API) is a part of Office 365 security and compliance offerings. Customers and partners can use this information to create new or enhance existing operations, security, and compliance-monitoring solutions for the enterprise. ## Authenticating with Office 365 Management Activity API To retrieve the activities Office 365, pipelines needs to authenticate with your account. Follow the instructions in Office 365 Management APIs: + Register an application in Azure with Supported account types, Accounts in this organizational directory only (Single tenant). After registration is complete, note down the Application (client) ID and Directory (tenant) ID. + Generate a new key for your application. Key is also known as client secret, which are used when exchanging an authorization code for an access token. + In the AWS Secrets Manager, create a secret and store the Application (client) ID under the key `client_id` and the client secret under the key `client_secret` + Specify the permissions your application requires to access the Office 365 Management APIs. The permissions you need are: + ActivityFeed.Read: Required for all the audit content types you listed, including Audit.AzureActiveDirectory, Audit.Exchange, Audit.SharePoint, and Audit.General. + ActivityFeed.ReadDlp: Required specifically for the DLP.All content type + Before you can access data through the Office 365 Management Activity API, you must enable unified audit logging for your Office 365 organization. You do this by turning on the Office 365 audit log. For instructions, see Turn Office 365 audit log search on or off. ## Configuring the CloudWatch Pipeline When configuring the pipeline to read activities from Office 365, choose Microsoft 365 as the data source. Fill in the required information like Tenant Id using Directory (tenant) ID and the secret where `client_id` and `client_secret` are stored. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and the actions from various workloads like Exchange, SharePoint, Teams, and Azure Active Directory are mapped to Account Change (3001), Authentication (3002), User Access Management (3005), Group Management (3006), Email Activity (4009), Web Resources Activity (6001), File Hosting Activity (6006), Application Lifecycle (6002), (2003), Detection Finding (2004), Incident Finding (2005), Vulnerability Finding (2002) and Unknown (0). ### Compliance Finding Compliance Finding contains the following actions: + ApplyRecordLabel + ComplianceSettingChanged + ExclusionConfigurationDeleted + NewComplianceTag + NewRetentionCompliancePolicy + NewRetentionComplianceRule + CreateRulePackage + CreateSuppressionRule + ApproveDisposal + RemoveComplianceTag + SetComplianceTag + SetRestrictiveRetentionUI + SupervisionPolicyCreated + SupervisionPolicyUpdated + SupervisionPolicyDeleted + HoldUpdated + HoldCreated + HoldRemoved + DlpInfo ### Detection Finding Detection Finding contains the following actions: + FileMalwareDetected + DocumentSensitivityMismatchDetected + TIMailData + DeviceOffBoarding + AddIndicator + ChangeCustomDetectionRuleStatus + CreateCustomDetection + DeleteIndicator + EditIndicator + MonitoringAlertUpdated + RunCustomDetection + Trigger CMD Agent Canary check. + DlpRuleMatch + AlertEntityGenerated + AlertTriggered ### Incident Finding Incident Finding contains the following actions: + AddCommentToIncident + AddTagsToIncident + AssignUserToIncident + CollectInvestigationPackage + EditIncidentClassification + RemediationActionAdded + RemediationActionUpdated + RemoveTagsFromIncident + UnAssignUserFromIncident + UpdateIncidentStatus + CaseUpdated + CaseAdded + CaseRemoved ### Account Change Account Change contains the following actions: + Add member to role + Add service principal + Add user + Added role + Change user license + Change user password + Delete user + Deleted app permission + Deleted role + Edited global role assignment + Edited role + NetworkUserSuspended + Remove member from role + Remove delegation entry + Reset user password + Set force change user password + AdministratorAddedToTermStore + AdministratorDeletedFromTermStore + AlertNotificationsRecipientDeleted + CaseAdminUpdated + CaseAdminAdded + CaseAdminRemoved + User added ### Authentication Authentication contains the following actions: + MailboxLogin + ClockedIn + ClockedOut + TeamsSessionStarted + Logon + SignInEvent + SSOUserCredentialsSet + User logged in + UserLoggedIn + UserLoggedOff + UserLoginFailed ### User Access Management User Access Management contains the following actions: + Add-MailboxPermission + ModifyFolderPermissions + Remove-MailboxPermission + ApplicableAdaptiveScopeChange + CaseMemberAdded ### Group Management Group Management contains the following actions: + RemovedFromSecureLink + BotAddedToTeam + BotRemovedFromTeam + MemberAdded + MemberRemoved + MemberRoleChanged + ScheduleGroupAdded + ScheduleGroupEdited + ScheduleGroupDeleted + TeamCreated + TeamDeleted + Add group + Add member to group + Created group + Delete group + Deleted group + Edited group memberships + Edited group + GroupCreation + GroupDeletion + GroupRemoved + GroupAdded + GroupUpdated + RemovedFromGroup + AddedToGroup + Remove member from group + RemoveSpecificResponder + RosterMemberAdded + RosterMemberDeleted + CaseMemberUpdated + CaseMemberRemoved + Team added + Team deleted + UserAddedToGroup + UserRemovedFromGroup ### Email Activity Email Activity contains the following actions: + Send + SendAs + SendOnBehalf + MessageDeletedNotification + QuarantineDelete + QuarantineExport + QuarantinePreview + QuarantineRelease + QuarantineReleaseRequest + QuarantineReleaseRequestDeny + QuarantineViewHeader + SupervisionRuleMatch + SupervisoryReviewTag + SupervisoryReviewOLAudit ### Web Resources Activity Web Resources Activity contains the following actions: + UpdateCalendarDelegation + AddFolderPermissions + Copy + Create + New-InboxRule + SoftDelete + Move + MailItemsAccessed + MoveToDeletedItems + Set-InboxRule + HardDelete + UpdateInboxRules + Update + LockRecord + UnlockRecord + SearchQueryPerformed + PageViewed + PageViewedExtended + FolderCreated + ClientViewSignaled + PagePrefetched + FolderModified + ListColumnCreated + ListContentTypeCreated + ListItemCreated + Site ContentType Created + List Column Deleted + ListCreated + List Item Deleted + SiteColumnDeleted + ListDeleted + ListContentTypeDeleted + ListRestored + SiteColumnCreated + ListItemRecycled + ListItemDeleted + ListItemRestored + ListContentTypeUpdated + ListUpdated + ListViewed + SiteContentTypeDeleted + ListItemUpdated + SiteColumnUpdated + AccessRequestAccepted + ListColumnUpdated + SiteContentTypeUpdated + AccessRequestCreated + PermissionLevelAdded + CompanyLinkCreated + AnonymousLinkCreated + SharingInvitationAccepted + SecureLinkCreated + SharingInvitationCreated + SecureLinkDeleted + CompanyLinkRemoved + AccessRequestDenied + AnonymousLinkRemoved + AccessRequestUpdated + SharingSet + AnonymousLinkUpdated + SharingInvitationBlocked + AnonymousLinkUsed + SecureLinkUsed + CompanyLinkUsed + SharingRevoked + AddedToSecureLink + SharingInvitationUpdated + SharingInvitationRevoked + ExemptUserAgentSet + AllowedDataLocationAdded + SiteGeoMoveCancelled + AllowGroupCreationSet + CustomizeExemptUsers + DeviceAccessPolicyChanged + NetworkAccessPolicyChanged + SiteCollectionCreated + SiteDeleted + SendToConnectionRemoved + SiteGeoMoveCompleted + SharingPolicyChanged + PreviewModeEnabledSet + HubSiteOrphanHubDeleted + SendToConnectionAdded + HubSiteJoined + SiteCollectionQuotaModified + LegacyWorkflowEnabledSet + OfficeOnDemandSet + NewsFeedEnabledSet + PeopleResultsScopeSet + AllowedDataLocationDeleted + SiteRenamed + HubSiteRegistered + HostSiteSet + GeoQuotaAllocated + HubSiteUnjoined + HubSiteUnregistered + SiteCollectionAdminAdded + PermissionLevelsInheritanceBroken + SharingInheritanceBroken + SiteGeoMoveScheduled + WebRequestAccessModified + WebMembersCanShareModified + PermissionLevelModified + PermissionLevelRemoved + SitePermissionsModified + SiteCollectionAdminRemoved + SiteAdminChangeRequest + SharingInheritanceReset + BreakEnded + ChannelAdded + BreakStarted + ChannelDeleted + ChannelOwnerResponded + ChatRetrieved + ChannelSettingChanged + ChatCreated + ChatUpdated + ConnectorAdded + ConnectorRemoved + ConnectorUpdated + CreateUpdateRequest + EditUpdateRequest + FailedValidation + InviteeResponded + InviteSent + MeetingDetail + MeetingParticipantDetail + MessageCreatedHasLink + MessageDeleted + MessageCreatedNotification + MessageEditedHasLink + MessageHostedContentRead + MessageRead + MessageReadReceiptReceived + MessageHostedContentsListed + MessageSent + MessagesExported + MessageUpdated + MessageUpdatedNotification + OffShiftDialogAccepted + MessagesListed + OpenShiftAdded + OpenShiftDeleted + OpenShiftEdited + PerformedCardAction + RequestAdded + RequestRespondedTo + RequestCancelled + ScheduleSettingChanged + ScheduleShared + SensitivityLabelApplied + ScheduleWithdrawn + SensitivityLabelChanged + SensitivityLabelRemoved + SharingRestored + ShiftAdded + ShiftDeleted + ShiftEdited + SubscribedToMessages + TabAdded + SubmitUpdate + TabRemoved + TabUpdated + TeamSettingChanged + TeamsTenantSettingChanged + TerminatedSharing + TimeClockEntryDeleted + TimeClockEntryAdded + TimeClockEntryEdited + TimeOffAdded + TimeOffEdited + ViewUpdate + TimeOffDeleted + TranscriptsExported + AccessedOdataLink + AcceptedSharingLinkOnFolder + Add delegation entry. + Add domain to company. + Add service principal credentials. + Add partner to company. + Update service principal. + AddedDataLossPreventionEvaluationResult + AddFormCoauthor + AddReviewer + AddSpecificResponder + Admin allowed third party apps + Admin modified app owner + Admin modified app permissions + Admin set app as featured + Admin set bypass consent state + Admin set conditional access + Admin set desired logical name + Admin set quarantine state + AlertExcelDownloaded + AlertNotificationsRecipientAdded + AllowAnonymousResponse + AllowShareFormForCopy + AppBypassInformationBarrier + CanceledQuery + Check PowerShell Execution Policy + ClassificationDefinitionDeleted + ClassificationAdded + ClassificationDefinitionUpdated + ClassificationDeleted + ClassificationDefinitionCreated + CollectionHardDeleted + CollectionCreated + CollectionRenamed + CollectionSoftDeleted + Commented on video + CommunityAccessFailure + CollectionUpdated + Consented to the app's APIs + ConnectToExcelWorkbook + Create LogCollection Request + Create new work items (Scheduler) + ConsentModificationRequest + Create Remote Action Operation in Acti... + CreateComment + CreateForm + CreateResponse + Dashboard created + Dashboard deleted + Dashboard updated + Data exported + DataAccessRequestOperation + DataExport + DataShareCreated + DeleteAllResponses + DeleteCustomDetection + Deleted video + DeletedResult + DeleteSummaryLink + DisableCollaboration + DisableSpecificResponse + DisallowShareFormForCopy + DisableSuppressionRule + DisallowAnonymousResponse + EditCustomDetection + Edited app + Edited app permission + Edited global role assignment + Edited channel + Edited tenant settings + Edited group + Edited user settings + Edited role + Edited video permission + EditForm + Edited video + EditRulePackage + EnableSameOrgCollaboration + EditSuppressionRule + EnableSpecificCollaboaration + EnableSpecificResponse + EnableSuppressionRule + EnableWorkOrSchoolCollaboration + EntityCreated + EntityDeleted + EntityRemediatorConfigurationUpdated + EntityUpdated + ExclusionConfigurationAdded + ExclusionConfigurationUpdated + ExecutedQuery + ExportForm + ExtendRetention + FileUpdateDescription + FileUpdateName + FileVisited + FolderSharingLinkShared + SharingLinkUsed + SharingLinkCreated + GenerateCopyOfLakeData + Get text track + Get transcript + Get video + GetSummaryLink + GlossaryTermAssigned + GlossaryTermCreated + GlossaryTermDisassociated + GlossaryTermDeleted + GlossaryTermUpdated + Goals policy updated + Group view + InformationBarriersInsightsReportOneDr... + InformationBarriersInsightsReportSched... + InformationBarriersInsightsReportShare... + InformationBarriersInsightsReportCompl... + Liked video + Linked on Video + LinkedEntityCreated + LinkedEntityDeleted + LinkedEntityUpdated + ListForms + Marked app as Featured + Marked app as Hero + MarkedMessageChanged + ReactedToMessage + MeetingExclusionCreated + MessageCreated + MessageAccessFailure + MessageViewed + MonitoringAlertNotificationRecipientAd... + MonitoringAlertNotificationRecipientDe... + MovedFormIntoCollection + MovedFormOutofCollection + NetworkConfigurationUpdated + NetworkSecurityConfigurationUpdated + MoveForm + NewAdaptiveScope + NotificationConfigurationUpdated + OCE Run Commands on VM + OKR or Project created + OKR or Project deleted + OKR or Project updated + Organization created + Organization integrations updated + Organization settings updated + PlanCreated + PlanCopied + PlanDeleted + PlanRead + Post Remote Action Operation + PlanListRead + PreviewForm + PlanModified + ProcessProfileFields + ProjectCreated + ProjectAccessed + ProInvitation + ProjectDeleted + ProjectForTheWebRoadmaptSettings + ProjectForTheWebProjectSettings + ProjectListAccessed + ProjectUpdated + RelabelItem + ReleaseFromIsolation + Remove domain from company. + Remove partner from company. + Remove service principal credentials. + RemoveAdaptiveScope + RemoveAppRestrictions + RemoveFormCoauthor + RemoveRetentionComplianceRule + RemoveRetentionCompliancePolicy + ReporterConfigurationUpdated + RestrictAppExecution + RoadmapAccessed + RoadmapCreated + RoadmapDeleted + RoadmapItemAccessed + RoadmapItemCreated + RoadmapItemDeleted + RoadmapItemUpdated + RoadmapUpdated + RosterCreated + RosterDeleted + RosterSensitivityLabelUpdated + Run hybrid AADJ extension + RunLiveResponseApi + SensorCreated + SensorConfigurationUpdated + SensorDeleted + SensorDeploymentAccessKeyUpdated + SensorDeploymentAccessKeyReceived + Set company contact information + Set channel thumbnail + Set delegation entry + Set company information + Set domain authentication + Set federation settings on domain + Set DirSyncEnabled flag + Set license properties + Set password policy + SetAdaptiveScope + SetAdvancedFeatures + SetRetentionCompliancePolicy + SiteIBModeChanged + Shared video + SiteIBModeSet + SetRetentionComplianceRule + SiteIBSegmentsChanged + SiteIBSegmentsRemoved + SiteIBSegmentsSet + SiteSensitivityLabelApplied + SensitivityLabelUpdated + SiteSensitivityLabelChanged + SiteSensitivityLabelRemoved + SoftDeleteSettingsUpdated + SPOIBIsDisabled + SPOIBIsEnabled + SubmitResponse + SubTaskCreated + SubTaskDeleted + SubTaskUpdated + SupervisorAdminToggled + SyslogServiceConfigurationUpdated + TaggingConfigurationUpdated + TaskAccessed + TaskAssigned + TaskCompleted + TaskDeleted + TaskCreated + TaskListCreated + TaskListRead + TaskListUpdated + TaskModified + TaskRead + TaskUpdated + Team updated + TenantSettingsUpdated + Trigger device remediation + Trigger generic action by SaaF + Trigger generic action + Trigger generic action with options + Unliked video + Trigger orchestrator + Update group. + Update user. + UpdatedDataAccessSetting + UpdatedOrganizationBriefingSettings + UpdatedOrganizationMyAnalyticsSettings + Update domain. + UpdatedPrivacySetting + UpdatedUserBriefingSettings + UpdatedUserMyAnalyticsSettings + UpdateFormSetting + UpdatePhishingStatus + UpdateResponse + UpdateUsageReportsPrivacySetting + UpdateUserSetting + URbacAuthorizationStatusChanged + UserInvited + UserSuspension + Viewed video + Verify domain + Verify email verified domain + ViewedExplore + ViewForm + ViewResponses + ViewRuntimeForm + ViewResponse + VpnConfigurationUpdated + WorkspaceCreated + WorkspaceDeleted + WorkspaceAlertThresholdLevelUpdated + SearchUpdated + SearchPermissionUpdated + PreviewItemListed + SearchCreated + SearchPermissionCreated + SearchRemoved + SearchExportDownloaded + SearchPreviewed + SearchPermissionRemoved + SearchResultsPurged + RemovedSearchResultsSentToZoom + RemovedSearchPreviewed + RemovedSearchExported + RemovedSearchResultsPurged + SearchResultsSentToZoom + SearchReportRemoved + SearchStarted + SearchReport + ThreadViewed + CaseViewed + SearchViewed + ViewedSearchExported + SearchStopped + ViewedSearchPreviewed + AddWorkingSetQueryToWorkingSet + AddQueryToWorkingSet + AddNonOffice365DataToWorkingSet + AnnotateDocument + LoadComparisonJob + RunAlgo + CreateWorkingSet + CreateWorkingSetSearch + CreateTag + DeleteTag + UpdateTag + DeleteWorkingSetSearch + UpdateCaseSettings + UpdateWorkingSetSearch + PreviewWorkingSetSearch + TagJob + LabelContentExplorerAccessedItem + AccessInvitationAccepted + AccessInvitationCreated + AccessInvitationExpired + AccessInvitationRevoked + AccessInvitationUpdated + AccessRequestApproved + AccessRequestRejected + AppCatalogCreated + AuditPolicyUpdate + ActivationEnabled + AuditPolicyRemoved + AzureStreamingEnabledSet + CollaborationTypeModified + CreateSSOApplication + ConnectedSiteSettingModified + CustomFieldOrLookupTableCreated + CustomFieldOrLookupTableDeleted + CustomFieldOrLookupTableModified + DelegateModified + DelegateRemoved + DefaultLanguageChangedInTermStore\$1 + eDiscoveryHoldApplied + eDiscoveryHoldRemoved + eDiscoverySearchPerformed + EngagementAccepted + EngagementModified + EnterpriseCalendarModified + EngagementRejected + EntityForceCheckedIn + LanguageAddedToTermStore + LookAndFeelModified + LanguageRemovedFromTermStore + MaxQuotaModified + MaxResourceUsageModified + MySitePublicEnabledSet + ODBNextUXSettings + PermissionSyncSettingModified + PermissionTemplateModified + PortfolioDataAccessed + PortfolioDataModified + ProjectCheckedOut + ProjectCheckedIn + ProjectModified + ProjectPublished + ProjectWorkflowRestarted + PWASettingsAccessed + ProjectForceCheckedIn + PWASettingsModified + QueueJobStateModified + QuotaWarningEnabledModified + RenderingEnabled + ReportingAccessed + ResourceCheckedIn + ResourceAccessed + ReportingSettingModified + ResourceCreated + ResourceCheckedOut + ResourceModified + ResourcePlanCheckedInOrOut + ResourceDeleted + ResourcePlanModified + ResourcePlanPublished + ResourceForceCheckedIn + ResourceWarningEnabledModified + ResourceRedacted + SSOGroupCredentialsSet + SearchCenterUrlSet + SecondaryMySiteOwnerSet + SecurityCategoryModified + SecurityGroupModified + SiteCollectionAdminAdded\$1 + StatusReportModified + SyntexBillingSubscriptionSettingsChang... + TaskStatusAccessed + TaskStatusApproved + TaskStatusRejected + TaskStatusSubmitted + TaskStatusSaved + TimesheetRejected + TimesheetApproved + TimesheetSaved + TimesheetSubmitted + TimesheetAccessed + UpdateSSOApplication + WorkflowModified + DlpRuleUndo + AlertUpdated + SensitivityLabelPolicyMatched + CopilotInteraction + Channel view + Deleted video comment + Deleted channel + Created channel + Created video + User deactivated + User deleted ### Application Lifecycle Application Lifecycle contains the following actions: + AppDeletedFromCatalog + AppPublishedToCatalog + AppInstalled + AppUninstalled + AppUpdatedInCatalog + AppUpgraded + DeletedAllOrganizationApps + WorkforceIntegrationAdded + AddDevicesToBackfill Operation + AddDevicesToReinstall Operation + Admin deleted app + Admin restored deleted app + Create VmExtention Request + Created app + Deleted app + Deleted app version + Execute AppHealthPlugin + Install RD agent + Update device. + MigrationJobCompleted + Patched app + Published app + Remove service principal. + Removed app as Featured + Removed app as Hero + TriggerClientAgentCheckBulkAction Opera... + Launched app + LaunchPowerApp + DeleteSSOApplication ### File Hosting Activity File Hosting Activity contains the following actions: + UpdateFolderPermissions + FileCheckedIn + FileCheckedOut + FileCopied + FileAccessedExtended + FileDeletedSecondStageRecycleBin + FileDeleted + FileAccessed + FileDeletedFirstStageRecycleBin + RecordDelete + FileDownloaded + FileCheckOutDiscarded + FileModified + FileModifiedExtended + FilePreviewed + FileRecycled + FolderRecycled + FileVersionsAllMinorsRecycled + FileMoved + FileVersionRecycled + FileUploaded + FileRenamed + FileVersionsAllRecycled + FileRestored + FolderDeleted + FolderDeletedFirstStageRecycleBin + FolderMoved + FolderCopied + FolderDeletedSecondStageRecycleBin + FolderRenamed + FolderRestored + RecordingExported + ManagedSyncClientAllowed + FileSyncDownloadedFull + FileSyncDownloadedPartial + FileSyncUploadedFull + UnmanagedSyncClientBlocked + FileSyncUploadedPartial + AttachmentDeleted + AttachmentUpdated + AttachmentCreated + DataShareDeleted + Deleted text track + Deleted thumbnail + DomainControllerCoverageExcelDownloaded + DownloadCopyOfLakeData + Downloaded video + DownloadedReport + DownloadOffboardingPkg + DownloadFile + DownloadOnboardingPkg + FileAccessFailure + FileCreated + FileSensitivityLabelChanged + FileSensitivityLabelApplied + FileSensitivityLabelRemoved + FileShared + WACTokenShared + LiveResponseGetFile + LogsCollection + AddRemediatedData + BurnJob + DownloadDocument + ExportJob + ErrorRemediationJob + TagFiles + PreviewItemRendered + ViewDocument + FileFetched + FileViewed + SharedLinkCreated + SharedLinkDisabled + SharingInvitationAccepted\$1 + SyncGetChanges + Restored app version + RunAntiVirusScan + StopAndQuarantineFile + Uploaded text track + Upload folder to blob + Uploaded thumbnail + Uploaded video + UploadedOrgData + ReportDownloaded + PreviewItemDownloaded + SearchExported + Published solution canvas app version # CloudWatch pipelines configuration for Microsoft 365 Collects log data from Microsoft Office 365 Management API using OAuth2 authentication. Configure the Microsoft Office 365 source with the following parameters: ``` source: microsoft_office365: tenant_id: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `tenant_id` (required) The Microsoft Entra ID tenant ID for your Office 365 organization. `authentication.oauth2.client_id` (required) OAuth2 client ID for Microsoft Office 365 Management API authentication. Can reference AWS Secrets Manager using `${{aws_secrets:secret-name:key}}` syntax. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Microsoft Office 365 Management API authentication. Can reference AWS Secrets Manager using `${{aws_secrets:secret-name:key}}` syntax. **Note** Store sensitive credentials like client IDs and secrets in AWS Secrets Manager and reference them using the `${{aws_secrets:secret-name:key}}` syntax in your configuration. # Cisco Umbrella To integrate Cisco Umbrella Data Replicator with CloudWatch Logs, you must configure both the source and the pipeline. First, set up your Cisco Umbrella source by configuring Amazon S3 and Amazon SQS to receive data. Then, configure the CloudWatch pipeline to ingest the data from your source into CloudWatch Logs. **Topics** + [ # Source configuration for Cisco Umbrella ](cisco-umbrella-source-setup.md) + [ # Pipeline configuration for Cisco Umbrella ](cisco-umbrella-pipeline-setup.md) # Source configuration for Cisco Umbrella ## Integrating with Cisco Umbrella Cisco Umbrella is a cloud-delivered security platform that provides secure internet access and threat protection across all devices, locations, and users. It uses DNS-layer security, web filtering, and cloud-delivered firewall features to block malicious domains and prevent cyberattacks before they reach your network. CloudWatch pipelines enables you to collect this data in CloudWatch Logs. ## Instructions to setup Amazon S3 and Amazon SQS Configuring Cisco Umbrella to send logs to an Amazon S3 bucket involves several steps, primarily focused on setting up the Amazon S3 bucket, Amazon SQS queue, and IAM roles, then configuring the CloudWatch pipeline. + Ensure Cisco Umbrella logs environment exporter is configured with S3. This is typically found under Admin → Logs Management in the Cisco Umbrella console. + Amazon S3 bucket that stores the Cisco Umbrella logs should reside in the same AWS region as your CloudWatch pipeline. + Create an Amazon SQS queue in the same AWS region as your Amazon S3 bucket. This queue will receive notifications when new log files are added to the Amazon S3 bucket. + Configure the Amazon S3 bucket to create event notifications, specifically for "Object Create" events. These notifications should be sent to the Amazon SQS queue you created in the previous step. ## Configuring the CloudWatch Pipeline When configuring the pipeline to read data from Cisco Umbrella, choose Cisco Umbrella as the data source. After filling in the required information and creating the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and [Cisco Umbrella events](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152763.dita) that map to DNS Activity (4003), Network Activity (4001), Data Security Finding (2006), and Entity Management (3004). Each event comes from a source as mentioned below. **DNS Activity** contains the following actions: + [DNS Logs](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152744.dita) **Network Activity** contains the following actions: + [Cloud Firewall Logs](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152740.dita) + [Secure Gateway logs](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152788.dita) + [IPS logs](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152760.dita) **Data Security Finding** contains the following actions: + [DLP (Data Loss Prevention) logs](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152741.dita) **Entity Management** contains the following actions: + [Admin Audit logs](https://securitydocs.cisco.com/docs/umbrella-sig/olh/152738.dita) # Pipeline configuration for Cisco Umbrella The Cisco Umbrella setup on AWS reads log data from Amazon S3 buckets using Amazon SQS notifications for new object events. Configure the Cisco Umbrella source using the following parameters: ``` source: s3: aws: region: "us-east-1" sts_role_arn: "arn:aws:iam:::role/" compression: "gzip" codec: csv: data_source_name: "cisco_umbrella" default_bucket_owner: "123456789012" bucket_owners: my-bucket: "123456789012" disable_bucket_ownership_validation: false notification_type: "sqs" sqs: queue_url: "https://sqs.region.amazonaws.com//" on_error: "retain_messages" ```Parameters `notification_type` (required) Specifies the notification mechanism. Must be "sqs" to use SQS for S3 event notifications. `data_source_name` (required) Identifies the data source. This can be any string value that represents your data source. Example: "cisco\$1umbrella". `aws.region` (required) The AWS region where the S3 bucket and SQS queue are located. `aws.sts_role_arn` (required) The ARN of the IAM role to assume for accessing S3 and SQS resources. `codec` (required) Codec configuration for parsing S3 objects. Supports csv, json, ndjson codecs. `compression` (optional) Compression type of the S3 objects. Valid values are "none", "gzip", "automatic". Defaults to "none". `sqs.queue_url` (required for SQS) The complete SQS queue URL that receives S3 bucket notifications when new objects are created. `on_error` (optional) Determines how to handle errors in Amazon SQS. Can be either retain\$1messages or delete\$1messages. Default is retain\$1messages. # PingIdentity PingOne integration configuration PingOne is a cloud-based identity and access management (IAM) platform offered by Ping Identity, designed to provide secure access to applications and services across various platforms. **Topics** + [ # Source configuration for PingIdentity PingOne ](pingidentity-pingone-source-setup.md) + [ # CloudWatch pipelines configuration for PingIdentity PingOne ](pingidentity-pingone-pipeline-setup.md) # Source configuration for PingIdentity PingOne ## Integrating with PingIdentity PingOne PingOne is Ping Identity's cloud-based identity-as-a-service (IDaaS) platform that provides identity and access management capabilities. CloudWatch Pipeline uses the PingOne Audit Logs API to retrieve information about authentication events, user activities, policy decisions, and administrative changes across your PingOne environment. The Audit Logs API enables access to event data through REST endpoints, allowing retrieval of security and access logs from your PingOne organization. ## Authenticating with PingIdentity PingOne To read the logs, the pipeline needs to authenticate with your PingOne environment. For PingOne, authentication is performed using OAuth2. **Configure OAuth2 authentication for PingOne** + Log in to the PingOne Console and navigate to Applications → Applications. Create a new application of type Worker. Note the Client ID and Environment ID. + Generate a new Client Secret from the Configuration tab. Copy the secret immediately. + In AWS Secrets Manager, create a secret and store the Client ID under key `client_id` and the client secret under key `client_secret`. + Assign Environment Admin and Application Owner roles to the application. + Identify your PingOne Region (NA, EU, AP, AU, CA, SG). + Note the Environment ID from Settings → Environment → Properties. ## Configuring the CloudWatch Pipeline To configure the pipeline to read logs, choose PingOne as the data source. Fill in the required information like Environment ID. Optionally, specify the Region (defaults to NA) and the Range duration format (for example, PT21H for the last 21 hours). The default range is 0 hours, and the maximum is 90 days. Once you create and activate the pipeline, audit log data from PingOne will begin flowing into the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and PingOne events that map to Account Change (3001), Authentication (3002), and Entity Management (3004). **Account Change** contains the following events: + USER.CREATED + USER.INVITED + USER.REINVITED + USER.INVITE\$1ACCEPTED + PASSWORD.FORCE\$1CHANGE + PASSWORD.RECOVERY + PASSWORD.RESET + USER.INVITE\$1REVOKED + USER.DELETED + USER.LOCKED + MFA\$1SETTINGS.UPDATED + PASSWORD.UNLOCKED + USER.UNLOCKED **Authentication** contains the following events: + AUTHENTICATION.CREATED + RADIUS\$1SESSION.CREATED + SESSION.CREATED + SESSION.UPDATED + SESSION.DELETED + USER.SLO\$1FAILURE + USER.SLO\$1PARTIAL\$1LOGOUT + USER.SLO\$1REQUESTED + USER.SLO\$1SUCCESS + USER.KERBEROS\$1FAILED + USER.KERBEROS\$1SUCCEEDED + DEVICE.ACTIVATION\$1OTP\$1FAILED + DEVICE.ACTIVATION\$1OTP\$1INVALID + DEVICE\$1PAYLOAD.CHECK\$1INVALID + DEVICE\$1PAYLOAD.CHECK\$1SUCCESS + OTP.CHECK\$1FAILED + OTP.CHECK\$1INVALID + OTP.CHECK\$1SUCCESS + PASSWORD.CHECK\$1FAILED + PASSWORD.CHECK\$1SUCCEEDED **Entity Management** contains the following events: + ACTION.CREATED + AGREEMENT.CREATED + AGREEMENT\$1LANGUAGE.CREATED + AGREEMENT\$1LANGUAGE\$1REVISION.CREATED + APPLICATION.CREATED + AUTHORIZE\$1POLICY.CREATED + CERTIFICATE.CREATED + DEVICE.CREATED + DEVICE\$1AUTHENTICATION\$1POLICY.CREATED + FIDO\$1POLICY.CREATED + FLOW.CREATED + FLOW\$1DEFINITION.CREATED + FLOW\$1EXECUTION.CREATED + GROUP.CREATED + IDENTITY\$1PROVIDER.CREATED + IDP\$1ATTRIBUTE.CREATED + INSTANT\$1MESSAGING\$1DELIVERY\$1SETTINGS.CREATED + KEY.CREATED + LICENSE.CREATED + NOTIFICATION.CREATED + NOTIFICATION\$1POLICY.CREATED + ORGANIZATION.CREATED + POLICY.CREATED + RISK\$1POLICY\$1SET.CREATED + SAML\$1ATTRIBUTE.CREATED + SCHEMA\$1ATTRIBUTE.CREATED + SIGN\$1ON\$1POLICY\$1ASSIGNMENT.CREATED + VERIFY\$1POLICY.CREATED + CERTIFICATE.READ + KEY.READ + SECRET.READ + ACTION.UPDATED + ADMIN\$1CONFIGURATION.UPDATED + AGREEMENT.UPDATED + AGREEMENT\$1LANGUAGE.UPDATED + AGREEMENT\$1LANGUAGE\$1REVISION.UPDATED + APPLICATION.UPDATED + AUTHORIZE\$1POLICY.UPDATED + CERTIFICATE.UPDATED + DEVICE.NICKNAME\$1UPDATED + DEVICE.UPDATED + DEVICE\$1AUTHENTICATION\$1POLICY.UPDATED + FIDO\$1POLICY.UPDATED + FLOW.UPDATED + FLOW\$1DEFINITION.UPDATED + FLOW\$1EXECUTION.UPDATED + GROUP.UPDATED + IDENTITY\$1PROVIDER.UPDATED + IDP\$1ATTRIBUTE.UPDATED + INSTANT\$1MESSAGING\$1DELIVERY\$1SETTINGS.UPDATED + KEY.UPDATED + LICENSE.UPDATED + NOTIFICATION.UPDATED + NOTIFICATION\$1POLICY.UPDATED + NOTIFICATIONS\$1SETTINGS.UPDATED + ORGANIZATION.UPDATED + POLICY.UPDATED + RISK\$1POLICY\$1SET.ORDER\$1UPDATED + RISK\$1POLICY\$1SET.UPDATED + SAML\$1ATTRIBUTE.UPDATED + SCHEMA\$1ATTRIBUTE.UPDATED + SECRET.UPDATED + SETTINGS.UPDATED + SIGN\$1ON\$1POLICY\$1ASSIGNMENT.UPDATED + USER.QUOTA\$1RESET + USER.UPDATED + VERIFY\$1POLICY.UPDATED + ACTION.DELETED + AGREEMENT.DELETED + AGREEMENT\$1LANGUAGE.DELETED + AGREEMENT\$1LANGUAGE\$1REVISION.DELETED + APPLICATION.DELETED + AUTHORIZE\$1POLICY.DELETED + CERTIFICATE.DELETED + DEVICE.DELETED + DEVICE\$1AUTHENTICATION\$1POLICY.DELETED + FIDO\$1POLICY.DELETED + FLOW.DELETED + FLOW\$1DEFINITION.DELETED + GROUP.DELETED + IDENTITY\$1PROVIDER.DELETED + IDP\$1ATTRIBUTE.DELETED + INSTANT\$1MESSAGING\$1DELIVERY\$1SETTINGS.DELETED + KEY.DELETED + LICENSE.DELETED + NOTIFICATION\$1POLICY.DELETED + ORGANIZATION.DELETED + POLICY.DELETED + RISK\$1POLICY\$1SET.DELETED + SAML\$1ATTRIBUTE.DELETED + SCHEMA\$1ATTRIBUTE.DELETED + SIGN\$1ON\$1POLICY\$1ASSIGNMENT.DELETED + VERIFY\$1POLICY.DELETED + DEVICE.UNBLOCKED + DEVICE.BLOCKED + NOTIFICATION.REJECTED + DEVICE.ACTIVATED + DEVICE.LOCKED + DEVICE.UNLOCKED + ROLE.CREATED + ROLE.UPDATED + ROLE.DELETED # CloudWatch pipelines configuration for PingIdentity PingOne Collects audit logs from PingIdentity PingOne using OAuth2 authentication. Configure the PingIdentity PingOne source with the following parameters: ``` source: pingidentity_pingone: range: "P7D" region: "NA" environment_id: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `range` (optional) For pulling historical logs. Uses ISO 8601 duration format (for example, `P7D` for the last 7 days, `PT21H` for the last 21 hours). The default is 0 hours, and the maximum is 90 days. `region` (optional) PingOne region code (NA, EU, AP, AU, CA, SG). The default is "NA". `environment_id` (required) The PingOne environment ID. `authentication.oauth2.client_id` (required) OAuth2 client ID for PingIdentity PingOne API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for PingIdentity PingOne API authentication. # OneLogin Identity integration configuration OneLogin is a cloud-based identity and access management (IAM) platform that provides single sign-on (SSO), multi-factor authentication (MFA), and user provisioning capabilities. **Topics** + [ # Source configuration for OneLogin Identity ](onelogin-identity-source-setup.md) + [ # CloudWatch pipelines configuration for OneLogin Identity ](onelogin-identity-pipeline-setup.md) # Source configuration for OneLogin Identity ## Integrating with OneLogin Identity OneLogin is a cloud-based identity and access management (IAM) platform that provides single sign-on (SSO), multi-factor authentication (MFA), and user provisioning capabilities. CloudWatch pipelines uses the OneLogin Events API to retrieve information about authentication events, user activities, policy decisions, and administrative changes across your OneLogin environment. The Events API enables access to event data through REST endpoints, allowing retrieval of security and access logs from your OneLogin account. ## Authenticating with OneLogin Identity To read the logs, the pipeline needs to authenticate with your OneLogin account. For OneLogin, authentication is performed using OAuth2. **Configure OAuth2 authentication for OneLogin** + Log in to the OneLogin Admin Portal and navigate to Developers → API Credentials. Create a new API credential pair. Note the Client ID and Client Secret immediately. + Assign the appropriate permissions. Select Read All or Manage All scope to ensure the credentials can access event log data. + In AWS Secrets Manager, create a secret and store the Client ID under key `client_id` and the client secret under key `client_secret`. + Note your Account ID (subdomain) from the OneLogin Admin Portal under Settings → Account Settings. ## Configuring the CloudWatch Pipeline To configure the pipeline to read logs, choose OneLogin as the data source. Fill in the required information like subdomain and authentication credentials. Optionally, specify the Range duration format (for example, PT21H for the last 21 hours). Once you create and activate the pipeline, event log data from OneLogin will begin flowing into the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and OneLogin events that map to Account Change (3001), Authentication (3002), and Entity Management (3004). **Account Change** contains the following events: + User requested new password + Changed password for user + User deactivated + Password request approved from user + User locked + User suspended + User locked out of app + OTP device unlocked for user + User suspended in app + User suspended in directory + Unlocked user in directory + User granted permission to manage role + User permission to manage role revoked + User enabled desktop SSO + User disabled desktop SSO + Admin changed password for user + Redirected to an external site for password reset + API - password updated for user + API - user locked + User suspended via API + User locked via API + User enabled adaptive login for account + User disabled adaptive login for account + Profile change password + Manually added user to app + Manually removed user from app + Failed to change password for user + User granted permission to manage role failed + User permission to manage role revoked failed + Smart password updated for user + Smart password could not be updated for user + API - password not updated for user **Authentication** contains the following events: + User logged into OneLogin + User logged out of OneLogin + User logged into app + User logged out of app + User authenticated by RADIUS configuration + User authenticated via API + User successfully authenticated with VLDAP + User signed in into OneLogin via social network + User successfully authenticated with VLDAP (OneLogin Desktop Mac) + API - user logged out + API - verify factor called + API - confirm OTP for user succeeded + User was force logged out + User successfully logged in on a trusted device + User successfully logged in via OneLogin Desktop + User denied auth via OTP push request + User challenged for OTP + User reauthenticated into app + User verified OTP device + OIDC password for app success + API - trigger factor for user succeeded + OIDC implicit flow for app success + OIDC authorization code for app success + OIDC get code for app success + OIDC validate token for app success + User failed authentication + User failed to log into app + User rejected by RADIUS configuration + Failed to login to app via IDP + Could not authenticate to app + User failed authentication via API + User failed authentication with VLDAP + User authentication policy does not allow sign-in via social network + User failed authentication with VLDAP (OneLogin Desktop Mac) + API - user failed to log out + API - verify factor failed + API - confirm OTP for user failed + User failed to log in on a trusted device + User failed to login via OneLogin Desktop + User failed to authenticate via OneLogin Desktop + User failed OTP challenge + OIDC implicit flow for app failed + OIDC authorization code for app failed + OIDC password for app failed + OIDC validate token for app failed + OIDC general fail + OIDC get code for app failed **Entity Management** contains the following events: + Assigned role to user + User was created + User updated + User deactivated + User was activated + User was deleted + OTP device registered for user + OTP device deregistered for user + Updated credit card + User provisioned in app + User updated in app + User suspended in app + User reactivated in app + User deleted in app + Account granted permission to privilege + Account revoked permission to privilege + User granted permission to privilege + User permission to privilege revoked + Added trusted IDP + Removed trusted IDP + Modified trusted IDP + User provisioned in directory + User updated by directory + User suspended in directory + User reactivated in directory + User deleted in directory + Deleted secure note + Updated user login information + Attempted to update login information + Changed the default trusted IDP + User added to role + User removed from role + Created policy + Updated policy + Deleted policy + Created proxy agent + Deleted proxy agent + Created RADIUS configuration + Updated RADIUS configuration + Deleted RADIUS configuration + Enabled VPN + Updated VPN settings + Disabled VPN + Enabled embedding + Updated embedding settings + Disabled embedding + Created authentication factor + Updated authentication factor + Deleted authentication factor + Updated security questions + Updated desktop SSO settings + Enabled desktop SSO + Disabled desktop SSO + Created certificate + Deleted certificate + Created API credential + Deleted API credential + Enabled API credential + Disabled API credential + Enabled virtual LDAP + Disabled virtual LDAP + Updated virtual LDAP settings + Enabled branding + Disabled branding + Updated branding + Deleted mapping + Disabled mapping + Enabled mapping + Updated mapping + Deleted custom user fields + Updated company info + Updated account settings + Deleted directory + Deleted connector instance from directory + Created self registration + Updated self registration + Deleted self registration + Created payment record + Updated payment record + Deleted payment record + Updated terms and conditions for policy + Manually updated user login for app + User was created by trusted IDP + Directory external ID was updated for user + Directory external ID was deleted for user + Updated broadcaster + Deleted broadcaster + API - roles added to user + API - roles removed for user + API - user updated + API - user deleted + API - user created + Updated directory + OUs were updated for directory + User suspended via API + User reactivated via API + App was updated + Connector was created + Connector was updated + Connector was deleted + Parameter was created + Parameter was updated + Parameter was deleted + Deleted device for OneLogin Desktop + Revoked user certificate + Revoked device certificate + App was created via API + App was updated via API + App was destroyed via API + Sandbox deleted + Sandbox created + Sandbox updated + User deleted security factor + User renamed security factor + Created RADIUS attribute + Updated RADIUS attribute + Deleted RADIUS attribute + Role created + Role deleted + SMTP configuration updated + Smart hook created + Smart hook updated + Smart hook deleted + Smart hook environment variable created + Smart hook environment variable updated + Smart hook environment variable deleted + API - privilege was created + Created privilege + API - privilege was updated + Updated privilege + API - privilege was deleted + Deleted privilege + API - privilege was assigned to user + Assigned privilege to user + API - privilege removed from user + Removed privilege from user + API - privilege assigned to role + Assigned privilege to role + API - privilege removed from role + Removed privilege from role + Report created + Report updated + Report destroyed + Created group + Updated group + Destroyed group + Created secure note + API - app rules create success + API - app rules update success + API - app rules delete success + API - roles update success + Credit card update failed + User could not be updated + User could not be deleted in app + User could not be updated in app + User not updated in app + API - user not deleted + API - user not updated + API - user not created + Connector could not be created + Connector could not be updated + Connector could not be deleted + Parameter could not be created + Parameter could not be updated + Parameter could not be deleted + App failed to create via API + App failed to update via API + App failed to destroy via API + Failed to delete sandbox + Failed to create sandbox + Failed to update sandbox + Smart hook update failed + Smart hook environment variable update failed + API - app rules create failed + API - app rules update failed + API - app rules delete failed + User added to role failed + Role created failed + Role deleted failed + API - roles update failed # CloudWatch pipelines configuration for OneLogin Identity Collects event logs from OneLogin using OAuth2 authentication. Configure the OneLogin Identity source with the following parameters: ``` source: onelogin_identity: range: "P7D" acknowledgments: true subdomain: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `acknowledgments` (required) Prevents data loss by only considering logs to be processed successfully after they are received by the sink. Set to `true` to enable. `range` (optional) For pulling historical logs. Uses ISO 8601 duration format (for example, `P7D` for the last 7 days, `PT21H` for the last 21 hours). The default is 0 hours, and the maximum is 90 days. `subdomain` (required) Your OneLogin account subdomain. Must be alphanumeric with hyphens only, between 1–35 characters. `authentication.oauth2.client_id` (required) OAuth2 client ID for OneLogin Events API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for OneLogin Events API authentication. **Note** The `client_id` and `client_secret` values are retrieved from AWS Secrets Manager. The above parameter information can be obtained from the API credentials generated while setting up your OneLogin application. # Entrust IDaaS integration configuration Entrust Identity as a Service (IDaaS) is a cloud-based Identity and Access Management (IAM) platform that provides multi-factor authentication (MFA), single sign-on (SSO), adaptive risk-based authentication, and comprehensive audit logging across workforce, consumer, and citizen use cases. **Topics** + [ # Source configuration for Entrust IDaaS ](entrust-idaas-source-setup.md) + [ # CloudWatch pipelines configuration for Entrust IDaaS ](entrust-idaas-pipeline-setup.md) # Source configuration for Entrust IDaaS ## Integrating with Entrust IDaaS Entrust Identity as a Service (IDaaS) is a cloud-based Identity and Access Management (IAM) platform that provides multi-factor authentication (MFA), single sign-on (SSO), adaptive risk-based authentication, and comprehensive audit logging across workforce, consumer, and citizen use cases. CloudWatch pipeline uses the Entrust IDaaS Administration REST API to retrieve identity and access events from your IDaaS tenant. The Administration REST API provides access to two primary log categories: Authentication Logs (capturing user authentication events across multiple event types including MFA, SSO, SAML, OIDC, and passwordless authentication methods) and Management Logs (tracking administrative actions and changes performed across various entity types such as users, groups, applications, tokens, and policies). ## Authenticating with Entrust IDaaS To read the logs, the pipeline needs to authenticate with your Entrust IDaaS tenant. The plugin supports Administration API authentication using an `applicationId` and `sharedSecret`. **Create an Administration API application** + Go to your IDaaS Admin portal and navigate to Security → Applications. + Choose \$1 and select Administration API from the list of available applications. + In the General tab, enter a name and description for your application, then choose Next. + In the Setup tab, assign the role with the permissions required by your application, then choose Submit. The Entrust IDaaS Administration API requires the Super Administrator role to access audit log endpoints. + In the Complete tab, choose Copy to copy your `applicationId` and `sharedSecret`, or download the JSON file. + In AWS Secrets Manager, create a secret and store the `applicationId` under the key `client_id` and the `sharedSecret` under the key `client_secret`. + Your IDaaS API base URL is `https://` where `hostname` is taken from the credentials (for example, `https://entrust.us.trustedauth.com`). ## Configuring the CloudWatch Pipeline To configure the pipeline to read audit logs from Entrust IDaaS, choose `entrust_idaas` as the data source. Fill in the required information such as your tenant `hostname` and the AWS Secrets Manager secret ARN for your credentials where `client_id` and `client_secret` are stored. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and events that map to Authentication (3002) and Entity Management (3004). **Authentication** contains the following events: + AuthenticationAdminApiSuccessEvent + AuthenticationDeniedEvent + AuthenticationExternalSecondFactorBypassEvent + AuthenticationExternalSuccessEvent + AuthenticationFaceSuccessEvent + AuthenticationFidoSuccessEvent + AuthenticationFirstFactorExternalSuccessEvent + AuthenticationFirstFactorIdpSuccessEvent + AuthenticationFirstFactorPasswordSuccessEvent + AuthenticationGridSuccessEvent + AuthenticationGridWithTempAccessCodeSuccessEvent + AuthenticationIdpSuccessEvent + AuthenticationKbaSuccessEvent + AuthenticationLockedEvent + AuthenticationMagicLinkSuccessEvent + AuthenticationOtpCreatedEvent + AuthenticationOtpEmailSentEvent + AuthenticationOtpNoCreditEvent + AuthenticationOtpSentToAllEvent + AuthenticationOtpSmsSentEvent + AuthenticationOtpSuccessEvent + AuthenticationOtpUnavailableEvent + AuthenticationOtpVoiceSentEvent + AuthenticationOtpWithTempAccessCodeSuccessEvent + AuthenticationPasskeySuccessEvent + AuthenticationPasswordSuccessEvent + AuthenticationSecondFactorFaceSuccessEvent + AuthenticationSecondFactorFIDOSuccessEvent + AuthenticationSecondFactorGridSuccessEvent + AuthenticationSecondFactorGridWithTempAccessCodeSuccessEvent + AuthenticationSecondFactorKbaSuccessEvent + AuthenticationSecondFactorMagicLinkSuccessEvent + AuthenticationSecondFactorOtpSuccessEvent + AuthenticationSecondFactorOtpWithTempAccessCodeSuccessEvent + AuthenticationSecondFactorSmartCredentialPushSuccessEvent + AuthenticationSecondFactorTempAccessCodeSuccessEvent + AuthenticationSecondFactorTokenSuccessEvent + AuthenticationSecondFactorTokenWithTempAccessCodeSuccessEvent + AuthenticationSecondFactorUserCertificateSuccessEvent + AuthenticationSmartCredentialPushSuccessEvent + AuthenticationSmartLoginSuccessEvent + AuthenticationTempAccessCodeSuccessEvent + AuthenticationTokenPushSuccessEvent + AuthenticationTokenSuccessEvent + AuthenticationTokenWithTempAccessCodeSuccessEvent + AuthenticationUserCertificateSuccessEvent + MachineLockedEvent + OidcAuthenticationFailedEvent + OidcAuthenticationSuccessEvent + SamlAuthenticationFailedEvent + SamlAuthenticationSuccessEvent + UserPasswordChangeFailedEvent + UserPasswordChangeLockedEvent + UserStepUpAuthenticationSuccessEvent + VerificationDeniedEvent + VerificationIdpSuccessEvent **Entity Management** contains the following events: + ACTIVESYNC + AD\$1CONNECTOR\$1DIRECTORIES + AGENTS + APPLICATIONS + ARCHIVES + AUTHENTICATIONFLOWS + AUTHORIZATIONGROUPS + AZURE\$1DIRECTORIES + BLACKLISTEDPASSWORDS + BULKENROLLMENTS + BULKGROUPS + BULKHARDWARETOKENS + BULKIDENTITYGUARD + BULKSMARTCARDS + BULKUSERS + CAS + CERTIFICATES + CLAIMS + CONTACTVERIFICATION + CONTEXTRULES + CREATETENANT + CREDENTIALDESIGNS + CUSTOMIZATIONVARIABLES + DIGITALIDCERTIFICATES + DIGITALIDCONFIGCERTTEMPS + DIGITALIDCONFIGS + DIGITALIDCONFIGSANS + DIGITALIDCONFIGVARIABLES + DIRECTORIES + DIRECTORYATTRIBUTES + DIRECTORYCONNECTIONS + DIRECTORYPASSWORD + DIRECTORYSEARCHATTRIBUTES + DIRECTORYSYNC + DOMAINCONTROLLERCERTS + EMAILTEMPLATES + EMAILVARIABLES + ENROLLMENTDESIGNS + ENROLLMENTS + ENTITLEMENTS + EXPECTEDLOCATIONS + EXPORTREPORTS + FACE + FIDOTOKENS + GATEWAYCSRS + GATEWAYS + GRIDCONTENTS + GRIDS + GROUPPOLICIES + GROUPS + HIGH\$1AVAILABILITY\$1GROUPS + HOSTNAMESETTINGS + IDENTITYPROVIDERS + IDPROOFING + IDPROOFINGLICENSE + INTELLITRUSTDESKTOPS + IPLISTS + ISSUANCE + MAGICLINKCONTENTS + MAGICLINKS + OAUTHROLES + ORGANIZATIONS + OTPPROVIDERS + OTPS + PIVCONTENTSIGNER + PKIAASCREDENTIALS + POLICYOVERRIDE + PREFERREDOTPPROVIDERS + PRINTERS + PUSHCREDENTIALS + QUESTIONS + RATELIMITING + REPORTS + RESOURCESERVERAPIS + RESOURCESERVERSCOPES + RISKENGINES + ROLES + SCDEFNPIVAPPLETCONFIGS + SCDEFNS + SCDEFNVARIABLES + SCHEDULEDTASKS + SCIMPROVISIONINGS + SENDAZUREAD + SENDEMAIL + SENDSCIM + SERVICEPROVIDERACCOUNTS + SERVICEPROVIDERS + SETTINGS + SMARTCARDS + SMARTCREDENTIALS + SMARTCREDENTIALSSIGNATURE + SPCLIENTCREDENTIALS + SPENTITLEMENTS + SPIDENTITYPROVIDERS + SPMANAGEMENTPLATFORM + SPROLES + SPUSERMGMT + SUBSCRIBERS + TEMPACCESSCODECONTENTS + TEMPACCESSCODES + TEMPLATES + TENANTS + TOKENACTIVATIONCONTENTS + TOKENS + TRANSACTIONITEMS + TRANSACTIONRULES + USERATTRIBUTES + USERATTRIBUTEVALUES + USERKBACHALLENGES + USERLOCATIONS + USERMACHINES + USEROAUTHTOKENS + USERPASSWORDS + USERQUESTIONANSWERS + USERQUESTIONS + USERRBASETTINGS + USERS + USERSITEROLES + USERSPROLES + WORDSYNONYMS # CloudWatch pipelines configuration for Entrust IDaaS Collects identity and access management audit logs from Entrust IDaaS using OAuth2 authentication. Configure the Entrust IDaaS source with the following parameters: ``` source: entrust_idaas: hostname: "" authentication: oauth2: client_id: "${{aws_secrets::client_id}}" client_secret: "${{aws_secrets::client_secret}}" ```Parameters `hostname` (required) Entrust IDaaS tenant hostname (for example, `entrust.us.trustedauth.com`). Do not include the `https://` prefix. `authentication.oauth2.client_id` (required) OAuth2 client ID for Entrust IDaaS Administration API authentication. `authentication.oauth2.client_secret` (required) OAuth2 client secret for Entrust IDaaS Administration API authentication. **Note** The parameter information should correspond to values received in the Authenticating with Entrust IDaaS section. # Drupal Core integration configuration Drupal Core is the foundational open-source web application framework built on PHP that provides the base platform for building websites, applications, and digital experiences. **Topics** + [ # Source configuration for Drupal Core ](drupal-core-source-setup.md) + [ # CloudWatch pipelines configuration for Drupal Core ](drupal-core-pipeline-setup.md) # Source configuration for Drupal Core ## Integrating with Drupal Core Drupal Core is the foundational open-source web application framework built on PHP that provides the base platform for building websites, applications, and digital experiences. CloudWatch Pipeline uses the custom View-based REST API to retrieve audit log data — including content changes, user authentication events, and administrative actions — from your Drupal Core site. The API enables access to time-filtered log data through REST endpoints, allowing retrieval of activity records scoped to a configurable time window. ## Authenticating with Drupal Core To read the logs, the pipeline needs to authenticate with your Drupal Core site. The plugin supports Basic Authentication (HTTP Basic Auth using a username and password). **Configure Basic Authentication for Drupal Core** + Log in to your Drupal Core admin interface and navigate to Administration → Extend (`/admin/modules`). + Enable the following modules: RESTful Web Services, Serialization, HTTP Basic Authentication, and Views. Choose Install. + Install and enable the Admin Audit Trail module via Composer (`composer require drupal/admin_audit_trail`) and run `drush en admin_audit_trail -y && drush cr` to activate it. + Navigate to Structure → Views and create a new View named `Audit Logs API`. Set Show to `Log entries`, enable Provide a REST export, and set the REST export path to `/api/v1/audit-logs`. + In the View editor, add two exposed Watchdog: Timestamp filters — one with operator `is greater than or equal to` and filter identifier `starttime`, and another with operator `is less than` and filter identifier `endtime`. + In the REST EXPORT SETTINGS section of the View, choose Authentication and enable `basic_auth`. + Navigate to People → Permissions and grant the roles that need API access the Access admin audit trail and Administer REST resource configuration permissions. Save the View. + In AWS Secrets Manager, create a secret and store the Drupal Core username under the key `username` and the account password under the key `password`. ## Configuring the CloudWatch Pipeline To configure the pipeline to read logs, choose Drupal Core as the data source. Fill in the required information: + **Domain** — The base URL of your Drupal Core site (for example, `https://your-drupal-site.example.com`). + **API Endpoint** — The path to the View REST export endpoint (for example, `/api/v1/audit-logs`). Must start with `/`. + **Range** — Specify the lookback duration in ISO 8601 format (for example, `PT21H` for the last 21 hours, `P7D` for the last 7 days). The default is 0 hours, and the maximum is 90 days. Once you create the pipeline, data will be available in the selected CloudWatch Logs log group. ## Supported Open Cybersecurity Schema Framework Event Classes This integration supports OCSF schema version v1.5.0 and transforms events that map to Authentication (3002), Entity Management (3004), HTTP Activity (4002), and Application Lifecycle (6002). Events that are not listed are not mapped to OCSF and will be forwarded to the sink as raw logs. **Authentication** contains the following event types: + user — Login and authentication related events **Entity Management** contains the following event types: + user — User creation and deletion + content + comment **HTTP Activity** contains the following event types: + access denied + page not found + php + new custom types **Application Lifecycle** contains the following event types: + system + cron # CloudWatch pipelines configuration for Drupal Core Collects audit logs from Drupal Core using basic authentication. Configure the Drupal source with the following parameters: ``` source: drupal_core: range: "P7D" domain: "" api_endpoint: "" authentication: basic: username: "${{aws_secrets::username}}" password: "${{aws_secrets::password}}" ```Parameters `range` (optional) For pulling historical logs. Uses ISO 8601 duration format (for example, `P7D` for the last 7 days, `PT21H` for the last 21 hours). The default is 0 hours, and the maximum is 90 days. `domain` (required) The domain name of the Drupal Core site. `api_endpoint` (required) The path of the Drupal Core audit logs API endpoint. Must start with `/` (for example, `/api/v1/audit-logs`). `authentication.basic.username` (required) The username for Basic Authentication. `authentication.basic.password` (required) The password for Basic Authentication. # Custom log data from CloudWatch Logs or an Amazon S3 bucket You can create pipelines for custom data sources using the following approaches: 1. **CloudWatch Custom Logs** – Define pipelines on your existing CloudWatch custom log groups by providing: + A data source name + A data source type For more information on data source name and type, see the [CloudWatch Logs User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html#how-to-get-started-data-sources). 1. **S3 Custom Sources** – Process arbitrary logs stored in S3 buckets by configuring source for the pipeline: ``` source: s3: aws: region: "us-east-1" sts_role_arn: "arn:aws:iam:::role/" compression: "gzip" codec: ndjson: data_source_name: "my_custom_logs" default_bucket_owner: "123456789012" bucket_owners: my-bucket: "123456789012" disable_bucket_ownership_validation: false notification_type: "sqs" sqs: queue_url: "https://sqs.region.amazonaws.com//" on_error: "retain_messages" ``` **Note** The CSV processor can't be used with S3 custom sources. Instead, use the CSV codec in the S3 source. **Parameters** `notification_type` (required) Specifies the notification mechanism. Must be "sqs" to use SQS for S3 event notifications. `data_source_name` Identifies the data source. This can be any string value that represents your data source. Example: "my\$1custom\$1logs". `aws.region` (required) The AWS region where the S3 bucket and SQS queue are located. `aws.sts_role_arn` (required) The ARN of the IAM role to assume for accessing S3 and SQS resources. `codec` (required) Codec configuration for parsing S3 objects. Supports `csv`, `json`, `ndjson` codecs. `compression` (optional) Compression type of the S3 objects. Valid values are "none", "gzip", "automatic". Defaults to "none". `sqs.queue_url` (required for SQS) The complete SQS queue URL that receives S3 bucket notifications when new objects are created. `on_error` (optional) Determines how to handle errors in Amazon SQS. Can be either `retain_messages` or `delete_messages`. Default is `retain_messages`. **Custom source configuration** When creating a pipeline for custom sources: + A parser must be the first processor in the pipeline if the data source is CloudWatch Logs + You can specify any supported processor for custom log pipelines # Configuring Custom S3 Bucket Sources With CloudWatch pipelines, you can process arbitrary logs stored in S3 buckets. ## Prerequisites To use Amazon S3 as the source for a pipeline, first create an S3 bucket. For instructions, see [Creating a general purpose bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)in the *Amazon S3 User Guide*. **Note** If the S3 bucket used as a source in the pipeline is in a different AWS account, you also need to enable cross-account read permissions on the bucket. This allows the pipeline to read and process the data. To enable cross-account permissions, see [Bucket owner granting cross-account bucket permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the *Amazon S3 User Guide*. If your S3 buckets are in multiple accounts, use a `bucket_owners` map. For an example, see [Cross-account S3 access](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sources/s3/#cross-account-s3-access) in the *OpenSearch* documentation. To set up S3-SQS processing, you also need to perform the following steps: + [Create an Amazon SQS queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/creating-sqs-standard-queues.html). + [ Enable event notifications on the S3 bucket with the SQS queue as a destination](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html). ## Configure the pipeline role Unlike other source plugins that push data to a pipeline, the S3 source plugin has a read-based architecture in which the pipeline pulls data from the source. Therefore, in order for a pipeline to read from S3, you must specify a role within the pipeline's S3 source configuration that has access to both the S3 bucket and the Amazon SQS queue. The pipeline will assume this role in order to read data from the queue. You can find example role permissions in [Source-specific IAM policies](pipeline-iam-reference.md#source-specific-iam-policies). Note that this role must have a trust relationship with the CloudWatch pipelines service principle. You can find an example trust policy configuration for your pipeline role in [Trust relationships](pipeline-iam-reference.md#trust-relationships). ## Create the pipeline After you've set up your permissions, you can configure a pipeline depending on your Amazon S3 use case. Select **Create pipeline** in the **Pipelines** tab under **Ingestion** in the CloudWatch Console. Follow the pipeline wizard steps and provide the SQS queue ARN and required pipeline IAM role when prompted. Optionally provide a data source name and type to attach to the log group destination in CloudWatch Logs. Be sure to configure a [CloudWatch Logs resource policy](https://docs.aws.amazon.com/resource-policies.html) if one isn't already configured to the destination log group and then select **Create pipeline** in the **Review and create** step. The pipeline will be created and data will begin to flow within 5 minutes if successful. ## Amazon S3 cross account as a source You can grant access across accounts with Amazon S3 so that CloudWatch pipelines can access S3 buckets in another account as a source. To enable cross-account access, see [Bucket owner granting cross-account bucket permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the *Amazon S3 User Guide*. After you have granted access, ensure that your pipeline role has the required permissions. Then, you can create a pipeline using `bucket_owners` to enable cross-account access to an Amazon S3 bucket as a source. **Custom source configuration** When creating a pipeline for custom sources: + A parser must be the first processor in the pipeline + You can specify any supported processor for custom log pipelines # AWS service logs from CloudWatch Logs Intercepts log events from CloudWatch Logs that match the log event metadata configuration. **Important** Pipelines with processors mutate the log events in the original CloudWatch log group they are intercepted from for logs from AWS services. To preserve unmodified copies of your log data, enable the `include_original` option in the `cloudwatch_logs` sink configuration, or use the **Keep original log** toggle in the console. For more information, see [Sinks](pipeline-sinks.md). **Configuration** Configure the CloudWatch Logs source with the following parameters: ``` source: cloudwatch_logs: aws: sts_role_arn: "arn:aws:iam::123456789012:role/MyCloudWatchLogsRole" log_event_metadata: data_source_name: "" data_source_type: "" ```Parameters `aws.sts_role_arn` (required) The ARN of the IAM role to assume for CloudWatch Logs interception. `log_event_metadata.data_source_name` (required) Identifies the specific AWS service that generated the log events or a custom log source name. For custom logs, this can be any string up to 15 characters when `data_source_type` is "default". `log_event_metadata.data_source_type` (required) Specifies the category or type of logs within the AWS service, or "default" for custom logs. Set to "default" to enable custom log source names. For more information on data source name and type, see the [CloudWatch Logs User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html#how-to-get-started-data-sources). **Restrictions** The following restrictions apply to CloudWatch Logs sources: + No two pipelines can use the `cloudwatch_logs` source with identical `data_source_name` and `data_source_type` metadata criteria. + When `data_source_name` is `aws_cloudtrail`, only empty processors (`[]`) or the `ocsf` processor are allowed. # CloudWatch pipelines processors CloudWatch pipelines processors transform, parse, and enrich log data as it flows through the pipeline. A pipeline can have up to 20 processors that are applied sequentially in the order they are defined. **Transformation metadata** When a pipeline processes log events, CloudWatch pipelines automatically adds transformation metadata to each processed log entry. This metadata indicates that the log has been transformed, making it easy to distinguish between original and processed data. If you enable the **Keep original log** option during pipeline creation, you can compare the original log with the transformed version at any time. **Processor categories** | Category | Description | | --- | --- | | Parsers | Convert raw log data into structured formats, such as Open Cybersecurity Schema Framework (OCSF), CSV, JSON, and so on | | Transformers | Modify log data structure; add, copy, move, or delete fields | | String Processors | Manipulate string values; case conversion, trimming, substitution | **Topics** + [ # Parser processors ](parser-processors.md) + [ # Transformation processors ](transformation-processors.md) + [ # String manipulation processors ](string-processors.md) + [ # Filter processors ](filter-processors.md) + [ # Common processor use cases ](processor-examples.md) + [ # Processor compatibility and restrictions ](processor-compatibility.md) # Parser processors Parser processors convert raw or semi-structured log data into structured formats. Each pipeline can have at most one parser processor, which must be the first processor in the pipeline. **Conditional processing not supported** Parser processors (except Grok) do not support conditional processing with the `when` parameter. This includes OCSF, CSV, JSON, KeyValue, VPC, Route53, WAF, Postgres, and CloudFront parsers. For more information, see [Expression syntax for conditional processing](conditional-processing.md). ## OCSF processor Parses and transforms log data according to Open Cybersecurity Schema Framework (OCSF) standards. **Configuration** Configure the OCSF processor with the following parameters: ``` processor: - ocsf: version: "1.5" mapping_version: 1.5.0 schema: microsoft_office365_management_activity: ```Parameters `version` (required) The OCSF schema version to use for transformation. Must be 1.5 `mapping_version` (required) The OCSF mapping version for transformation. Must be 1.5.0. `schema` (required) Schema object specifying the data source type. The supported schemas depend on the pipeline source type - each source type has its own set of compatible OCSF schemas. You must use a schema that matches your pipeline's source type. This table lists the supported schema combinations. | Pipeline Source Type | Supported Schemas | Version | Mapping Version | | --- | --- | --- | --- | | cloudwatch\$1logs | cloud\$1trail: | 1.5 | Not required | | cloudwatch\$1logs | route53\$1resolver: | 1.5 | Not required | | cloudwatch\$1logs | vpc\$1flow: | 1.5 | Not required | | cloudwatch\$1logs | eks\$1audit: | 1.5 | Not required | | cloudwatch\$1logs | aws\$1waf: | 1.5 | Not required | | s3 | Any OCSF schema | Any | Any | | microsoft\$1office365 | microsoft\$1office365: | 1.5 | 1.5.0 | | microsoft\$1entraid | microsoft\$1entraid: | 1.5 | 1.5.0 | | microsoft\$1windows\$1event | microsoft\$1windows\$1event: | 1.5 | 1.5.0 | | paloaltonetworks\$1nextgenerationfirewall | paloaltonetworks\$1nextgenerationfirewall: | 1.5 | 1.5.0 | | okta\$1auth0 | okta\$1auth0: | 1.5 | 1.5.0 | | okta\$1sso | okta\$1sso: | 1.5 | 1.5.0 | | crowdstrike\$1falcon | crowdstrike\$1falcon: | 1.5 | 1.5.0 | | github\$1auditlogs | github\$1auditlogs: | 1.5 | 1.5.0 | | sentinelone\$1endpointsecurity | sentinelone\$1endpointsecurity: | 1.5 | 1.5.0 | | servicenow\$1cmdb | servicenow\$1cmdb: | 1.5 | 1.5.0 | | wiz\$1cnapp | wiz\$1cnapp: | 1.5 | 1.5.0 | | zscaler\$1internetaccess | zscaler\$1internetaccess: | 1.5 | 1.5.0 | ## CSV processor Parses CSV formatted data into structured fields. **Configuration** Configure the CSV processor with the following parameters: ``` processor: - csv: column_names: ["col1", "col2", "col3"] delimiter: "," quote_character: '"' ```Parameters `column_names` (optional) Array of column names for parsed fields. Maximum 100 columns, each name up to 128 characters. If not provided, defaults to column\$11, column\$12, and so on. `delimiter` (optional) Character used to separate CSV fields. Must be a single character. Defaults to comma (,). `quote_character` (optional) Character used to quote CSV fields containing delimiters. Must be a single character. Defaults to double quote ("). To use the processor without specifying additional parameters, use the following command: ``` processor: - csv: {} ``` ## Grok processor Parses unstructured data using Grok patterns. At most 1 Grok is supported per pipeline. For details on the Grok transformer in CloudWatch Logs see [Processors that you can use](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Transformation-Processors.html) in the *CloudWatch Logs User Guide*. **Configuration** Configure the Grok processor with the following parameters: When the data source is a dictionary, you can use this configuration: ``` processor: - grok: match: source_key: ["%{WORD:level} %{GREEDYDATA:msg}"] ``` When the data source is CloudWatch Logs, you can use this configuration: ``` processor: - grok: match: source_key: ["%{WORD:level} %{GREEDYDATA:msg}"] ```Parameters `match` (required) Field mapping with Grok patterns. Only one field mapping allowed. `match.` (required) Array with single Grok pattern. Maximum 512 characters per pattern. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). **Important** If the Grok processor is used as the parser (first processor) in a pipeline and its `when` condition evaluates to false, the entire pipeline does not execute for that log event. Parsers must run for downstream processors to receive structured data. ## VPC processor Parses VPC Flow Log data into structured fields. **Configuration** Configure the VPC processor with the following parameters: ``` processor: - parse_vpc: {} ``` ## JSON processor Parses JSON data into structured fields. **Configuration** Configure the JSON processor with the following parameters: ``` processor: - parse_json: source: "message" destination: "parsed_json" ```Parameters `source` (optional) The field containing the JSON data to parse. If omitted, the entire log message is processed `destination` (optional) The field where the parsed JSON will be stored. If omitted, parsed fields are added to the root level ## Route 53 processor Parses Route 53 resolver log data into structured fields. **Configuration** Configure the Route 53 processor with the following parameters: ``` processor: - parse_route53: {} ``` ## Key-value processor Parses key-value pair formatted data into structured fields. **Configuration** Configure the key-value processor with the following parameters: ``` processor: - key_value: source: "message" destination: "parsed_kv" field_delimiter: "&" key_value_delimiter: "=" ```Parameters `source` (optional) Field containing key-value data. Maximum 128 characters. `destination` (optional) Target field for parsed key-value pairs. Maximum 128 characters. `field_delimiter` (optional) Pattern to split key-value pairs. Maximum 10 characters. `key_value_delimiter` (optional) Pattern to split keys from values. Maximum 10 characters. `overwrite_if_destination_exists` (optional) Whether to overwrite existing destination field. `prefix` (optional) Prefix to add to extracted keys. Maximum 128 characters. `non_match_value` (optional) Value for keys without matches. Maximum 128 characters. To use the processor without specifying additional parameters, use the following command: ``` processor: - key_value: {} ``` # Transformation processors Transformation processors modify the structure of log events by adding, copying, moving, or removing fields. ## add\$1entries processor Adds static key-value pairs to log events. At most 1 `add_entries` processor can be added to a pipeline. **Configuration** Configure the add\$1entries processor with the following parameters: ``` processor: - add_entries: entries: - key: "environment" value: "production" overwrite_if_key_exists: false ```Parameters `entries` (required) Array of key-value pairs to add to each log event. `entries[].key` (required) The field name to add to the log event. Supports nested fields using dot notation. `entries[].value` (required) The static value to assign to the key. `entries[].overwrite_if_key_exists` (optional) Boolean flag that determines behavior when the key already exists. Defaults to false. `when` (optional) Processor-level conditional expression. When specified, the entire processor is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when` (optional) Entry-level conditional expression. When specified, only this entry is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when_else` (optional) Fallback entry that executes only when none of the other `when` conditions in the same processor matched. The expression value identifies which `when` conditions to consider. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## copy\$1values processor Copies values from one field to another. At most 1 `copy_values` processor can be added to a pipeline. **Configuration** Configure the copy\$1values processor with the following parameters: ``` processor: - copy_values: entries: - from_key: "user_id" to_key: "backup_user" overwrite_if_to_key_exists: false ```Parameters `entries` (required) Array of copy operations to perform on each log event. `entries[].from_key` (required) The field name to copy the value from. Uses dot notation for nested fields. `entries[].to_key` (required) The field name to copy the value to. Will create nested structures if using dot notation. `entries[].overwrite_if_to_key_exists` (optional) Boolean flag controlling behavior when target field already exists. Defaults to false. `when` (optional) Processor-level conditional expression. When specified, the entire processor is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when` (optional) Entry-level conditional expression. When specified, only this entry is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when_else` (optional) Fallback entry that executes only when none of the other `when` conditions in the same processor matched. The expression value identifies which `when` conditions to consider. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## delete\$1entries processor Removes specified fields from log events. **Configuration** Configure the delete\$1entries processor with the following parameters: ``` processor: - delete_entries: with_keys: ["temp_field", "debug_info"] ```Parameters `with_keys` (required) Array of field names to remove from each log event. Supports nested field deletion using dot notation. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## move\$1keys processor Moves fields from one location to another. **Configuration** Configure the move\$1keys processor with the following parameters: ``` processor: - move_keys: entries: - from_key: "old_field" to_key: "new_field" overwrite_if_to_key_exists: true ```Parameters `entries` (required) Array of move operations. Maximum 5 entries. `entries[].from_key` (required) Source field name. Maximum 128 characters. `entries[].to_key` (required) Target field name. Maximum 128 characters. `entries[].overwrite_if_to_key_exists` (optional) Whether to overwrite existing target field. `when` (optional) Processor-level conditional expression. When specified, the entire processor is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when` (optional) Entry-level conditional expression. When specified, only this entry is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when_else` (optional) Fallback entry that executes only when none of the other `when` conditions in the same processor matched. The expression value identifies which `when` conditions to consider. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## flatten processor Flattens nested object structures. **Configuration** Configure the flatten processor with the following parameters: ``` processor: - flatten: source: "metadata" target: "flattened" remove_processed_fields: true exclude_keys: ["sensitive_data"] ```Parameters `source` (required) Field containing nested object to flatten. `target` (required) Target field prefix for flattened keys. `remove_processed_fields` (optional) Whether to remove the original nested field after flattening. `exclude_keys` (optional) Array of keys to exclude from flattening. Maximum 20 keys, each up to 128 characters. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). # String manipulation processors String processors modify text values within log events through operations like case conversion, trimming, and pattern matching. ## lowercase\$1string processor Converts specified fields to lowercase. **Configuration** Configure the lowercase\$1string processor with the following parameters: ``` processor: - lowercase_string: with_keys: ["status", "method"] ```Parameters `with_keys` (required) Array of field names to convert to lowercase. Only processes string values. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## uppercase\$1string processor Converts specified fields to uppercase. **Configuration** Configure the uppercase\$1string processor with the following parameters: ``` processor: - uppercase_string: with_keys: ["status_code", "method"] ```Parameters `with_keys` (required) Array of field names to convert to uppercase. Only processes string values. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## trim\$1string processor Removes leading and trailing whitespace from specified fields. **Configuration** Configure the trim\$1string processor with the following parameters: ``` processor: - trim_string: with_keys: ["message", "user_input"] ```Parameters `with_keys` (required) Array of field names to trim whitespace from. Only processes string values. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## substitute\$1string processor Performs string substitution using regular expressions. **Configuration** Configure the substitute\$1string processor with the following parameters: ``` processor: - substitute_string: entries: - source: "message" from: "ERROR" to: "WARN" ```Parameters `entries` (required) Array of substitution operations to perform on each log event. `entries[].source` (required) The field to perform string substitution on. `entries[].from` (required) The regular expression pattern to match and replace. `entries[].to` (required) The replacement string for matched patterns. `when` (optional) Processor-level conditional expression. When specified, the entire processor is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when` (optional) Entry-level conditional expression. When specified, only this entry is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when_else` (optional) Fallback entry that executes only when none of the other `when` conditions in the same processor matched. The expression value identifies which `when` conditions to consider. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## truncate processor Truncates field values to specified length. **Configuration** Configure the truncate processor with the following parameters: ``` processor: - truncate: source_keys: ["message", "description"] length: 100 start_at: 0 ```Parameters `source_keys` (required) Array of field names to truncate. Each field name maximum 128 characters. `length` (optional) Maximum length after truncation. Range: 1-8192. `start_at` (optional) Starting position for truncation. Range: 0-8192. Defaults to 0. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## extract\$1value processor Extracts values using regular expressions. **Configuration** Configure the extract\$1value processor with the following parameters: ``` processor: - extract_value: entries: - source: "message" target: "extracted_data" from: "user=(?\\w+)" to: "${user}" target_type: "string" ```Parameters `entries` (required) Array of extraction operations. Maximum 20 entries. `entries[].source` (required) Field to extract from. Maximum 128 characters. `entries[].target` (required) Target field for extracted value. Maximum 128 characters. `entries[].from` (required) Regular expression pattern. Maximum 128 characters. `entries[].to` (required) Replacement pattern with capture groups. Maximum 128 characters. `entries[].target_type` (optional) Target data type ("integer", "double", "string", "boolean"). `when` (optional) Processor-level conditional expression. When specified, the entire processor is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when` (optional) Entry-level conditional expression. When specified, only this entry is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when_else` (optional) Fallback entry that executes only when none of the other `when` conditions in the same processor matched. The expression value identifies which `when` conditions to consider. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## convert\$1entry\$1type processor Converts field values between different data types. **Configuration** Configure the convert\$1entry\$1type processor with the following parameters: ``` processor: - convert_entry_type: key: "count" type: "integer" ```Parameters `key` (required) Single field name to convert. `type` (required) Target data type. Options: "integer", "double", "string", "boolean". `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## date processor Parses and formats date/time fields. **Configuration** Configure the date processor with the following parameters: ``` processor: - date: match: - key: "timestamp" patterns: ["yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"] destination: "@timestamp" source_timezone: "UTC" destination_timezone: "America/New_York" ```Parameters `match` (required) Array of date matching configurations. Maximum 10 entries. `match[].key` (required) Field containing the date string. Maximum 128 characters. `match[].patterns` (required) Array of date format patterns to try. Maximum 5 patterns, each up to 256 characters. `destination` (optional) Single target field for all parsed dates. Maximum 128 characters. `source_timezone` (optional) Source timezone for parsing. `destination_timezone` (optional) Target timezone for output. `output_format` (optional) Output date format. Maximum 64 characters. `destination_type` (optional) Output type - "timestampz", "long", or "string". `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## dissect processor Extracts structured data using pattern matching. **Configuration** Configure the dissect processor with the following parameters: ``` processor: - dissect: map: message: "%{timestamp} %{level}" ```Parameters `map` (required) Field mapping with dissect patterns. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## list\$1to\$1map processor Converts array fields to map structures. **Configuration** Configure the list\$1to\$1map processor with the following parameters: ``` processor: - list_to_map: source: "tags" key: "name" value_key: "value" target: "tag_map" ```Parameters `source` (required) Field containing array data. Maximum 128 characters. `key` (required) Field name to use as map key. Maximum 128 characters. `value_key` (optional) Field name to use as map value. Maximum 128 characters. `target` (optional) Target field for map structure. Maximum 128 characters. `flatten` (optional) Whether to flatten the resulting map. `flattened_element` (optional) Which element to use when flattening ("first" or "last"). `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## rename\$1keys processor Renames fields in log events. **Configuration** Configure the rename\$1keys processor with the following parameters: ``` processor: - rename_keys: entries: - from_key: "old_name" to_key: "new_name" overwrite_if_to_key_exists: true ```Parameters `entries` (required) Array of rename operations. Maximum 5 entries. `entries[].from_key` (required) Current field name. Maximum 128 characters. `entries[].to_key` (required) New field name. Maximum 128 characters. `entries[].overwrite_if_to_key_exists` (optional) Whether to overwrite existing target field. `when` (optional) Processor-level conditional expression. When specified, the entire processor is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when` (optional) Entry-level conditional expression. When specified, only this entry is skipped if the expression evaluates to false. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). `entries[].when_else` (optional) Fallback entry that executes only when none of the other `when` conditions in the same processor matched. The expression value identifies which `when` conditions to consider. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## select\$1entries processor Selects only specified fields from events. **Configuration** Configure the select\$1entries processor with the following parameters: ``` processor: - select_entries: include_keys: ["timestamp", "level", "message"] ```Parameters `include_keys` (required) Array of field names to keep. Maximum 50 keys, each up to 128 characters. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). ## translate processor Translates field values using lookup tables. **Configuration** Configure the translate processor with the following parameters: ``` processor: - translate: mappings: - source: "status_code" targets: - target: "status_text" map: "200": "OK" "404": "Not Found" ```Parameters `mappings` (required) Array of translation configurations. Maximum 10 mappings. `mappings[].source` (required) Field to translate. Maximum 128 characters. `mappings[].targets` (required) Array of target configurations. Maximum 10 targets. `mappings[].targets[].target` (required) Target field name. Maximum 128 characters. `mappings[].targets[].map` (required) Translation mapping. Maximum 100 entries, each value up to 512 characters. `when` (optional) Conditional expression that determines whether this processor executes. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md). # Filter processors Filter processors let you selectively remove log entries from the pipeline based on conditions you define. ## drop\$1events processor Filters out unwanted log entries based on conditional expressions. Use this processor to reduce noise from third-party pipeline connectors and lower storage costs by removing log events that match specified conditions. **Configuration** Configure the drop\$1events processor with the following parameters: ``` processor: - drop_events: when: "log.level == 'DEBUG' or log.level == 'TRACE'" ```Parameters `when` (required) Conditional expression that determines which log entries to drop. Log entries matching this expression are removed from the pipeline. Maximum length is 256 characters. See [Expression syntax for conditional processing](conditional-processing.md) for expression syntax. `handle_expression_failure` (optional) Behavior when the `when` expression evaluation fails. Allowed values: `"skip"` (default) keeps the event, or `"apply"` drops the event regardless of the failure. **Example Drop low-severity log entries** The following configuration drops all DEBUG and TRACE log entries, keeping only higher-severity events: ``` processor: - drop_events: when: "log.level in {'DEBUG', 'TRACE'}" handle_expression_failure: "skip" ``` # Common processor use cases Here are common scenarios and example configurations for combining processors: **Example Standardize log formats and add metadata** Parse JSON logs, standardize field names, and add environment information: ``` processor: - parse_json: {} - rename_keys: entries: - from_key: "timestamp" to_key: "@timestamp" - from_key: "log_level" to_key: "level" - add_entries: entries: - key: "environment" value: "production" - key: "application" value: "payment-service" ``` **Example Clean and normalize field values** Standardize status codes and remove sensitive data: ``` processor: - uppercase_string: with_keys: ["status", "method"] - delete_entries: with_keys: ["credit_card", "password"] - substitute_string: entries: - source: "status" from: "SUCCESS" to: "OK" ``` **Example Extract and transform specific fields** Extract user information and format for analysis: ``` processor: - extract_value: entries: - source: "user_agent" target: "browser" from: "(?Chrome|Firefox|Safari)" to: "${browser}" - lowercase_string: with_keys: ["browser"] - move_keys: entries: - from_key: "browser" to_key: "user_data.browser" ``` **Example Conditional processing with entry-level conditions** Add different metadata based on log severity using entry-level `when` conditions: ``` processor: - add_entries: entries: - key: "alert_level" value: "critical" when: "log.level == 'ERROR'" - key: "alert_level" value: "info" when_else: "log.level == 'ERROR'" ``` **Example Drop unwanted log entries** Filter out debug and trace log entries from a third-party source to reduce noise and storage costs: ``` processor: - drop_events: when: "log.level in {'DEBUG', 'TRACE'}" handle_expression_failure: "skip" ``` **Example Processor-level conditional with delete\$1entries** Remove sensitive fields only when the environment is production: ``` processor: - delete_entries: with_keys: ["password", "api_key", "ssn"] when: "environment in {'prod', 'staging'}" ``` # Processor compatibility and restrictions General processor rules Maximum count A pipeline can have at most 20 processors. Parser placement Parser processors (OCSF, CSV, Grok, etc.), if used, must be the first processor in a pipeline. Unique processors The following processors can appear only once per pipeline: + `add_entries` + `copy_values` | Processor Type | CloudWatch Logs Source | S3 Source | API-based Sources | | --- | --- | --- | --- | | OCSF | Must be first processor | Must be first processor | Must be first processor | | parse\$1vpc | Must be first processor | Not applicable | Not applicable | | parse\$1route53 | Must be first processor | Not applicable | Not applicable | | parse\$1json | Must be first processor | Must be first processor | Must be first processor | | grok | Must be first processor | Must be first processor | Must be first processor | | csv | Must be first processor | Not compatible | Not compatible | | key\$1value | Must be first processor | Must be first processor | Must be first processor | | add\$1entries | Must be first processor | Must be first processor | Must be first processor | | copy\$1values | Must be first processor | Must be first processor | Must be first processor | | String processors (lowercase, uppercase, trim) | Must be first processor | Must be first processor | Must be first processor | | Field processors (move\$1keys, rename\$1keys) | Must be first processor | Must be first processor | Must be first processor | | Data transformation (date, flatten) | Must be first processor | Must be first processor | Must be first processor | **Compatibility definitions** Must be first processor When used, must be the first processor in the pipeline configuration Not compatible Cannot be used with this source type Not applicable Processor is not relevant for this source type ## Processor-specific restrictions **Processor restrictions by source type** | Processor | Source Type | Restrictions | | --- | --- | --- | | OCSF | CloudWatch Logs with CloudTrail | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/processor-compatibility.html) | | OCSF | API-based Sources | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/processor-compatibility.html) | | parse\$1vpc | CloudWatch Logs | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/processor-compatibility.html) | | parse\$1route53 | CloudWatch Logs | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/processor-compatibility.html) | | add\$1entries | All Sources | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/processor-compatibility.html) | | copy\$1values | All Sources | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/processor-compatibility.html) | **Important** When using processors with restrictions: Always validate your pipeline configuration using the `ValidateTelemetryPipelineConfiguration` API before deployment Test the pipeline with sample data using the `TestTelemetryPipeline` API to ensure proper processing Monitor pipeline metrics after deployment to ensure events are being processed as expected # Expression syntax for conditional processing CloudWatch pipelines processors that support conditional processing accept a `when` parameter containing an expression. When the expression evaluates to true, the processor or entry executes. Expressions use dot notation (`.`) for nested field access. For example, `user.role` accesses the `role` field inside the `user` object. For more details on processors that support conditional processing and their specific parameters, see [CloudWatch pipelines processors](pipeline-processors.md). For configuration examples, see [Common processor use cases](processor-examples.md). ## Processor-level and entry-level conditions There are two levels at which you can apply a `when` condition, depending on the processor. Processor-level `when` (outer level) A `when` placed at the top level of the processor configuration. If the expression evaluates to false, the entire processor is skipped and no operations within it execute. All processors that support conditional processing support this level. **Example Processor-level condition — skip entire processor** The following `delete_entries` processor only runs when the environment is production or staging. If the condition is false, none of the keys are deleted. ``` processor: - delete_entries: with_keys: ["password", "api_key", "ssn"] when: "environment in {'prod', 'staging'}" ``` Entry-level `when` (within each entry) A `when` placed inside an individual entry in the `entries` array. Each entry is evaluated independently — if the expression is false, only that specific entry is skipped while other entries in the same processor still execute. Only processors with an `entries` array support this level (such as `add_entries`, `copy_values`, `rename_keys`, `move_keys`, `extract_value`, and `substitute_string`). **Example Entry-level condition — skip individual entries** The following `add_entries` processor adds different keys depending on each entry's condition. The first entry only adds `severity` when the log level is ERROR. The second entry always adds `processed` because it has no condition. ``` processor: - add_entries: entries: - key: "severity" value: "high" when: "log.level == 'ERROR'" - key: "processed" value: "true" ``` Processors that support both levels can use them together. When both are specified, the processor-level condition is evaluated first. If it is false, the entire processor is skipped and no entry-level conditions are evaluated. **Example Both levels combined** The processor-level `when` ensures the entire processor only runs for production traffic. Within that, each entry has its own condition to control which key is added. ``` processor: - add_entries: when: "environment == 'prod'" entries: - key: "alert_level" value: "critical" when: "log.level == 'ERROR'" - key: "alert_level" value: "warning" when: "log.level == 'WARN'" ``` For a table showing which processors support which level, see the [Conditional processing support](#conditional-support) section below. **Fallback with `when_else`** Processors that support entry-level conditions also support `when_else`. An entry with `when_else` acts as a fallback — it executes only when none of the other `when` conditions in the same processor matched. The expression value provided to `when_else` identifies which set of `when` conditions to consider, but the entry itself runs based solely on whether those conditions all evaluated to false. There is no explicit negation check — the entry simply runs when no other `when` matched. **Example Fallback entry with when\$1else** The first entry runs when the log level is ERROR. The second entry uses `when_else` and runs only when the first entry's `when` condition did not match (i.e., the log level is anything other than ERROR). ``` processor: - add_entries: entries: - key: "alert_level" value: "critical" when: "log.level == 'ERROR'" - key: "alert_level" value: "info" when_else: "log.level == 'ERROR'" ``` ## Conditional processing support The following table shows which processors support conditional processing and at what level. **Processor conditional support** | Processor | Conditional support | Level | | --- | --- | --- | | add\$1entries | Yes | Processor and entry | | copy\$1values | Yes | Processor and entry | | delete\$1entries | Yes | Processor | | move\$1keys | Yes | Processor and entry | | flatten | Yes | Processor | | lowercase\$1string | Yes | Processor | | uppercase\$1string | Yes | Processor | | trim\$1string | Yes | Processor | | substitute\$1string | Yes | Processor and entry | | truncate | Yes | Processor | | extract\$1value | Yes | Processor and entry | | convert\$1entry\$1type | Yes | Processor | | date | Yes | Processor | | dissect | Yes | Processor | | list\$1to\$1map | Yes | Processor | | rename\$1keys | Yes | Processor and entry | | select\$1entries | Yes | Processor | | translate | Yes | Processor | | grok | Yes | Processor | | drop\$1events | Yes | Processor (required) | | OCSF, CSV, JSON, KeyValue, WAF, Postgres, CloudFront, VPC, Route53 | No | — | ## Operators **Supported operators** | Category | Operators | Example | | --- | --- | --- | | Relational | <, <=, >, >= | status\$1code >= 200 and status\$1code < 300 | | Equality | ==, \$1= | log.level == "ERROR" | | Conditional | and, or, not | log.level == "ERROR" or log.level == "FATAL" | | Arithmetic | \$1, -, \$1, / | response\$1time \$1 1000 > 5000 | | Set membership | in, not in | environment in \$1"prod", "staging", "preprod"\$1 | | Regex matching | =\$1, \$1\$1 | message =\$1 "^ERROR.\$1timeout" | ## Functions `length(value)` Returns the length of a string or array. Example: `length(message) > 100` `contains(value, search)` Checks whether a string contains a substring or an array contains an element. Example: `contains(message, "error")` `startsWith(field, prefix)` Checks whether a string starts with a specified prefix. Example: `startsWith(message, "ERROR")` ## Expression examples ``` log.level == "ERROR" status_code >= 200 and status_code < 300 environment in {"prod", "staging", "preprod"} message =~ "^ERROR.*timeout" user.role == "admin" and user.permissions.write == true length(message) > 100 and contains(message, "error") (log.level == "ERROR" or log.level == "FATAL") and environment == "prod" ``` ## Limitations + Expression maximum length is 256 characters. + Parser processors (except Grok) do not support conditional processing. This includes JSON, CSV, KeyValue, WAF, Postgres, CloudFront, VPC, Route53, and OCSF parsers. + If the Grok processor is used as the parser (first processor) in a pipeline and its `when` condition evaluates to false, the entire pipeline does not execute for that log event. # Sinks Sinks define the destination where processed log data is sent. Each pipeline must have exactly one sink. Currently, only CloudWatch Logs sink is supported. **Sink behavior by source type** | Source Type | Log Group Configuration | Behavior | | --- | --- | --- | | CloudWatch Logs | Must use @original | Events are sent back to their original log group | | S3 | Custom log group path | Events are sent to the specified log group | | Third-party APIs | Custom log group path | Events are sent to the specified log group | **Configuration** Configure the sink with the following parameters: **Example Non-CloudWatch Logs source configuration** ``` sink: cloudwatch_logs: log_group: "/aws/my-application/logs" ``` **Example CloudWatch Logs source configuration** ``` sink: cloudwatch_logs: log_group: "@original" ```Parameters `log_group` (required) The name of the CloudWatch Logs log group where processed events will be sent. For pipelines with non-`cloudwatch_logs` sources, this must be an existing log group name. For pipelines using the `cloudwatch_logs` source, the ONLY allowed value is `@original`. `include_original` (optional) When present, stores a copy of each raw log event before any transformation takes place. This preserves the original data for audit or compliance purposes. Specify as an empty object (`{}`). Available only for pipelines with `cloudwatch_logs` sources. At least one processor must be configured when this option is enabled. **Example CloudWatch Logs sink with original log preservation** ``` sink: - cloudwatch_logs: log_group: "@original" include_original: {} ``` ## Requirements and limitations Log group existence If created using the AWS Management Console, CloudWatch will attempt to create the specified log group and appropriate resource policy if it does not exist when using a non-CloudWatch logs source. Otherwise, the specified log group must exist before creating the pipeline. Event size Each log event cannot exceed 256 KB in size after processing. Log group retention The pipeline uses the retention settings configured on the destination log group. Log group resource policy CloudWatch Logs resource policies are required for pipelines that write to log groups, except for pipelines using the `cloudwatch_logs` source. When you use the AWS Management Console to configure the pipeline, CloudWatch will attempt to add the resource policy if needed. If you are creating the pipeline using the AWS CLI or an API, you must create the policy manually and add it using the `logs:PutResourcePolicy` request. For more information, see [Resource policies](pipeline-iam-reference.md#resource-policies). Cross-Region support The destination log group must be in the same Region as the pipeline. **Important** For pipelines using the `cloudwatch_logs` source type: You must use `@original` as the log group value. Events are always sent back to their original log group. The original log group must exist throughout the pipeline's lifecycle. Pipelines with processors mutate the log events in the original CloudWatch log group they are intercepted from for logs from AWS services. To preserve the original data, enable `include_original` in the sink configuration. **Note** Log events are subject to CloudWatch Logs quotas and limitations. # CloudWatch pipelines IAM policies and permissions This section provides detailed IAM requirements for CloudWatch pipelines, including permissions for API callers, source-specific policies, trust relationships, and resource policies. ## API caller permissions Any role specified in the pipeline configuration that calls the `CreateTelemetryPipeline` API (such as S3 source roles, Secrets Manager access roles, or CloudWatch Logs source roles) must have specific permissions to pass roles. **PassRole permissions** Required for any roles specified in the pipeline configuration (S3 source roles, Secrets Manager access roles, or CloudWatch Logs source roles). **Example IAM policy for S3 sources** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForS3Source", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-s3-source-role" } ] } ``` **Example IAM policy for Secrets Manager sources** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForSecretsManagerSource", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-secrets-manager-role" } ] } ``` **Example IAM Policy for CloudWatch Logs Sources** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForCloudWatchLogsSource", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-cloudwatch-logs-role"" } ] } ``` **Pipeline rule permissions** When using `cloudwatch_logs` source for Create/Update operations (`logs:PutPipelineRule`) and Delete operations (`logs:DeletePipelineRule`) the role must also have permissions to perform those operations. **Example IAM policy for CloudWatch Logs pipeline rules** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PipelineRuleForCloudWatchLogs", "Effect": "Allow", "Action": [ "logs:PutPipelineRule", "logs:DeletePipelineRule" ], "Resource": "*" } ] } ``` **Reducing scope with condition keys** To scope down the permission policy to telemetry pipelines, you can specify Condition Keys as shown in the following examples: **Example IAM policy for S3 sources (basic)** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForS3Source", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-s3-source-role" } ] } ``` **Example IAM policy for S3 sources (scoped down with condition keys)** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForS3Source", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-s3-source-role", "Condition": { "StringEquals": { "iam:PassedToService": [ "telemetry-pipelines.observabilityadmin.amazonaws.com" ], "iam:AssociatedResourceARN": [ "arn:aws:observabilityadmin:your-region:your-account-id:telemetry-pipeline/*" ] } } } ] } ``` **Example IAM policy for Secrets Manager sources (basic)** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForSecretsManagerSource", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-secrets-manager-role" } ] } ``` **Example IAM policy for Secrets Manager sources (scoped down with condition keys)** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForSecretsManagerSource", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-secrets-manager-role", "Condition": { "StringEquals": { "iam:PassedToService": [ "telemetry-pipelines.observabilityadmin.amazonaws.com" ], "iam:AssociatedResourceARN": [ "arn:aws:observabilityadmin:your-region:your-account-id:telemetry-pipeline/*" ] } } } ] } ``` **Example IAM policy for CloudWatch Logs sources (scoped down with condition keys)** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PassRoleForCloudWatchLogsSource", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::your-account-id:role/your-cloudwatch-logs-role", "Condition": { "StringEquals": { "iam:PassedToService": [ "logs.amazonaws.com" ], "iam:AssociatedResourceARN": [ "arn:aws:observabilityadmin:your-region:your-account-id:telemetry-pipeline/*" ] } } } ] } ``` ## Pipeline condition keys CloudWatch pipelines supports IAM condition keys that let you restrict who can create pipelines based on the log source name and type. Use these condition keys to enforce governance policies across your organization.Available condition keys `observabilityadmin:SourceName` Restricts pipeline creation to specific log source names. `observabilityadmin:SourceType` Restricts pipeline creation to specific log source types. **Example IAM policy restricting pipeline creation by source type** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowPipelineCreationForSpecificSourceType", "Effect": "Allow", "Action": "observabilityadmin:CreateTelemetryPipeline", "Resource": "*", "Condition": { "StringEquals": { "observabilityadmin:SourceType": "cloudwatch_logs" } } } ] } ``` **Example IAM policy restricting pipeline creation by source name** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowPipelineCreationForSpecificSource", "Effect": "Allow", "Action": "observabilityadmin:CreateTelemetryPipeline", "Resource": "*", "Condition": { "StringEquals": { "observabilityadmin:SourceName": "your-source-name" } } } ] } ``` ## Source-specific IAM policies Different source types require specific IAM permissions to access their respective data sources. **CloudWatch Logs sources** For CloudWatch Logs sources, any IAM role specified in the pipeline configuration must have a trust relationship with `logs.amazonaws.com`. **Example IAM role trust policy for CloudWatch Logs sources (basic)** ``` { "Version": "2012-10-17", "Statement": [ { ""Effect": "Allow", "Principal": { "Service": "logs.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` **S3 sources** For S3 sources, customers must provide an IAM role with permissions to access S3 objects and SQS queues. **Example IAM policy for S3 sources** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "s3-access", "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::your-bucket-name/*" }, { "Sid": "sqs-access", "Effect": "Allow", "Action": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:ChangeMessageVisibility" ], "Resource": "arn:aws:sqs:your-region:your-account-id:your-queue-name" }, { "Sid": "kms-access", "Effect": "Allow", "Action": "kms:Decrypt", "Resource": "arn:aws:kms:your-region:your-account-id:key/your-key-id", "Condition": { "Comment": "Only required if S3 buckets and/or SQS queue uses KMS encryption" } } ] } ``` **Sources using AWS Secrets Manager** For sources that reference AWS Secrets Manager (Microsoft Office 365, Microsoft Entra ID, Palo Alto NGFW), customers must provide an IAM role with Secrets Manager access. **Example IAM policy for Secrets Manager sources** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "secrets-manager-access", "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:your-region:your-account-id:secret:your-secret-name*" }, { "Sid": "kms-access", "Effect": "Allow", "Action": "kms:Decrypt", "Resource": "arn:aws:kms:your-region:your-account-id:key/your-key-id", "Condition": { "Comment": "Only required if Secrets Manager uses KMS encryption" } } ] } ``` ## Trust relationships Any IAM role specified in the pipeline configuration must have a trust relationship with the CloudWatch pipelines service principal. **Pipeline role trust policy** All pipeline roles must trust the `telemetry-pipelines.observabilityadmin.amazonaws.com` service principal. **Example Trust policy for pipeline roles** ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "telemetry-pipelines.observabilityadmin.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ## Resource policies CloudWatch Logs resource policies are required for pipelines that write to log groups, except for pipelines using the `cloudwatch_logs` source. **CloudWatch Logs** resource policy After calling `CreateTelemetryPipeline` API, you will receive a pipeline ARN. For pipelines where the source is not `cloudwatch_logs`, customers must call `[logs:PutResourcePolicy](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutResourcePolicy.html)` to allow the CloudWatch pipelines service principal to write to the configured log group. **Timing constraint** You have a limited time window (less than 5 minutes) to create the resource policy after receiving the pipeline ARN. If the pipeline becomes active before the policy is in place, data will be dropped. **Example logs:PutResourcePolicy request** ``` { "policyName": "resourceArn=arn:aws:logs:your-region:your-account-id:log-group:your-log-group-name:*", "policyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "telemetry-pipelines.observabilityadmin.amazonaws.com" }, "Action": [ "logs:CreateLogStream", "logs:PutLogEvents" ], "Condition": { "StringEquals": { "aws:SourceArn": "arn:aws:observabilityadmin:your-region:your-account-id:telemetry-pipeline/your-pipeline-id" } } } ] } } ``` ## Managing resource policies This guide provides steps for creating or updating a CloudWatch Logs resource policy for telemetry pipelines using the AWS CLI. Check for existing policies: ``` aws logs describe-resource-policies --resource-arn arn:aws:logs:your-region:your-account-id:log-group:your-log-group-name:* ``` This returns all existing resource policies attached to the log group. Look for any policy that might already be associated with your log group. If no resource policy exists, create a new one: ``` aws logs put-resource-policy \ --region \ --policy-name "resourceArn": "arn:aws:logs:your-region:your-account-id:log-group:your-log-group-name:*"\ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "telemetry-pipelines.observabilityadmin.amazonaws.com" }, "Action": [ "logs:CreateLogStream", "logs:PutLogEvents" ], "Condition": { "StringEquals": { "aws:SourceArn": "arn:aws:observabilityadmin:your-region:your-account-id:telemetry-pipeline/your-pipeline-id" } } } ] }' ``` Replace the following placeholders: + *your-region* - Your AWS region (e.g., us-east-1) + *your-account-id* - Your 12-digit AWS account ID + *your-log-group-name* - Your CloudWatch Logs log group name + *your-pipeline-id* - Your telemetry pipeline ID If a resource policy already exists, merge the new statement with it: 1. Retrieve the existing policy: ``` aws logs describe-resource-policies --resource-arn arn:aws:logs:your-region:your-account-id:log-group:your-log-group-name:* ``` 1. Open `existing-policy.json` and add the new statement to the existing `Statement` array: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "existing-service.amazonaws.com" }, "Action": [ "logs:SomeAction" ] }, { "Effect": "Allow", "Principal": { "Service": "telemetry-pipelines.observabilityadmin.amazonaws.com" }, "Action": [ "logs:CreateLogStream", "logs:PutLogEvents" ], "Condition": { "StringEquals": { "aws:SourceArn": "arn:aws:observabilityadmin:your-region:your-account-id:telemetry-pipeline/your-pipeline-id" } } } ] } ``` 1. Update the policy: ``` aws logs put-resource-policy \ --region your-region \ --policy-name resourceArn=arn:aws:logs:your-region:your-account-id:log-group:your-log-group-name:* \ --policy-document file://existing-policy.json ``` Confirm the policy was created or updated successfully: ``` aws logs describe-resource-policies --resource-arn arn:aws:logs:your-region:your-account-id:log-group:your-log-group-name:* ``` # CloudWatch pipelines extensions CloudWatch pipelines extensions provide additional functionality to the pipeline. You can use the AWS Secrets Manager integration for credential management. ## AWS Secrets Manager extension Configures access to AWS Secrets Manager for retrieving credentials and sensitive configuration values. This extension is only supported for third-party sources that require authentication credentials. **Configuration** Configure the AWS Secrets Manager extension with the following parameters: ``` extension: aws: secrets: : secret_id: "" region: "" sts_role_arn: "arn:aws:iam::123456789012:role/Example-Role" refresh_interval: PT1H disable_refresh: false ```Parameters `aws.secrets..secret_id` (required) The ARN of the AWS Secrets Manager secret containing the credentials. `aws.secrets..region` (required) The AWS region where the secret is stored. `aws.secrets..sts_role_arn` (required) The ARN of the IAM role to assume for accessing the AWS Secrets Manager secret. `aws.secrets..refresh_interval` (optional) How often to refresh the secret from AWS Secrets Manager. Uses ISO 8601 duration format. Defaults to PT1H (1 hour). `aws.secrets..disable_refresh` (optional) Whether to disable automatic secret refresh. Defaults to false. ### Secret reference syntax Reference secrets in your pipeline configuration using the following syntax: ``` ${{aws_secrets::}} ``` For example, to reference a client ID and secret: ``` source: microsoft_office365: authentication: oauth2: client_id: "${{aws_secrets:office365-creds:client_id}}" client_secret: "${{aws_secrets:office365-creds:client_secret}}" ``` ### Requirements and limitations Secret format Secrets must be stored as JSON key-value pairs in AWS Secrets Manager. Cross-Region access Secrets can be accessed from any Region where AWS Secrets Manager is available. Refresh interval limits Minimum refresh interval is 5 minutes (PT5M). Maximum is 24 hours (PT24H). Maximum secrets A pipeline can reference up to 10 different secrets. **Important** Consider the following when using secrets: Ensure the IAM role has appropriate permissions to access the secrets Monitor secret access using AWS CloudTrail Use separate secrets for different environments (development, production) # Monitoring Pipelines Using CloudWatch Metrics CloudWatch pipelines publishes metrics to Amazon CloudWatch in the `AWS/Observability Admin` namespace. You can use these metrics to monitor your pipelines' health, performance, and data flow. ## Available metrics The following tables list the available metrics for CloudWatch pipelines. **Note** Pipelines metrics are only emitted when the value is non-zero. ### Core metrics | Metric | Description | Dimension | Unit | | --- | --- | --- | --- | | `PipelineBytesIn` | Volume of log records going into pipeline in uncompressed bytes | PipelineName | Bytes | | `PipelineBytesInByDataSource` | Volume of incoming data with source/type breakdown | PipelineName, DataSource, DataType | Bytes | | `PipelineBytesOut` | Volume of data routed to destination | PipelineName | Bytes | | `PipelineBytesOutByDataSource` | Volume of outgoing data with source/type breakdown | PipelineName, DataSource, DataType | Bytes | | `PipelineRecordsIn` | Number of records entering the pipeline | PipelineName | Count | | `PipelineRecordsInByDataSource` | Number of incoming records with source/type breakdown | PipelineName, DataSource, DataType | Count | | `PipelineRecordsOut` | Number of records exiting the pipeline | PipelineName | Count | | `PipelineRecordsOutByDataSource` | Number of outgoing records with source/type breakdown | PipelineName, DataSource, DataType | Count | ### Error and warning metrics | Metric | Description | Dimension | Unit | | --- | --- | --- | --- | | `PipelineErrors` | Aggregate count of errors in pipeline | PipelineName | Count | | `PipelineErrorsByErrorType` | Detailed error counts by type | PipelineName, ErrorSource, ErrorComponent, ErrorType | Count | | `PipelineWarnings` | Number of warnings encountered | PipelineName | Count | | `PipelineWarningsByWarningType` | Detailed warnings by type | PipelineName, WarningSource, WarningComponent, WarningType | Count | | `PipelineRecordsUnprocessed` | Number of records that couldn't be processed | PipelineName, DataSource, DataType | Count | | `PipelineRecordsDropped` | Number of records dropped (third-party sources only) | PipelineName, DataSource, DataType | Count | ## Dimensions CloudWatch pipelines metrics use the following dimensions: **PipelineName** Name of the pipeline **DataSource** Source of the data (AWS service name or third-party source) **DataType** Type of data being processed **ErrorSource** Origin of the error (s3, aws.secrets, cloudwatch\$1logs) **ErrorComponent** Component where error occurred (source, sink, extension) **ErrorType** Type of error encountered ## Error types The following error types are tracked in `PipelineErrorsByErrorType`: **`ACCESS_DENIED`** Permission-related failures **`ALL`** The total count of all errors on the pipeline **`RESOURCE_NOT_FOUND`** Specified resource doesn't exist **`SOURCE_READ_FAILURE`** Failures reading from source **`PARSE_FAILURE`** Data parsing errors **`PROCESSOR_ERRORS`** Processing operation failures **`PAYLOAD_SIZE_EXCEEDED`** Data size limit exceeded ## Warning types The following warning type can occur on a pipeline: **`THROTTLED`** Indicates that the volume of data being sent has exceeded existing rate limits, causing some data points or events to be dropped or delayed to protect the system and ensure stability. ## Viewing metrics You can view CloudWatch pipelines metrics using the following methods: ### Using the CloudWatch console 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/) 1. In the navigation pane, choose **Metrics** 1. Choose the **AWS/Observability Admin** namespace 1. Select the metric dimension to view ### Using the AWS CLI ``` aws cloudwatch get-metric-statistics \ --namespace "AWS/Observability Admin" \ --metric-name "PipelineBytesIn" \ --dimensions Name=PipelineName,Value=my-pipeline \ --start-time "2025-10-29T00:00:00" \ --end-time "2025-10-29T23:59:59" \ --period 300 \ --statistics Sum ``` ## Creating alarms You can create CloudWatch alarms based on any of these metrics. Here's an example of creating an alarm for pipeline errors: ``` aws cloudwatch put-metric-alarm \ --alarm-name "HighPipelineErrors" \ --alarm-description "Alert on high error rate" \ --metric-name "PipelineErrors" \ --namespace "AWS/Observability Admin" \ --dimensions Name=PipelineName,Value=my-pipeline \ --period 300 \ --evaluation-periods 2 \ --threshold 10 \ --comparison-operator GreaterThanThreshold \ --statistic Sum \ --alarm-actions arn:aws:sns:region:account-id:topic-name ``` ## Best practices for CloudWatch pipelines metrics ### Monitor data flow + Use `PipelineBytesIn` and `PipelineBytesOut` to track data volume + Monitor `PipelineRecordsIn` and `PipelineRecordsOut` to track record counts + Watch for unexpected changes in throughput patterns ### Track errors and warnings + Create alarms for `PipelineErrors` to detect issues quickly + Use `PipelineErrorsByErrorType` to diagnose specific problems + Monitor `PipelineWarnings` to identify potential issues early ### Configure appropriate thresholds + Base thresholds on your expected data patterns + Account for normal variations in data volume + Consider peak usage periods when setting alarm thresholds # Troubleshooting This section covers common issues you might encounter when working with CloudWatch pipelines and their solutions. **Pipeline creation failures** + **Issue:** Pipeline creation fails with permission errors **Solution:** Verify that your IAM role has the required PassRole permissions for any roles specified in the pipeline configuration + **Issue:** Configuration validation errors **Solution:** Use the ValidateTelemetryPipeline API to check your configuration syntax before creating the pipeline + **Issue:** Log source not available to select **Possible reason:** There is already a log source with the same data source name and type. Duplicate combinations are greyed out since you can only create one of each. **Data processing issues** + **Issue:** No data flowing through pipeline **Solution:** Check that your data source is properly configured and generating data, and verify that CloudWatch Logs resource policies are correctly set + **Issue:** Processor errors in pipeline **Solution:** Review processor configuration and ensure input data format matches processor expectations