Working with Amazon OpenSearch Service direct queries - Amazon OpenSearch Service
QuotasSupported AWS Regions

Working with Amazon OpenSearch Service direct queries

You can use Amazon OpenSearch Service direct query to analyze data in Amazon CloudWatch Logs, Amazon S3, and Amazon Security Lake. OpenSearch Service provides a zero-ETL integration as a way to analyze your log data using OpenSearch SQL or OpenSearch Piped Processing Language (PPL) without incurring the friction of building ingestion pipelines or switching between analytics tools. This approach eliminates the need for data movement or duplication, allowing you to analyze your data where it rests using OpenSearch Discover. When you want to switch from querying data at rest to actively monitoring with dashboards or alerts, you can build indexed views on the data and ingest it into an OpenSearch Service index.

To get started, you configure a data source in the OpenSearch Service console. For Amazon S3, you use the domain's connections, while for CloudWatch Logs and Security Lake, you use connected data sources under Central Management in the console. Amazon S3 and Security Lake both use tables in AWS Glue Data Catalog to represent your data structure, including schema, file type, and partitioning. For Amazon S3, you create these tables within OpenSearch Query Workbench using CREATE TABLE SQL statements. For Amazon Security Lake, the tables in AWS Glue are already set up during the Security Lake setup process. CloudWatch Logs similarly has pre-configured log groups.

After you set up your data source, you sign in to Discover, where you can select your data source and choose the relevant tables (for Amazon S3 and Security Lake) or log groups (for CloudWatch Logs). From there, you can start querying your data directly.

To use advanced analytics features of OpenSearch Service for data monitoring, such as building dashboards and full-text search, you ingest data from your direct query data source by creating an indexed view on the data. You can create indexed views using common SQL indexing techniques, such as skipping indexes, materialized views, and covering indexes (where supported). To help you get started quickly building dashboards, you can use pre-built templates for common log types like VPC Flow Logs, AWS CloudTrail logs, and AWS WAF logs.

Direct query quotas

Your account has the following quotas related to OpenSearch Service direct queries.

Quotas for Amazon S3

Each time you initiate a query to an Amazon S3 data source, OpenSearch Service opens a session and keeps it alive for at least three minutes. This reduces query latency by removing session start-up time in subsequent queries.

Description Maximum Can override
Connections per domain 10 Yes
Data sources per domain 20 Yes
Indexes per domain 5 Yes
Concurrent sessions per data source 10 Yes
Maximum OCU per query 60 Yes
Maximum query execution time (minutes) 30 Yes
Maximum OCUs per acceleration 20 Yes
Maximum ephemeral storage 20 Yes

Quotas for CloudWatch Logs

Note

If you're looking to perform direct queries using CloudWatch Logs Insights, make sure that you refer to Additional information for CloudWatch Logs Insights users using OpenSearch SQL.

Description Value Soft limit? Notes
Account-level TPS limit across direct query APIs 3 TPS Yes
Maximum number of data sources 20 Yes Limit is per AWS account.
Maximum auto-refreshing indexes or materialized views 30 Yes Limit is per data source.
Maximum concurrent queries 15 Yes

Limit applies to queries in pending or running state.

Includes interactive queries (for example, data retrieval commands like SELECT) and index queries (for example, operations like CREATE/ALTER/DROP).

Maximum concurrent OCU per query 512 Yes

OpenSearch Compute Units (OCU). Limit based on 15 executors and 1 driver, each with 16 vCPU and 32 GB memory. Represents concurrent processing power.
Maximum query execution time in minutes 60 No Limit applies to OpenSearch PPL/SQL queries in CloudWatch Logs Insights.
Period for purging stale query IDs 90 days Yes This is the time period after which OpenSearch Service purges query metadata for older entries. For example, calling GetDirectQuery or GetDirectQueryResult fails for queries older than 90 days.

Quotas for Security Lake

Description Value Soft limit? Notes
Account-level TPS limit across direct query APIs 3 TPS Yes
Maximum number of data sources 20 Yes Limit is per AWS account.
Maximum auto-refreshing indexes or materialized views 30 Yes

Limit applies per data source.

Only includes indices and materialized views (MVs) with auto-refresh set to true.
Maximum concurrent queries 30 Yes

Limit applies to queries in pending or running state.

Includes interactive queries (for example, data retrieval commands like SELECT) and index queries (for example, operations like CREATE/ALTER/DROP).

Maximum concurrent OCU per query 512 Yes

OpenSearch Compute Units (OCU). Limit based on 15 executors and 1 driver, each with 16 vCPU and 32 GB memory. Represents concurrent processing power.
Maximum query execution time in minutes 30 No Applies only to interactive queries (for example, data retrieval commands like SELECT). For REFRESH queries, the limit is 6 hours.
Period for purging stale query IDs 90 days Yes

This is the time period after which OpenSearch Service purges query metadata for older entries. For example, calling GetDirectQuery or GetDirectQueryResult fails for queries older than 90 days.

Supported AWS Regions

The following AWS Regions are supported for OpenSearch Service direct queries in Amazon S3, CloudWatch Logs, and Security Lake:

Available AWS Regions for Amazon S3

  • Asia Pacific (Hong Kong)

  • Asia Pacific (Mumbai)

  • Asia Pacific (Seoul)

  • Asia Pacific (Singapore)

  • Asia Pacific (Sydney)

  • Asia Pacific (Tokyo)

  • Canada (Central)

  • Europe (Frankfurt)

  • Europe (Ireland)

  • Europe (Stockholm)

  • US East (N. Virginia)

  • US East (Ohio)

  • US West (Oregon)

Available AWS Regions for CloudWatch Logs

  • Asia Pacific (Mumbai)

  • Asia Pacific (Singapore)

  • Asia Pacific (Sydney)

  • Asia Pacific (Tokyo)

  • Canada (Central)

  • Europe (Frankfurt)

  • Europe (Ireland)

  • US East (N. Virginia)

  • US East (Ohio)

  • US West (Oregon)

  • Europe (Paris)

  • Europe (London)

  • South America (Sao Paulo)

Available AWS Regions for Security Lake

  • Asia Pacific (Mumbai)

  • Asia Pacific (Singapore)

  • Asia Pacific (Sydney)

  • Asia Pacific (Tokyo)

  • Canada (Central)

  • Europe (Frankfurt)

  • Europe (Ireland)

  • US East (N. Virginia)

  • US East (Ohio)

  • US West (Oregon)

  • Europe (Paris)

  • Europe (London)

  • South America (Sao Paulo)