Using CloudWatch to monitor and log GraphQL API data

You can log and debug your GraphQL API using CloudWatch metrics and CloudWatch logs. These tools enable developers to monitor performance, troubleshoot issues, and optimize their GraphQL operations effectively.

CloudWatch metrics is a tool that provides a wide range of metrics to monitor API performance and usage. These metrics fall into two main categories:

The guide also introduces Enhanced Metrics, which offer more granular data on resolver performance, data source interactions, and individual GraphQL operations. These metrics provide deeper insights but come with additional costs.

CloudWatch Logs is a tool that enables logging capabilities for your GraphQL APIs. Logs can be set at two levels of the API:

You can configure logging, interpret log entries, and use log data for troubleshooting and optimization. AWS AppSync provides various log types that reveal your query’s execution, parsing, validation, and field resolution data.

Setup and configuration

To turn on automatic logging on a GraphQL API, use the AWS AppSync console.

Manual IAM role configuration

If you choose to use an existing IAM role, the role must grant AWS AppSync the required permissions to write logs to CloudWatch. To configure this manually, you must provide a service role ARN so that AWS AppSync can assume the role when writing the logs.

In the IAM console, create a new policy with the name AWSAppSyncPushToCloudWatchLogsPolicy that has the following definition:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Next, create a new role with the name AWSAppSyncPushToCloudWatchLogsRole, and attach the newly created policy to the role. Edit the trust relationship for this role to the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal": {
            "Service": "appsync.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
        }
    ]
}

Copy the role ARN and use it when setting up logging for an AWS AppSync GraphQL API.

CloudWatch metrics

You can use CloudWatch metrics to monitor and provide alerts about specific events that can result in HTTP status codes or from latency. The following metrics are emitted:

Real-time subscriptions

All metrics are emitted in one dimension: GraphQLAPIId. This means that all metrics are coupled with GraphQL API IDs. The following metrics are related to GraphQL subscriptions over pure WebSockets:

Comparing inbound and outbound messages

When a mutation is executed, subscription fields with the @aws_subscribe directive for that mutation are invoked. Each subscription invocation generates one inbound message. For example, if two subscription fields specify the same mutation in @aws_subscribe, then two inbound messages are generated when that mutation is called.

One outbound message equals 5 kB of data delivered to WebSocket clients. For example, sending 15 kB of data to 10 clients results in 30 outbound messages (15 kB * 10 clients / 5 kB per message = 30 messages).

You can request quota increases for either inbound or outbound messages. For more information, see AWS AppSync endpoints and quotas in the AWS General Reference guide and the instructions for Requesting a quota increase in the Service Quotas User Guide.

Enhanced metrics

Enhanced metrics emit granular data on API usage and performance such as AWS AppSync request and error counts, latency, and cache hits/misses. All enhanced metric data is sent to your CloudWatch account, and you can configure the types of data that will be sent.

These metrics can be found on various settings pages in the AWS AppSync console. On the API settings page, the Enhanced Metrics section allows you to enable or disable the following items:

Resolver metrics behavior: These options control how additional metrics for resolvers are collected. You can choose to enable full request resolver metrics (metrics enabled for all resolvers in requests) or per-resolver metrics (metrics only enabled for resolvers where the configuration is set to enabled). The following options are available:

Data source metrics behavior: These options control how additional metrics for data sources are collected. You can choose to enable full request data source metrics (metrics enabled for all data sources in requests) or per-data source metrics (metrics only enabled for data sources where the configuration is set to enabled). The following options are available:

Operation metrics: Enables GraphQL operation-level metrics.

CloudWatch logs

You can configure two types of logging on any new or existing GraphQL API: request-level and field-level.

Request-level logs

When request-level logging (Include verbose content) is configured, the following information is logged:

Field-level logs

When field-level logging is configured, the following information is logged:

If you turn on logging, AWS AppSync manages the CloudWatch Logs. The process includes creating log groups and log streams, and reporting to the log streams with these logs.

When you turn on logging on a GraphQL API and make requests, AWS AppSync creates a log group and log streams under the log group. The log group is named following the /aws/appsync/apis/{graphql_api_id} format. Within each log group, the logs are further divided into log streams. These are ordered by Last Event Time as logged data is reported.

Every log event is tagged with the x-amzn-RequestId of that request. This helps you filter log events in CloudWatch to get all logged information about that request. You can get the RequestId from the response headers of every GraphQL AWS AppSync request.

The field-Level logging is configured with the following log levels:

Benefits of monitoring

You can use logging and metrics to identify, troubleshoot, and optimize your GraphQL queries. For example, these will help you debug latency issues using the tracing information that is logged for each field in the query. To demonstrate this, suppose you are using one or more resolvers nested in a GraphQL query. A sample field operation in CloudWatch Logs might look similar to the following:

{
    "path": [
        "singlePost",
        "authors",
        0,
        "name"
    ],
    "parentType": "Post",
    "returnType": "String!",
    "fieldName": "name",
    "startOffset": 416563350,
    "duration": 11247
}

This might correspond to a GraphQL schema, similar to the following:

type Post {
  id: ID!
  name: String!
  authors: [Author]
}

type Author {
  id: ID!
  name: String!
}

type Query {
  singlePost(id:ID!): Post
}

In the preceding log results, path shows a single item in your data returned from running a query named singlePost(). In this example, it’s representing the name field at the first index (0). The startOffset gives an offset from the start of the GraphQL query operation. The duration is the total time to resolve the field. These values can be useful to troubleshoot why data from a particular data source might be running slower than expected, or if a specific field is slowing down the entire query. For example, you might choose to increase provisioned throughput for an Amazon DynamoDB table, or remove a specific field from a query that is causing the overall operation to perform poorly.

As of May 8, 2019, AWS AppSync generates log events as fully structured JSON. This can help you use log analytics services such as CloudWatch Logs Insights and Amazon OpenSearch Service to understand the performance of your GraphQL requests and usage characteristics of your schema fields. For example, you can easily identify resolvers with large latencies that may be the root cause of a performance issue. You can also identify the most and least frequently used fields in your schema and assess the impact of deprecating GraphQL fields.

Conflict detection and sync logging

If an AWS AppSync API has logging to CloudWatch Logs configured with the Field resolver log level set to All, then AWS AppSync emits conflict detection and resolution information to the log group. This provides granular insight into how the AWS AppSync API responded to a conflict. To help you interpret the response, the following information is provided in the logs:

Using token counts to optimize your requests

Requests that consume less than or equal to 1,500 KB-seconds of memory and vCPU time are allocated one token. Requests with resource consumption greater than 1,500 KB-seconds receive additional tokens. For example, if a request consumes 3,350 KB-seconds, AWS AppSync allocates three tokens (rounded up to the next integer value) to the request. By default, AWS AppSync allocates a maximum of 5,000 or 10,000 request tokens per second to the APIs in your account, depending upon the AWS Region in which it's deployed. If your APIs each use an average of two tokens per second, you'll be limited to 2,500 or 5,000 requests per second, respectively. If you need more tokens per second than the allotted amount, you can submit a request to increase the default quota for the rate of request tokens. For more information, see AWS AppSync endpoints and quotas in the AWS General Reference guide and Requesting a quota increase in the Service Quotas User Guide.

A high per-request token count could indicate that there's an opportunity to optimize your requests and improve the performance of your API. Factors that can increase your per-request token count include:

Log size limits

By default, if logging has been enabled, AWS AppSync will send up to 1 MB of logs per request. Logs exceeding this size will be truncated. To reduce log sizes, choose the ERROR logging level for field-level logs and disable VERBOSE logging, or disable field-level logs entirely if not needed. As an alternative to the ALL log level, you can use Enhanced Metrics to obtain metrics on specific resolvers, data sources, or GraphQL operations, or utilize the logging utilities provided by AppSync to log only the necessary information.

Log type reference

RequestSummary

{
     "logType": "RequestSummary",
     "requestId": "dbe87af3-c114-4b32-ae79-8af11f3f96f1",
     "graphQLAPIId": "pmo28inf75eepg63qxq4ekoeg4",
     "statusCode": 200,
     "latency": 242000000
}

ExecutionSummary

{
    "duration": 217406145,
    "logType": "ExecutionSummary",
    "requestId": "dbe87af3-c114-4b32-ae79-8af11f3f96f1",
    "startTime": "2019-01-01T06:06:18.956Z",
    "endTime": "2019-01-01T06:06:19.174Z",
    "parsing": {
        "startOffset": 49033,
        "duration": 34784
    },
    "version": 1,
    "validation": {
        "startOffset": 129048,
        "duration": 69126
    },
    "graphQLAPIId": "pmo28inf75eepg63qxq4ekoeg4"
}

Tracing

{
    "duration": 216820346,
    "logType": "Tracing",
    "path": [
        "putItem"
    ],
    "fieldName": "putItem",
    "startOffset": 178156,
    "resolverArn": "arn:aws:appsync:us-east-1:111111111111:apis/pmo28inf75eepg63qxq4ekoeg4/types/Mutation/fields/putItem",
    "requestId": "dbe87af3-c114-4b32-ae79-8af11f3f96f1",
    "parentType": "Mutation",
    "returnType": "Item",
    "graphQLAPIId": "pmo28inf75eepg63qxq4ekoeg4"
}

Analyzing your logs with CloudWatch Logs Insights

The following are examples of queries you can run to get actionable insights into the performance and health of your GraphQL operations. These examples are available as sample queries in the CloudWatch Logs Insights console. In the CloudWatch console, choose Logs Insights, select the AWS AppSync log group for your GraphQL API, and then choose AWS AppSync queries under Sample queries.

The following query returns the top 10 GraphQL requests with maximum tokens consumed:

filter @message like "Tokens Consumed"
| parse @message "* Tokens Consumed: *" as requestId, tokens
| sort tokens desc
| display requestId, tokens
| limit 10

The following query returns the top 10 resolvers with maximum latency:

  fields resolverArn, duration
| filter logType = "Tracing"
| limit 10
| sort duration desc

The following query returns the most frequently invoked resolvers:

  fields ispresent(resolverArn) as isRes
| stats count() as invocationCount by resolverArn
| filter isRes and logType = "Tracing"
| limit 10
| sort invocationCount desc

The following query returns resolvers with the most errors in mapping templates:

  fields ispresent(resolverArn) as isRes
| stats count() as errorCount by resolverArn, logType
| filter isRes and (logType = "RequestMapping" or logType = "ResponseMapping") and fieldInError
| limit 10
| sort errorCount desc

The following query returns resolver latency statistics:

  fields ispresent(resolverArn) as isRes
| stats min(duration), max(duration), avg(duration) as avg_dur by resolverArn
| filter isRes and logType = "Tracing"
| limit 10
| sort avg_dur desc

The following query returns field latency statistics:

  stats min(duration), max(duration), avg(duration) as avg_dur
  by concat(parentType, '/', fieldName) as fieldKey
| filter logType = "Tracing"
| limit 10
| sort avg_dur desc

The results of CloudWatch Logs Insights queries can be exported to CloudWatch dashboards.

Analyze your logs with OpenSearch Service

You can search, analyze, and visualize your AWS AppSync logs with Amazon OpenSearch Service to identify performance bottlenecks and root causes of operational issues. You can identify resolvers with the maximum latency and errors. In addition, you can use OpenSearch Dashboards to create dashboards with powerful visualizations. OpenSearch Dashboards is an open source data visualization and exploration tool available in OpenSearch Service. Using OpenSearch Dashboards, you can continuously monitor the performance and health of your GraphQL operations. For example, you can create dashboards to visualize the P90 latency of your GraphQL requests and drill down into the P90 latencies of each resolver.

When using OpenSearch Service, use “cwl*” as the filter pattern to search OpenSearch indexes. OpenSearch Service indexes the logs streamed from CloudWatch Logs with a prefix of “cwl-”. To differentiate AWS AppSync API logs from other CloudWatch logs sent to OpenSearch Service, we recommend adding an additional filter expression of graphQLAPIID.keyword=YourGraphQLAPIID to your search.

Log format migration

Log events that AWS AppSync generates on or after May 8, 2019 are formatted as fully structured JSON. To analyze GraphQL requests prior to May 8, 2019, you can migrate older logs to fully structured JSON using a script available in the GitHub Sample. If you need to use the log format prior to May 8, 2019, create a support ticket with the following settings: set Type to Account Management and then set Category to General Account Question.

You can also use metric filters in CloudWatch to turn log data into numerical CloudWatch metrics, so that you can graph or set an alarm on them.