Observability for the Custom Event Bus: metrics, logs, and CloudTrail

Three sources tell you what a bus and its subscribers are doing. Amazon CloudWatch metrics in the AWS/EventsV2 namespace are on for every subscriber; to alarm when deliveries fail, alarm on OnFailureDestinationDelivered or EventsDropped, or on EventDeliveryAttempts minus EventsDelivered with metric math. Subscriber logs record each delivery attempt as a JSON record; they are off until you set LogConfiguration.Level on the subscriber. AWS CloudTrail records the API calls that create and change buses, subscribers, and event sources.

Metrics

Subscriber metrics carry two dimensions, EventBus and Subscriber, each holding the resource ARN. When the subscriber belongs to a different account from the bus, EventBridge also emits the metric with EventBus and SubscriberAccount so that the bus owner can see consumption per account. The following table lists the metrics.

Your account's bus count is also reported in the AWS/Usage namespace as ResourceCount with Service EventBridge and a Resource value that begins with EventsV2/, so you can alarm as you approach the bus quota. For the quotas, see Custom Event Bus quotas.

Publish metrics

Every PutEvents and PutRawEvents call also produces metrics in the AWS/EventsV2 namespace, so you can alarm on throttled or failed publish calls for a bus, for example on PublishEventsApproximateThrottledCallCount with the EventBus dimension set to orders. The bus owner's account receives every publish metric. An account that publishes to a bus it does not own also receives them, in its own account, for its own calls.

The dimensions depend on the metric and on who published.

Subscriber logs

A subscriber can record what happened to each event it tried to deliver. Logging is per subscriber and off when you create one, so a new subscriber records nothing until you set LogConfiguration. Each record is a JSON document with a message_type that says what it describes.

Log delivery is best effort. A record that cannot be written is counted in the SubscriberLogRecordsDropped metric rather than retried indefinitely. When the bus is encrypted with a customer managed key, EventBridge encrypts the payload fields of each record under that key before it leaves EventBridge, so a destination account that cannot use the key sees the record without them. A replayed event's record carries details.delivery_type REPLAY; a live event's carries LIVE.

The two settings that produce a record

Two independent settings must both exist before you can read a single record.

Make both settings before you start diagnosing a delivery problem rather than after. Only subscribers have a log configuration. Buses and event sources have none, so you turn logging on one subscriber at a time. LogConfiguration has two members.

You can set LogConfiguration when you create a subscriber, or later with UpdateSubscriber. An update takes effect without recreating the subscriber. Neither the create response nor the update response echoes the field back, so confirm the stored value with DescribeSubscriber, which does return it.

Turn on logging for one subscriber

The following steps record every delivery attempt for one existing subscriber and deliver the records to a CloudWatch Logs log group in the same account. The first command is an EventBridge call, and the rest are CloudWatch Logs calls. Start by raising the subscriber's log level. INFO records successes as well as failures, which is what tells you whether an event was delivered at all.

aws eventsv2 update-subscriber \
    --subscriber-arn arn:aws:events:us-east-1:111122223333:subscriber/large-orders/EXAMPLE1234567890abcdef \
    --log-configuration '{ "Level": "INFO", "IncludePayload": "FULL" }'

Next, create the destination log group. The name must begin with /aws/vendedlogs/. CloudWatch Logs manages the delivery resource policy for you only under that prefix; for a log group outside it you must manage that policy yourself.

aws logs create-log-group \
    --log-group-name /aws/vendedlogs/large-orders-delivery

Create a delivery source. Its --resource-arn is the subscriber ARN, which is what makes the source produce that subscriber's records. Set --log-type to match the level you configured: INFO_LOGS for Level INFO, or ERROR_LOGS for Level ERROR.

aws logs put-delivery-source \
    --name large-orders-source \
    --resource-arn arn:aws:events:us-east-1:111122223333:subscriber/large-orders/EXAMPLE1234567890abcdef \
    --log-type INFO_LOGS

Create a delivery destination naming the log group. The response contains the destination ARN, which the next command needs.

aws logs put-delivery-destination \
    --name large-orders-destination \
    --delivery-destination-configuration destinationResourceArn=arn:aws:logs:us-east-1:111122223333:log-group:/aws/vendedlogs/large-orders-delivery

Finally, pair the source with the destination. Use the destination ARN from the previous response.

aws logs create-delivery \
    --delivery-source-name large-orders-source \
    --delivery-destination-arn destination-arn

Records now appear in the log group. They arrive later than the delivery itself, because the log pipeline batches them, so allow for that lag when you read a log group just after publishing. Three properties of this wiring decide how far it stretches. A destination is a log group, an Amazon S3 bucket, or an Amazon Data Firehose stream, and the commands above are the same for each: only destinationResourceArn changes. One delivery pairs exactly one source with exactly one destination, so sending a subscriber's records to a second destination takes a second delivery. The destination can be in a different account from the subscriber, and in that case the destination account must call PutDeliveryDestinationPolicy on the destination to allow the delivery, which is how a central account collects records for subscribers it does not own.

Reading a delivery record

EventBridge emits one record per delivery attempt per event. Four fields are at the top level. message_type is EVENT_DELIVERY_ATTEMPT for a delivery attempt; the other types are listed at the start of this section. resource_arn is the ARN of the subscriber that made the attempt, which is how you separate subscribers that share one log group. log_level is the level of the record itself: a successful attempt is INFO, and a failed attempt is ERROR, which is why Level ERROR still records failures. details is an object holding the delivery state, and it holds the fields you build queries and alarms on.

To diagnose a delivery that never arrived, read the subscriber's records in this order. outcome says whether EventBridge reached the target at all. attempt_count says whether it is still retrying, because an event with no terminal_kind record has not finished. terminal_kind says how it ended, and separates an event that dead-lettered from one that was dropped. target_arn, target_properties, and target_input say what was sent and where, which is where a transformer or target parameter problem shows up. A failure is visible in the record as soon as the attempt fails, so you do not have to wait for the retries to run out. For the full procedure, see Troubleshooting: you published and nothing arrived at the target.

Payloads in log records

IncludePayload controls two fields and no others: details.target_input and details.event_detail. Every other field is emitted whichever value you choose, so a subscriber that keeps payloads out of its records still reports the outcome, the target, the attempt count, and the latencies. FULL embeds the payload in every record, including the records for successful deliveries. ON_ERROR_ONLY, the default, embeds it in failure records only, and omits both fields from a record for a successful delivery.

API calls in AWS CloudTrail

Subscriber logs record deliveries, not the calls you make to configure them. AWS CloudTrail records the management API calls, so a change to a bus, a subscriber, or an event source is auditable from your trail rather than from a log group. For how EventBridge integrates with CloudTrail, see Logging Amazon EventBridge API calls using AWS CloudTrail.