The following table shows how Firehose handles errors during data decompression and processing, including delivering records to an error S3 bucket, logging errors, and emitting metrics. It also explains the error message returned for unauthorized data put operations.
| Issue | Solution |
|---|---|
| What happens to the source data in case of an error during decompression? |
If Amazon Data Firehose is not able to decompress the record, the record is
delivered as is (in compressed format) to error S3 bucket you
specified during Firehose stream creation time. Along with the record,
the delivered object also includes error code and error message and
these objects will be delivered to an S3 bucket prefix called
|
| What happens to the source data in case of an error in the processing pipeline after successful decompression? |
If Amazon Data Firehose errors out in the processing steps after decompression like Dynamic Partitioning and Data Format Conversion, the record is delivered in compressed format to the error S3 bucket you specified during Firehose stream creation time. Along with the record, the delivered object also includes error code and error message. |
| How are you informed in case of an error or an exception? |
In case of an error or an exception during decompression, if you configure CloudWatch Logs, Firehose will log error messages into CloudWatch Logs. Additionally, Firehose sends metrics to CloudWatch metrics that you can monitor. You can also optionally create alarms based on metrics emitted by Firehose. |
What happens when put operations don't come from CloudWatch Logs? |
When customer puts do not come from CloudWatch Logs, then the
following error message is returned:
|
| What metrics does Firehose emit for the decompression feature? | Firehose emits metrics for decompression of every record. You should select the period (1 min), statistic (sum), date range to get the number of DecompressedRecords failed or succeeded or DecompressedBytes failed or succeeded. For more information, see CloudWatch Logs Decompression Metrics. |
