# Understand data delivery in Amazon Data Firehose When you send data to your Firehose stream, it's automatically delivered to your chosen destination. The following table explains data delivery to different destinations. | Destination | Details | | --- | --- | | Amazon S3 | For data delivery to Amazon S3, Firehose concatenates multiple incoming records based on the buffering configuration of your Firehose stream. It then delivers the records to Amazon S3 as an Amazon S3 object. By default, Firehose concatenates data without any delimiters. If you want to have new line delimiters between records, you can add new line delimiters by enabling the feature in the [Firehose console configuration](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-s3) or [API parameter](https://docs.aws.amazon.com/firehose/latest/APIReference/API_Processor.html). Data delivery between Firehose and Amazon S3 destination is encrypted with TLS (HTTPS). | | Amazon Redshift | For data delivery to Amazon Redshift, Firehose first delivers incoming data to your S3 bucket in the format described earlier. Firehose then issues an Amazon Redshift **COPY** command to load the data from your S3 bucket to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup. Ensure that after Amazon Data Firehose concatenates multiple incoming records to an Amazon S3 object, the Amazon S3 object can be copied to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup. For more information, see [Amazon Redshift COPY Command Data Format Parameters](https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-data-format.html). | | OpenSearch Service and OpenSearch Serverless | For data delivery to OpenSearch Service and OpenSearch Serverless, Amazon Data Firehose buffers incoming records based on the buffering configuration of your Firehose stream. It then generates an OpenSearch Service or OpenSearch Serverless bulk request to index multiple records to your OpenSearch Service cluster or OpenSearch Serverless collection. Make sure that your record is UTF-8 encoded and flattened to a single-line JSON object before you send it to Amazon Data Firehose. Also, the rest.action.multi.allow\$1explicit\$1index option for your OpenSearch Service cluster must be set to true (default) to take bulk requests with an explicit index that is set per record. For more information, see [OpenSearch Service Configure Advanced Options](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-advanced-options) in the Amazon OpenSearch Service Developer Guide. | | Splunk | For data delivery to Splunk, Amazon Data Firehose concatenates the bytes that you send. If you want delimiters in your data, such as a new line character, you must insert them yourself. Make sure that Splunk is configured to parse any such delimiters. To redrive the data that was delivered to S3 error bucket (S3 backup) back to Splunk, follow the steps mentioned in the [Splunk documentation](https://www.splunk.com/en_us/blog/tips-and-tricks/aws-technical-add-on-simplifying-error-data-re-ingestion.html). | | HTTP endpoint | For data delivery to an HTTP endpoint owned by a supported third-party service provider, you can use the integrated Amazon Lambda service to create a function to transform the incoming record(s) to the format that matches the format the service provider's integration is expecting. Contact the third-party service provider whose HTTP endpoint you've chosen for your destination to learn more about their accepted record format. | | Snowflake | For data delivery to Snowflake, Amazon Data Firehose internally buffers data for one second and uses Snowflake streaming API operations to insert data to Snowflake. By default, records that you insert are flushed and committed to the Snowflake table every second. After you make the insert call, Firehose emits a CloudWatch metric that measures how long it took for the data to be committed to Snowflake. Firehose currently supports only single JSON item as record payload and doesn’t support JSON arrays. Make sure that your input payload is a valid JSON object and is well formed without any extra double quotes, quotes, or escape characters. | Each Firehose destination has its own data delivery frequency. For more information, see [Configure buffering hints](create-configure-backup.md#buffering-hints). **Duplicate records** Amazon Data Firehose uses at-least-once semantics for data delivery. In some circumstances, such as when data delivery times out, delivery retries by Amazon Data Firehose might introduce duplicates if the original data-delivery request eventually goes through. This applies to all destination types that Amazon Data Firehose supports, except for Amazon S3 destinations, Apache Iceberg Tables, and Snowflake destinations. **Topics** + [ # Understand delivery across AWS accounts and regions ](across.md) + [ # Understand HTTP endpoint delivery request and response specifications ](httpdeliveryrequestresponse.md) + [ # Handle data delivery failures ](retry.md) + [ # Configure Amazon S3 object name format ](s3-object-name.md) + [ # Configure index rotation for OpenSearch Service ](es-index-rotation.md) + [ # Pause and resume data delivery ](pause-restart-stream.md) # Understand delivery across AWS accounts and regions Amazon Data Firehose supports data delivery to HTTP endpoint destinations across AWS accounts. The Firehose stream and the HTTP endpoint that you choose as your destination can belong to different AWS accounts. Amazon Data Firehose also supports data delivery to HTTP endpoint destinations across AWS regions. You can deliver data from a Firehose stream in one AWS region to an HTTP endpoint in another AWS region. You can also delivery data from a Firehose stream to an HTTP endpoint destination outside of AWS regions, for example to your own on-premises server by setting the HTTP endpoint URL to your desired destination. For these scenarios, additional data transfer charges are added to your delivery costs. For more information, see the [Data Transfer](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer) section in the "On-Demand Pricing" page. # Understand HTTP endpoint delivery request and response specifications For Amazon Data Firehose to successfully deliver data to custom HTTP endpoints, these endpoints must accept requests and send responses using certain Amazon Data Firehose request and response formats. This section describes the format specifications of the HTTP requests that the Amazon Data Firehose service sends to custom HTTP endpoints, as well as the format specifications of the HTTP responses that the Amazon Data Firehose service expects. HTTP endpoints have 3 minutes to respond to a request before Amazon Data Firehose times out that request. Amazon Data Firehose treats responses that do not adhere to the proper format as delivery failures. ## Request format **Path and URL Parameters** These are configured directly by you as part of a single URL field. Amazon Data Firehose sends them as configured without modification. Only https destinations are supported. URL restrictions are applied during delivery-stream configuration. Currently, only port 443 is supported for HTTP endpoint data delivery. **HTTP Headers - X-Amz-Firehose-Protocol-Version** This header is used to indicate the version of the request/response formats. Currently the only version is 1.0. **HTTP Headers - X-Amz-Firehose-Request-Id** The value of this header is an opaque GUID that can be used for debugging and deduplication purposes. Endpoint implementations should log the value of this header if possible, for both successful and unsuccessful requests. The request ID is kept the same between multiple attempts of the same request. **HTTP Headers - Content-Type** The value of the Content-Type header is always `application/json`. **HTTP Headers - Content-Encoding** A Firehose stream can be configured to use GZIP to compress the body when sending requests. When this compression is enabled, the value of the Content-Encoding header is set to gzip, as per standard practice. If compression is not enabled, the Content-Encoding header is absent altogether. **HTTP Headers - Content-Length** This is used in the standard way. **HTTP Headers - X-Amz-Firehose-Source-Arn:** The ARN of the Firehose stream represented in ASCII string format. The ARN encodes region, AWS account ID and the stream name. For example, `arn:aws:firehose:us-east-1:123456789:deliverystream/testStream`. **HTTP Headers - X-Amz-Firehose-Access-Key** This header carries an API key or other credentials. You have the ability to create or update the API-key (aka authorization token) when creating or updating your delivery-stream. Amazon Data Firehose restricts the size of the access key to 4096 bytes. Amazon Data Firehose does not attempt to interpret this key in any way. The configured key is copied verbatim into the value of this header. However, if you use Secrets Manager to configure the key, then the secret must follow a specific JSON object format: `{"api_key": "..."}`. The contents can be arbitrary and can potentially represent a JWT token or an ACCESS\$1KEY. If an endpoint requires multi-field credentials (for example, username and password), the values of all of the fields should be stored together within a single access-key in a format that the endpoint understands (JSON or CSV). This field can be base-64 encoded if the original contents are binary. Amazon Data Firehose does not modify and/or encode the configured value and uses the contents as is. **HTTP Headers - X-Amz-Firehose-Common-Attributes** This header carries the common attributes (metadata) that pertain to the entire request, and/or to all records within the request. These are configured directly by you when creating a Firehose stream. The value of this attribute is encoded as a JSON object with the following schema: ``` "$schema": http://json-schema.org/draft-07/schema# properties: commonAttributes: type: object minProperties: 0 maxProperties: 50 patternProperties: "^.{1,256}$": type: string minLength: 0 maxLength: 1024 ``` Here's an example: ``` "commonAttributes": { "deployment -context": "pre-prod-gamma", "device-types": "" } ``` **Body - Max Size** The maximum body size is configured by you, and can be up to a maximum of 64 MiB, before compression. **Body - Schema** The body carries a single JSON document with the following JSON Schema (written in YAML): ``` "$schema": http://json-schema.org/draft-07/schema# title: FirehoseCustomHttpsEndpointRequest description: > The request body that the Firehose service sends to custom HTTPS endpoints. type: object properties: requestId: description: > Same as the value in the X-Amz-Firehose-Request-Id header, duplicated here for convenience. type: string timestamp: description: > The timestamp (milliseconds since epoch) at which the Firehose server generated this request. type: integer records: description: > The actual records of the Firehose stream, carrying the customer data. type: array minItems: 1 maxItems: 10000 items: type: object properties: data: description: > The data of this record, in Base64. Note that empty records are permitted in Firehose. The maximum allowed size of the data, before Base64 encoding, is 1024000 bytes; the maximum length of this field is therefore 1365336 chars. type: string minLength: 0 maxLength: 1365336 required: - requestId - records ``` Here's an example: ``` { "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f", "timestamp": 1578090901599 "records": [ { "data": "aGVsbG8=" }, { "data": "aGVsbG8gd29ybGQ=" } ] } ``` ## Response format **Default Behavior on Error** If a response fails to conform to the requirements below, the Firehose server treats it as though it had a 500 status code with no body. **Status Code** The HTTP status code MUST be in the 2XX, 4XX or 5XX range. The Amazon Data Firehose server does NOT follow redirects (3XX status codes). Only response code 200 is considered as a successful delivery of the records to HTTP/EP. Response code 413 (size exceeded) is considered as a permanent failure and the record batch is not sent to error bucket if configured. All other response codes are considered as retriable errors and are subjected to back-off retry algorithm explained later. **Headers - Content Type** The only acceptable content type is application/json. **HTTP Headers - Content-Encoding** Content-Encoding MUST NOT be used. The body MUST be uncompressed. **HTTP Headers - Content-Length** The Content-Length header MUST be present if the response has a body. **Body - Max Size** The response body must be 1 MiB or less in size. ``` "$schema": http://json-schema.org/draft-07/schema# title: FirehoseCustomHttpsEndpointResponse description: > The response body that the Firehose service sends to custom HTTPS endpoints. type: object properties: requestId: description: > Must match the requestId in the request. type: string timestamp: description: > The timestamp (milliseconds since epoch) at which the server processed this request. type: integer errorMessage: description: > For failed requests, a message explaining the failure. If a request fails after exhausting all retries, the last Instance of the error message is copied to error output S3 bucket if configured. type: string minLength: 0 maxLength: 8192 required: - requestId - timestamp ``` Here's an example: ``` Failure Case (HTTP Response Code 4xx or 5xx) { "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f", "timestamp": "1578090903599", "errorMessage": "Unable to deliver records due to unknown error." } Success case (HTTP Response Code 200) { "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f", "timestamp": 1578090903599 } ``` **Error Response Handling** In all error cases the Amazon Data Firehose server reattempts delivery of the same batch of records using an exponential back-off algorithm. The retries are backed off using an initial back-off time (1 second) with a jitter factor of (15%) and each subsequent retry is backed off using the formula (initial-backoff-time \$1 (multiplier(2) ^ retry\$1count)) with added jitter. The backoff time is capped by a maximum interval of 2 minutes. For example on the ‘n’-th retry the back off time is = MAX(120, 2^n) \$1 random(0.85, 1.15). The parameters specified in the previous equation are subject to change. Refer to the AWS Firehose documentation for exact initial back off time, max backoff time, multiplier and jitter percentages used in exponential back off algorithm. In each subsequent retry attempt the access key and/or destination to which records are delivered might change based on updated configuration of the Firehose stream. Amazon Data Firehose service uses the same request-id across retries in a best-effort manner. This last feature can be used for deduplication purpose by the HTTP end point server. If the request is still not delivered after the maximum time allowed (based on Firehose stream configuration) the batch of records can optionally be delivered to an error bucket based on stream configuration. ## Examples Example of a CWLog sourced request. ``` { "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f", "timestamp": 1578090901599, "records": [ { "data": { "messageType": "DATA_MESSAGE", "owner": "123456789012", "logGroup": "log_group_name", "logStream": "log_stream_name", "subscriptionFilters": [ "subscription_filter_name" ], "logEvents": [ { "id": "0123456789012345678901234567890123456789012345", "timestamp": 1510109208016, "message": "log message 1" }, { "id": "0123456789012345678901234567890123456789012345", "timestamp": 1510109208017, "message": "log message 2" } ] } } ] } ``` # Handle data delivery failures Each Amazon Data Firehose destination has its own data delivery failure handling. When you setup a Firehose stream, for many destinations such as OpenSearch, Splunk, and HTTP endpoints, you also setup an S3 bucket where data that fails to be delivered can be backed up. For more information about how Firehose backs up data in case of failed deliveries, see the relevant destination sections on this page. For more information about how to grant access to S3 buckets where data that fails to be delivered can be backed up, see [Grant Firehose Access to an Amazon S3 Destination](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#using-iam-s3). When Firehose (a) fails to deliver data to the stream destination, and (b) fails to write data to the backup S3 bucket for failed deliveries, it effectively pauses stream delivery until such time that data can either be delivered to the destination or written to the backup S3 location. ## Amazon S3 Data delivery to your S3 bucket might fail for various reasons. For example, the bucket might not exist anymore, the IAM role that Amazon Data Firehose assumes might not have access to the bucket, the network failed, or similar events. Under these conditions, Amazon Data Firehose keeps retrying for up to 24 hours until the delivery succeeds. The maximum data storage time of Amazon Data Firehose is 24 hours. If data delivery fails for more than 24 hours, your data is lost. Data delivery to your S3 bucket can fail for various reasons, such as: + The bucket no longer exists. + The IAM role assumed by Amazon Data Firehose lacks access to the bucket. + Network issues. + S3 errors, such as HTTP 500s or other API failures. In these cases, Amazon Data Firehose will retry delivery: + **DirectPut sources:** Retries continue for up to 24 hours. + **Kinesis Data Streams or Amazon MSK sources:** Retries continue indefinitely, up to the retention policy defined on the stream. Amazon Data Firehose delivers failed records to a S3 error bucket only when Lambda processing or parquet conversion fails. Other failure scenarios will result in continuous retry attempts to S3 until the retention period is reached. When Firehose successfully delivers records to S3, it creates an S3 object file, and in cases of partial record failures, it automatically retries delivery and updates the same S3 object file with the successfully processed records. ## Amazon Redshift For an Amazon Redshift destination, you can specify a retry duration (0–7200 seconds) when creating a Firehose stream. Data delivery to your Amazon Redshift provisioned cluster or Amazon Redshift Serverless workgroup might fail for several reasons. For example, you might have an incorrect cluster configuration of your Firehose stream, a cluster or workgroup under maintenance, or a network failure. Under these conditions, Amazon Data Firehose retries for the specified time duration and skips that particular batch of Amazon S3 objects. The skipped objects' information is delivered to your S3 bucket as a manifest file in the `errors/` folder, which you can use for manual backfill. For information about how to COPY data manually with manifest files, see [Using a Manifest to Specify Data Files](https://docs.aws.amazon.com/redshift/latest/dg/loading-data-files-using-manifest.html). ## Amazon OpenSearch Service and OpenSearch Serverless For the OpenSearch Service and OpenSearch Serverless destination, you can specify a retry duration (0–7200 seconds) during Firehose stream creation. Data delivery to your OpenSearch Service cluster or OpenSearch Serverless collection might fail for several reasons. For example, you might have an incorrect OpenSearch Service cluster or OpenSearch Serverless collection configuration of your Firehose stream, an OpenSearch Service cluster or OpenSearch Serverless collection under maintenance, a network failure, or similar events. Under these conditions, Amazon Data Firehose retries for the specified time duration and then skips that particular index request. The skipped documents are delivered to your S3 bucket in the `AmazonOpenSearchService_failed/` folder, which you can use for manual backfill. For OpenSearch Service, each document has the following JSON format: ``` { "attemptsMade": "(number of index requests attempted)", "arrivalTimestamp": "(the time when the document was received by Firehose)", "errorCode": "(http error code returned by OpenSearch Service)", "errorMessage": "(error message returned by OpenSearch Service)", "attemptEndingTimestamp": "(the time when Firehose stopped attempting index request)", "esDocumentId": "(intended OpenSearch Service document ID)", "esIndexName": "(intended OpenSearch Service index name)", "esTypeName": "(intended OpenSearch Service type name)", "rawData": "(base64-encoded document data)" } ``` For OpenSearch Serverless, each document has the following JSON format: ``` { "attemptsMade": "(number of index requests attempted)", "arrivalTimestamp": "(the time when the document was received by Firehose)", "errorCode": "(http error code returned by OpenSearch Serverless)", "errorMessage": "(error message returned by OpenSearch Serverless)", "attemptEndingTimestamp": "(the time when Firehose stopped attempting index request)", "osDocumentId": "(intended OpenSearch Serverless document ID)", "osIndexName": "(intended OpenSearch Serverless index name)", "rawData": "(base64-encoded document data)" } ``` ## Splunk When Amazon Data Firehose sends data to Splunk, it waits for an acknowledgment from Splunk. If an error occurs, or the acknowledgment doesn’t arrive within the acknowledgment timeout period, Amazon Data Firehose starts the retry duration counter. It keeps retrying until the retry duration expires. After that, Amazon Data Firehose considers it a data delivery failure and backs up the data to your Amazon S3 bucket. Every time Amazon Data Firehose sends data to Splunk, whether it's the initial attempt or a retry, it restarts the acknowledgement timeout counter. It then waits for an acknowledgement to arrive from Splunk. Even if the retry duration expires, Amazon Data Firehose still waits for the acknowledgment until it receives it or the acknowledgement timeout is reached. If the acknowledgment times out, Amazon Data Firehose checks to determine whether there's time left in the retry counter. If there is time left, it retries again and repeats the logic until it receives an acknowledgment or determines that the retry time has expired. A failure to receive an acknowledgement isn't the only type of data delivery error that can occur. For information about the other types of data delivery errors, see [Splunk Data Delivery Errors](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-logs.html#monitoring-splunk-errors). Any data delivery error triggers the retry logic if your retry duration is greater than 0. The following is an example error record. ``` { "attemptsMade": 0, "arrivalTimestamp": 1506035354675, "errorCode": "Splunk.AckTimeout", "errorMessage": "Did not receive an acknowledgement from HEC before the HEC acknowledgement timeout expired. Despite the acknowledgement timeout, it's possible the data was indexed successfully in Splunk. Amazon Data Firehose backs up in Amazon S3 data for which the acknowledgement timeout expired.", "attemptEndingTimestamp": 13626284715507, "rawData": "MiAyNTE2MjAyNzIyMDkgZW5pLTA1ZjMyMmQ1IDIxOC45Mi4xODguMjE0IDE3Mi4xNi4xLjE2NyAyNTIzMyAxNDMzIDYgMSA0MCAxNTA2MDM0NzM0IDE1MDYwMzQ3OTQgUkVKRUNUIE9LCg==", "EventId": "49577193928114147339600778471082492393164139877200035842.0" } ``` ## HTTP endpoint destination When Amazon Data Firehose sends data to an HTTP endpoint destination, it waits for a response from this destination. If an error occurs, or the response doesn’t arrive within the response timeout period, Amazon Data Firehose starts the retry duration counter. It keeps retrying until the retry duration expires. After that, Amazon Data Firehose considers it a data delivery failure and backs up the data to your Amazon S3 bucket. Every time Amazon Data Firehose sends data to an HTTP endpoint destination, whether it's the initial attempt or a retry, it restarts the response timeout counter. It then waits for a response to arrive from the HTTP endpoint destination. Even if the retry duration expires, Amazon Data Firehose still waits for the response until it receives it or the response timeout is reached. If the response times out, Amazon Data Firehose checks to determine whether there's time left in the retry counter. If there is time left, it retries again and repeats the logic until it receives a response or determines that the retry time has expired. A failure to receive a response isn't the only type of data delivery error that can occur. For information about the other types of data delivery errors, see [HTTP Endpoint Data Delivery Errors](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-logs.html#monitoring-http-errors) The following is an example error record. ``` { "attemptsMade":5, "arrivalTimestamp":1594265943615, "errorCode":"HttpEndpoint.DestinationException", "errorMessage":"Received the following response from the endpoint destination. {"requestId": "109777ac-8f9b-4082-8e8d-b4f12b5fc17b", "timestamp": 1594266081268, "errorMessage": "Unauthorized"}", "attemptEndingTimestamp":1594266081318, "rawData":"c2FtcGxlIHJhdyBkYXRh", "subsequenceNumber":0, "dataId":"49607357361271740811418664280693044274821622880012337186.0" } ``` ## Snowflake For Snowflake destination, when you create a Firehose stream, you can specify an optional retry duration (0-7200 seconds). The default value for retry duration is 60 seconds. Data delivery to your Snowflake table might fail for several reasons like an incorrect Snowflake destination configuration, Snowflake outage, a network failure, etc. The retry policy doesn’t apply to non-retriable errors. For example, if Snowflake rejects your JSON payload because it had an extra column that's missing in the table, Firehose doesn’t attempt to deliver it again. Instead, it creates a back up for all the insert failures due to JSON payload issues to your S3 error bucket. Similarly, if delivery fails due to an incorrect role, table, or database, Firehose doesn’t retry and writes the data to your S3 bucket. Retry duration only applies to failure due to a Snowflake service issue, transient network glitches, etc. Under these conditions, Firehose retries for the specified time duration before delivering them to S3. The failed records are delivered in snowflake-failed/ folder, which you can use for manual backfill. The following is an example JSON for each record that you deliver to S3. ``` { "attemptsMade": 3, "arrivalTimestamp": 1594265943615, "errorCode": "Snowflake.InvalidColumns", "errorMessage": "Snowpipe Streaming does not support columns of type AUTOINCREMENT, IDENTITY, GEO, or columns with a default value or collation", "attemptEndingTimestamp": 1712937865543, "rawData": "c2FtcGxlIHJhdyBkYXRh" } ``` # Configure Amazon S3 object name format When Firehose delivers data to Amazon S3, S3 object key name follows the format **, where the suffix has the format *-------- * begins with 1 and increases by 1 for every configuration change of Firehose stream. You can change Firehose stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption). You can do so by using the Firehose console or the [UpdateDestination](https://docs.aws.amazon.com/firehose/latest/APIReference/API_UpdateDestination.html) API operation. For **, Firehose adds a default time prefix in the format `YYYY/MM/dd/HH`. This prefix creates a logical hierarchy in the bucket, where each forward slash (/) creates a level in the hierarchy. You can modify this structure by specifying a custom prefix that includes expressions that are evaluated at runtime. For information about how to specify a custom prefix, see [Custom Prefixes for Amazon Simple Storage Service Objects](https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html). By default, the time zone used for time prefix and suffix is in UTC, but you can change it to a time zone that you prefer. For example, to use Japan Standard Time instead of UTC, you can configure the time zone to Asia/Tokyo in the AWS Management Console or in [API parameter setting (CustomTimeZone)](https://docs.aws.amazon.com/firehose/latest/APIReference/API_ExtendedS3DestinationConfiguration.html). The following list contains time zones that Firehose supports for S3 prefix configuration. ## Supported time zones Following is a list of time zones that Firehose supports for S3 prefix configuration. ------ #### [ Africa ] ``` Africa/Abidjan Africa/Accra Africa/Addis_Ababa Africa/Algiers Africa/Asmera Africa/Bangui Africa/Banjul Africa/Bissau Africa/Blantyre Africa/Bujumbura Africa/Cairo Africa/Casablanca Africa/Conakry Africa/Dakar Africa/Dar_es_Salaam Africa/Djibouti Africa/Douala Africa/Freetown Africa/Gaborone Africa/Harare Africa/Johannesburg Africa/Kampala Africa/Khartoum Africa/Kigali Africa/Kinshasa Africa/Lagos Africa/Libreville Africa/Lome Africa/Luanda Africa/Lubumbashi Africa/Lusaka Africa/Malabo Africa/Maputo Africa/Maseru Africa/Mbabane Africa/Mogadishu Africa/Monrovia Africa/Nairobi Africa/Ndjamena Africa/Niamey Africa/Nouakchott Africa/Ouagadougou Africa/Porto-Novo Africa/Sao_Tome Africa/Timbuktu Africa/Tripoli Africa/Tunis Africa/Windhoek ``` ------ #### [ America ] ``` America/Adak America/Anchorage America/Anguilla America/Antigua America/Aruba America/Asuncion America/Barbados America/Belize America/Bogota America/Buenos_Aires America/Caracas America/Cayenne America/Cayman America/Chicago America/Costa_Rica America/Cuiaba America/Curacao America/Dawson_Creek America/Denver America/Dominica America/Edmonton America/El_Salvador America/Fortaleza America/Godthab America/Grand_Turk America/Grenada America/Guadeloupe America/Guatemala America/Guayaquil America/Guyana America/Halifax America/Havana America/Indianapolis America/Jamaica America/La_Paz America/Lima America/Los_Angeles America/Managua America/Manaus America/Martinique America/Mazatlan America/Mexico_City America/Miquelon America/Montevideo America/Montreal America/Montserrat America/Nassau America/New_York America/Noronha America/Panama America/Paramaribo America/Phoenix America/Port_of_Spain America/Port-au-Prince America/Porto_Acre America/Puerto_Rico America/Regina America/Rio_Branco America/Santiago America/Santo_Domingo America/Sao_Paulo America/Scoresbysund America/St_Johns America/St_Kitts America/St_Lucia America/St_Thomas America/St_Vincent America/Tegucigalpa America/Thule America/Tijuana America/Tortola America/Vancouver America/Winnipeg ``` ------ #### [ Antarctica ] ``` Antarctica/Casey Antarctica/DumontDUrville Antarctica/Mawson Antarctica/McMurdo Antarctica/Palmer ``` ------ #### [ Asia ] ``` Asia/Aden Asia/Almaty Asia/Amman Asia/Anadyr Asia/Aqtau Asia/Aqtobe Asia/Ashgabat Asia/Ashkhabad Asia/Baghdad Asia/Bahrain Asia/Baku Asia/Bangkok Asia/Beirut Asia/Bishkek Asia/Brunei Asia/Calcutta Asia/Colombo Asia/Dacca Asia/Damascus Asia/Dhaka Asia/Dubai Asia/Dushanbe Asia/Hong_Kong Asia/Irkutsk Asia/Jakarta Asia/Jayapura Asia/Jerusalem Asia/Kabul Asia/Kamchatka Asia/Karachi Asia/Katmandu Asia/Krasnoyarsk Asia/Kuala_Lumpur Asia/Kuwait Asia/Macao Asia/Magadan Asia/Manila Asia/Muscat Asia/Nicosia Asia/Novosibirsk Asia/Phnom_Penh Asia/Pyongyang Asia/Qatar Asia/Rangoon Asia/Riyadh Asia/Saigon Asia/Seoul Asia/Shanghai Asia/Singapore Asia/Taipei Asia/Tashkent Asia/Tbilisi Asia/Tehran Asia/Thimbu Asia/Thimphu Asia/Tokyo Asia/Ujung_Pandang Asia/Ulaanbaatar Asia/Ulan_Bator Asia/Vientiane Asia/Vladivostok Asia/Yakutsk Asia/Yekaterinburg Asia/Yerevan ``` ------ #### [ Atlantic ] ``` Atlantic/Azores Atlantic/Bermuda Atlantic/Canary Atlantic/Cape_Verde Atlantic/Faeroe Atlantic/Jan_Mayen Atlantic/Reykjavik Atlantic/South_Georgia Atlantic/St_Helena Atlantic/Stanley ``` ------ #### [ Australia ] ``` Australia/Adelaide Australia/Brisbane Australia/Broken_Hill Australia/Darwin Australia/Hobart Australia/Lord_Howe Australia/Perth Australia/Sydney ``` ------ #### [ Europe ] ``` Europe/Amsterdam Europe/Andorra Europe/Athens Europe/Belgrade Europe/Berlin Europe/Brussels Europe/Bucharest Europe/Budapest Europe/Chisinau Europe/Copenhagen Europe/Dublin Europe/Gibraltar Europe/Helsinki Europe/Istanbul Europe/Kaliningrad Europe/Kiev Europe/Lisbon Europe/London Europe/Luxembourg Europe/Madrid Europe/Malta Europe/Minsk Europe/Monaco Europe/Moscow Europe/Oslo Europe/Paris Europe/Prague Europe/Riga Europe/Rome Europe/Samara Europe/Simferopol Europe/Sofia Europe/Stockholm Europe/Tallinn Europe/Tirane Europe/Vaduz Europe/Vienna Europe/Vilnius Europe/Warsaw Europe/Zurich ``` ------ #### [ Indian ] ``` Indian/Antananarivo Indian/Chagos Indian/Christmas Indian/Cocos Indian/Comoro Indian/Kerguelen Indian/Mahe Indian/Maldives Indian/Mauritius Indian/Mayotte Indian/Reunion ``` ------ #### [ Pacific ] ``` Pacific/Apia Pacific/Auckland Pacific/Chatham Pacific/Easter Pacific/Efate Pacific/Enderbury Pacific/Fakaofo Pacific/Fiji Pacific/Funafuti Pacific/Galapagos Pacific/Gambier Pacific/Guadalcanal Pacific/Guam Pacific/Honolulu Pacific/Kiritimati Pacific/Kosrae Pacific/Majuro Pacific/Marquesas Pacific/Nauru Pacific/Niue Pacific/Norfolk Pacific/Noumea Pacific/Pago_Pago Pacific/Palau Pacific/Pitcairn Pacific/Ponape Pacific/Port_Moresby Pacific/Rarotonga Pacific/Saipan Pacific/Tahiti Pacific/Tarawa Pacific/Tongatapu Pacific/Truk Pacific/Wake Pacific/Wallis ``` ------ You cannot change the suffix field except **. When you enable data format conversion or compression, Firehose will append a file extension based on the configuration. The following table explains the default file extension appended by Firehose: | Configuration | File extension | | --- | --- | | Data Format Conversion: Parquet | .parquet | | Data Format Conversion: ORC | .orc | | Compression: Gzip | .gz | | Compression: Zip | .zip | | Compression: Snappy | .snappy | | Compression: Hadoop-Snappy | .hsnappy | You can also specify a file extension that you prefer in the Firehose console or API. File extension must start with a period (.) and can contain allowed characters: 0-9a-z\$1-\$1.\$1‘(). File extension cannot exceed 128 characters. **Note** When you specify a file extension, it will override the default file extension that Firehose adds when [data format conversion](https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html) or compression is enabled. # Understand custom prefixes for Amazon S3 objects Objects delivered to Amazon S3 follow the [name format](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-namekey) of . You can specify your custom prefix that includes expressions that are evaluated at runtime. Custom prefix you specify will override the default prefix of `yyyy/MM/dd/HH`. You can use expressions of the following forms in your custom prefix: `!{namespace:value}`, where `namespace` can be one of the following, as explained in the following sections. + `firehose` + `timestamp` + `partitionKeyFromQuery` + `partitionKeyFromLambda` If a prefix ends with a slash, it appears as a folder in the Amazon S3 bucket. For more information, see [Amazon S3 Object Name Format](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-name) in the *Amazon Data FirehoseDeveloper Guide*. ## `timestamp` namespace Valid values for this namespace are strings that are valid [Java DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) strings. As an example, in the year 2018, the expression `!{timestamp:yyyy}` evaluates to `2018`. When evaluating timestamps, Firehose uses the approximate arrival timestamp of the oldest record that's contained in the Amazon S3 object being written. By default, timestamp is in UTC. But, you can specify a time zone that you prefer. For example, you can configure the time zone to Asia/Tokyo in the AWS Management Console or in API parameter setting ([CustomTimeZone](https://docs.aws.amazon.com/firehose/latest/APIReference/API_ExtendedS3DestinationConfiguration.html)) if you want to use Japan Standard Time instead of UTC. To see the list of supported time zones, see [Amazon S3 Object Name Format](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-name). If you use the `timestamp` namespace more than once in the same prefix expression, every instance evaluates to the same instant in time. ## `firehose` namespace There are two values that you can use with this namespace: `error-output-type` and `random-string`. The following table explains how to use them. **The `firehose` namespace values** | Conversion | Description | Example input | Example output | Notes | | --- | --- | --- | --- | --- | | error-output-type | Evaluates to one of the following strings, depending on the configuration of your Firehose stream, and the reason of failure: \$1processing-failed, AmazonOpenSearchService-failed, splunk-failed, format-conversion-failed, http-endpoint-failed\$1.If you use it more than once in the same expression, every instance evaluates to the same error string.. | myPrefix/result=\$1\$1firehose:error-output-type\$1/\$1\$1timestamp:yyyy/MM/dd\$1 | myPrefix/result=processing-failed/2018/08/03 | The error-output-type value can only be used in the ErrorOutputPrefix field. | | random-string | Evaluates to a random string of 11 characters. If you use it more than once in the same expression, every instance evaluates to a new random string. | myPrefix/\$1\$1firehose:random-string\$1/ | myPrefix/046b6c7f-0b/ | You can use it with both prefix types.You can place it at the beginning of the format string to get a randomized prefix, which is sometimes necessary for attaining extremely high throughput with Amazon S3. | ## `partitionKeyFromLambda` and `partitionKeyFromQuery` namespaces For [dynamic partitioning](dynamic-partitioning.md), you must use the following expression format in your S3 bucket prefix: `!{namespace:value}`, where namespace can be either `partitionKeyFromQuery` or `partitionKeyFromLambda`, or both. If you are using inline parsing to create the partitioning keys for your source data, you must specify an S3 bucket prefix value that consists of expressions specified in the following format: `"partitionKeyFromQuery:keyID"`. If you are using an AWS Lambda function to create partitioning keys for your source data, you must specify an S3 bucket prefix value that consists of expressions specified in the following format: `"partitionKeyFromLambda:keyID"`. For more information, see the "Choose Amazon S3 for Your Destination" in [Creating an Amazon Firehose stream](basic-create.md#basic-create.title). ## Semantic rules The following rules apply to `Prefix` and `ErrorOutputPrefix` expressions. + For the `timestamp` namespace, any character that isn't in single quotes is evaluated. In other words, any string escaped with single quotes in the value field is taken literally. + If you specify a prefix that doesn't contain a timestamp namespace expression, Firehose appends the expression `!{timestamp:yyyy/MM/dd/HH/}`to the value in the `Prefix` field. + The sequence `!{` can only appear in `!{namespace:value}` expressions. + `ErrorOutputPrefix` can be null only if `Prefix` contains no expressions. In this case, `Prefix` evaluates to `yyyy/MM/DDD/HH/` and `ErrorOutputPrefix` evaluates to `yyyy/MM/DDD/HH/`. `DDD` represents the day of the year. + If you specify an expression for `ErrorOutputPrefix`, you must include at least one instance of `!{firehose:error-output-type}`. + `Prefix` can't contain `!{firehose:error-output-type}`. + Neither `Prefix` nor `ErrorOutputPrefix` can be greater than 512 characters after they're evaluated. + If the destination is Amazon Redshift, `Prefix` must not contain expressions and `ErrorOutputPrefix` must be null. + When the destination is Amazon OpenSearch Service or Splunk, and no `ErrorOutputPrefix` is specified, Firehose uses the `Prefix` field for failed records. + When the destination is Amazon S3, the `Prefix` and `ErrorOutputPrefix` in the Amazon S3 destination configuration are used for successful records and failed records, respectively. If you use the AWS CLI or the API, you can use `ExtendedS3DestinationConfiguration` to specify an Amazon S3 *backup* configuration with its own `Prefix` and `ErrorOutputPrefix`. + When you use the AWS Management Console and set the destination to Amazon S3, Firehose uses the `Prefix` and `ErrorOutputPrefix` in the destination configuration for successful records and failed records, respectively. If you specify a prefix using expressions, you must specify the error prefix including `!{firehose:error-output-type}`. + When you use `ExtendedS3DestinationConfiguration` with the AWS CLI, the API, or CloudFormation, if you specify a `S3BackupConfiguration`, Firehose doesn't provide a default `ErrorOutputPrefix`. + You cannot use `partitionKeyFromLambda` and `partitionKeyFromQuery` namespaces when creating ErrorOutputPrefix expressions. ## Example prefixes **`Prefix` and `ErrorOutputPrefix` examples** | Input | Evaluated prefix (at 10:30 AM UTC on Aug 27, 2018) | | --- | --- | | `Prefix`: Unspecified `ErrorOutputPrefix`: `myFirehoseFailures/!{firehose:error-output-type}/` | `Prefix`: `2018/08/27/10` `ErrorOutputPrefix`: `myFirehoseFailures/processing-failed/` | | `Prefix`: `!{timestamp:yyyy/MM/dd}` `ErrorOutputPrefix`: Unspecified | Invalid input: ErrorOutputPrefix can't be null when Prefix contains expressions | | `Prefix`: `myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}` `ErrorOutputPrefix`: `myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}` | `Prefix`: `myFirehose/DeliveredYear=2018/anyMonth/rand=5abf82daaa5` `ErrorOutputPrefix`: `myFirehoseFailures/processing-failed/2018/anyMonth/10` | | `Prefix`: `myPrefix/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/` `ErrorOutputPrefix`: `myErrorPrefix/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/!{firehose:error-output-type}` | `Prefix`: `myPrefix/year=2018/month=07/day=06/hour=23/` `ErrorOutputPrefix`: `myErrorPrefix/year=2018/month=07/day=06/hour=23/processing-failed` | | `Prefix`: `myFirehosePrefix/` `ErrorOutputPrefix`: Unspecified | `Prefix`: `myFirehosePrefix/2018/08/27/` `ErrorOutputPrefix`: `myFirehosePrefix/processing-failed/2018/08/27/` | # Configure index rotation for OpenSearch Service For the OpenSearch Service destination, you can specify a time-based index rotation option from one of the following five options: **NoRotation**, **OneHour**, **OneDay**, **OneWeek**, or **OneMonth**. Depending on the rotation option you choose, Amazon Data Firehose appends a portion of the UTC arrival timestamp to your specified index name. It rotates the appended timestamp accordingly. The following example shows the resulting index name in OpenSearch Service for each index rotation option, where the specified index name is **myindex** and the arrival timestamp is `2016-02-25T13:00:00Z`. | RotationPeriod | IndexName | | --- | --- | | NoRotation | myindex | | OneHour | myindex-2016-02-25-13 | | OneDay | myindex-2016-02-25 | | OneWeek | myindex-2016-w08 | | OneMonth | myindex-2016-02 | **Note** With the `OneWeek` option, Data Firehose auto-create indexes using the format of -w (for example, `2020-w33`), where the week number is calculated using UTC time and according to the following US conventions: A week starts on Sunday The first week of the year is the first week that contains a Saturday in this year # Pause and resume data delivery After you setup a Firehose stream, data available in the stream source is continuously delivered to the destination. If you encounter situations where your stream destination is temporarily unavailable (for example, during planned maintenance operations), you may want to temporarily pause data delivery, and resume when the destination becomes available again. **Important** When you use the approach described below to pause and resume a stream, after you resume the stream, you will see that few records get delivered to the error bucket in Amazon S3 while the rest of the stream continues to get delivered to the destination. This is a known limitation of the approach, and it occurs because a small number of records that could not be previously delivered to the destination after multiple retries are tracked as failed. ## Pause a Firehose stream To pause stream delivery in Firehose, first remove permissions for Firehose to write to the S3 backup location for failed deliveries. For example, if you want to pause the Firehose stream with an OpenSearch destination, you can do this by updating permissions. For more information, see [Grant Firehose Access to a Public OpenSearch Service Destination](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#using-iam-es). Remove the `"Effect": "Allow"` permission for the action `s3:PutObject`, and explicitly add a statement that applies `Effect": "Deny"` permission on the action `s3:PutObject` for the S3 bucket used for backing up failed deliveries. Next, turn off the stream destination (for example, turning off the destination OpenSearch domain), or remove permissions for Firehose to write to the destination. To update permissions for other destinations, check the section for your destination in [Controlling Access with Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html). After you complete these two actions, Firehose will stop delivering streams, and you can monitor this using [CloudWatch metrics for Firehose](https://docs.aws.amazon.com/firehose/latest/dev/cloudwatch-metrics.html). **Important** When you pause stream delivery in Firehose, you need to ensure that the source of the stream (for example, in Kinesis Data Streams or in Managed Service for Kafka) is configured to retain data until stream delivery is resumed and the data gets delivered to the destination. If the source is DirectPUT, Firehose will retain data for 24 hours. Data loss could happen if you do not resume the stream and deliver the data before the expiration of data retention period. ## Resume a Firehose stream To resume delivery, first revert the change made earlier to the stream destination by turning on the destination and ensuring that Firehose has permissions to deliver the stream to the destination. Next, revert the changes made earlier to permissions applied to the S3 bucket for backing up failed deliveries. That is, apply `"Effect": "Allow"` permission for the action `s3:PutObject`, and remove `"Effect": "Deny"` permission on the action `s3:PutObject` for the S3 bucket used for backing up failed deliveries. Finally, monitor using [CloudWatch metrics for Firehose](https://docs.aws.amazon.com/firehose/latest/dev/cloudwatch-metrics.html) to confirm that the stream is being delivered to the destination. To view and troubleshoot errors, use [Amazon CloudWatch Logs monitoring for Firehose](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-logs.html).