# Configure Amazon S3 object name format When Firehose delivers data to Amazon S3, S3 object key name follows the format **, where the suffix has the format *-------- * begins with 1 and increases by 1 for every configuration change of Firehose stream. You can change Firehose stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption). You can do so by using the Firehose console or the [UpdateDestination](https://docs.aws.amazon.com/firehose/latest/APIReference/API_UpdateDestination.html) API operation. For **, Firehose adds a default time prefix in the format `YYYY/MM/dd/HH`. This prefix creates a logical hierarchy in the bucket, where each forward slash (/) creates a level in the hierarchy. You can modify this structure by specifying a custom prefix that includes expressions that are evaluated at runtime. For information about how to specify a custom prefix, see [Custom Prefixes for Amazon Simple Storage Service Objects](https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html). By default, the time zone used for time prefix and suffix is in UTC, but you can change it to a time zone that you prefer. For example, to use Japan Standard Time instead of UTC, you can configure the time zone to Asia/Tokyo in the AWS Management Console or in [API parameter setting (CustomTimeZone)](https://docs.aws.amazon.com/firehose/latest/APIReference/API_ExtendedS3DestinationConfiguration.html). The following list contains time zones that Firehose supports for S3 prefix configuration. ## Supported time zones Following is a list of time zones that Firehose supports for S3 prefix configuration. ------ #### [ Africa ] ``` Africa/Abidjan Africa/Accra Africa/Addis_Ababa Africa/Algiers Africa/Asmera Africa/Bangui Africa/Banjul Africa/Bissau Africa/Blantyre Africa/Bujumbura Africa/Cairo Africa/Casablanca Africa/Conakry Africa/Dakar Africa/Dar_es_Salaam Africa/Djibouti Africa/Douala Africa/Freetown Africa/Gaborone Africa/Harare Africa/Johannesburg Africa/Kampala Africa/Khartoum Africa/Kigali Africa/Kinshasa Africa/Lagos Africa/Libreville Africa/Lome Africa/Luanda Africa/Lubumbashi Africa/Lusaka Africa/Malabo Africa/Maputo Africa/Maseru Africa/Mbabane Africa/Mogadishu Africa/Monrovia Africa/Nairobi Africa/Ndjamena Africa/Niamey Africa/Nouakchott Africa/Ouagadougou Africa/Porto-Novo Africa/Sao_Tome Africa/Timbuktu Africa/Tripoli Africa/Tunis Africa/Windhoek ``` ------ #### [ America ] ``` America/Adak America/Anchorage America/Anguilla America/Antigua America/Aruba America/Asuncion America/Barbados America/Belize America/Bogota America/Buenos_Aires America/Caracas America/Cayenne America/Cayman America/Chicago America/Costa_Rica America/Cuiaba America/Curacao America/Dawson_Creek America/Denver America/Dominica America/Edmonton America/El_Salvador America/Fortaleza America/Godthab America/Grand_Turk America/Grenada America/Guadeloupe America/Guatemala America/Guayaquil America/Guyana America/Halifax America/Havana America/Indianapolis America/Jamaica America/La_Paz America/Lima America/Los_Angeles America/Managua America/Manaus America/Martinique America/Mazatlan America/Mexico_City America/Miquelon America/Montevideo America/Montreal America/Montserrat America/Nassau America/New_York America/Noronha America/Panama America/Paramaribo America/Phoenix America/Port_of_Spain America/Port-au-Prince America/Porto_Acre America/Puerto_Rico America/Regina America/Rio_Branco America/Santiago America/Santo_Domingo America/Sao_Paulo America/Scoresbysund America/St_Johns America/St_Kitts America/St_Lucia America/St_Thomas America/St_Vincent America/Tegucigalpa America/Thule America/Tijuana America/Tortola America/Vancouver America/Winnipeg ``` ------ #### [ Antarctica ] ``` Antarctica/Casey Antarctica/DumontDUrville Antarctica/Mawson Antarctica/McMurdo Antarctica/Palmer ``` ------ #### [ Asia ] ``` Asia/Aden Asia/Almaty Asia/Amman Asia/Anadyr Asia/Aqtau Asia/Aqtobe Asia/Ashgabat Asia/Ashkhabad Asia/Baghdad Asia/Bahrain Asia/Baku Asia/Bangkok Asia/Beirut Asia/Bishkek Asia/Brunei Asia/Calcutta Asia/Colombo Asia/Dacca Asia/Damascus Asia/Dhaka Asia/Dubai Asia/Dushanbe Asia/Hong_Kong Asia/Irkutsk Asia/Jakarta Asia/Jayapura Asia/Jerusalem Asia/Kabul Asia/Kamchatka Asia/Karachi Asia/Katmandu Asia/Krasnoyarsk Asia/Kuala_Lumpur Asia/Kuwait Asia/Macao Asia/Magadan Asia/Manila Asia/Muscat Asia/Nicosia Asia/Novosibirsk Asia/Phnom_Penh Asia/Pyongyang Asia/Qatar Asia/Rangoon Asia/Riyadh Asia/Saigon Asia/Seoul Asia/Shanghai Asia/Singapore Asia/Taipei Asia/Tashkent Asia/Tbilisi Asia/Tehran Asia/Thimbu Asia/Thimphu Asia/Tokyo Asia/Ujung_Pandang Asia/Ulaanbaatar Asia/Ulan_Bator Asia/Vientiane Asia/Vladivostok Asia/Yakutsk Asia/Yekaterinburg Asia/Yerevan ``` ------ #### [ Atlantic ] ``` Atlantic/Azores Atlantic/Bermuda Atlantic/Canary Atlantic/Cape_Verde Atlantic/Faeroe Atlantic/Jan_Mayen Atlantic/Reykjavik Atlantic/South_Georgia Atlantic/St_Helena Atlantic/Stanley ``` ------ #### [ Australia ] ``` Australia/Adelaide Australia/Brisbane Australia/Broken_Hill Australia/Darwin Australia/Hobart Australia/Lord_Howe Australia/Perth Australia/Sydney ``` ------ #### [ Europe ] ``` Europe/Amsterdam Europe/Andorra Europe/Athens Europe/Belgrade Europe/Berlin Europe/Brussels Europe/Bucharest Europe/Budapest Europe/Chisinau Europe/Copenhagen Europe/Dublin Europe/Gibraltar Europe/Helsinki Europe/Istanbul Europe/Kaliningrad Europe/Kiev Europe/Lisbon Europe/London Europe/Luxembourg Europe/Madrid Europe/Malta Europe/Minsk Europe/Monaco Europe/Moscow Europe/Oslo Europe/Paris Europe/Prague Europe/Riga Europe/Rome Europe/Samara Europe/Simferopol Europe/Sofia Europe/Stockholm Europe/Tallinn Europe/Tirane Europe/Vaduz Europe/Vienna Europe/Vilnius Europe/Warsaw Europe/Zurich ``` ------ #### [ Indian ] ``` Indian/Antananarivo Indian/Chagos Indian/Christmas Indian/Cocos Indian/Comoro Indian/Kerguelen Indian/Mahe Indian/Maldives Indian/Mauritius Indian/Mayotte Indian/Reunion ``` ------ #### [ Pacific ] ``` Pacific/Apia Pacific/Auckland Pacific/Chatham Pacific/Easter Pacific/Efate Pacific/Enderbury Pacific/Fakaofo Pacific/Fiji Pacific/Funafuti Pacific/Galapagos Pacific/Gambier Pacific/Guadalcanal Pacific/Guam Pacific/Honolulu Pacific/Kiritimati Pacific/Kosrae Pacific/Majuro Pacific/Marquesas Pacific/Nauru Pacific/Niue Pacific/Norfolk Pacific/Noumea Pacific/Pago_Pago Pacific/Palau Pacific/Pitcairn Pacific/Ponape Pacific/Port_Moresby Pacific/Rarotonga Pacific/Saipan Pacific/Tahiti Pacific/Tarawa Pacific/Tongatapu Pacific/Truk Pacific/Wake Pacific/Wallis ``` ------ You cannot change the suffix field except **. When you enable data format conversion or compression, Firehose will append a file extension based on the configuration. The following table explains the default file extension appended by Firehose: | Configuration | File extension | | --- | --- | | Data Format Conversion: Parquet | .parquet | | Data Format Conversion: ORC | .orc | | Compression: Gzip | .gz | | Compression: Zip | .zip | | Compression: Snappy | .snappy | | Compression: Hadoop-Snappy | .hsnappy | You can also specify a file extension that you prefer in the Firehose console or API. File extension must start with a period (.) and can contain allowed characters: 0-9a-z\$1-\$1.\$1‘(). File extension cannot exceed 128 characters. **Note** When you specify a file extension, it will override the default file extension that Firehose adds when [data format conversion](https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html) or compression is enabled. # Understand custom prefixes for Amazon S3 objects Objects delivered to Amazon S3 follow the [name format](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-namekey) of . You can specify your custom prefix that includes expressions that are evaluated at runtime. Custom prefix you specify will override the default prefix of `yyyy/MM/dd/HH`. You can use expressions of the following forms in your custom prefix: `!{namespace:value}`, where `namespace` can be one of the following, as explained in the following sections. + `firehose` + `timestamp` + `partitionKeyFromQuery` + `partitionKeyFromLambda` If a prefix ends with a slash, it appears as a folder in the Amazon S3 bucket. For more information, see [Amazon S3 Object Name Format](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-name) in the *Amazon Data FirehoseDeveloper Guide*. ## `timestamp` namespace Valid values for this namespace are strings that are valid [Java DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) strings. As an example, in the year 2018, the expression `!{timestamp:yyyy}` evaluates to `2018`. When evaluating timestamps, Firehose uses the approximate arrival timestamp of the oldest record that's contained in the Amazon S3 object being written. By default, timestamp is in UTC. But, you can specify a time zone that you prefer. For example, you can configure the time zone to Asia/Tokyo in the AWS Management Console or in API parameter setting ([CustomTimeZone](https://docs.aws.amazon.com/firehose/latest/APIReference/API_ExtendedS3DestinationConfiguration.html)) if you want to use Japan Standard Time instead of UTC. To see the list of supported time zones, see [Amazon S3 Object Name Format](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-name). If you use the `timestamp` namespace more than once in the same prefix expression, every instance evaluates to the same instant in time. ## `firehose` namespace There are two values that you can use with this namespace: `error-output-type` and `random-string`. The following table explains how to use them. **The `firehose` namespace values** | Conversion | Description | Example input | Example output | Notes | | --- | --- | --- | --- | --- | | error-output-type | Evaluates to one of the following strings, depending on the configuration of your Firehose stream, and the reason of failure: \$1processing-failed, AmazonOpenSearchService-failed, splunk-failed, format-conversion-failed, http-endpoint-failed\$1.If you use it more than once in the same expression, every instance evaluates to the same error string.. | myPrefix/result=\$1\$1firehose:error-output-type\$1/\$1\$1timestamp:yyyy/MM/dd\$1 | myPrefix/result=processing-failed/2018/08/03 | The error-output-type value can only be used in the ErrorOutputPrefix field. | | random-string | Evaluates to a random string of 11 characters. If you use it more than once in the same expression, every instance evaluates to a new random string. | myPrefix/\$1\$1firehose:random-string\$1/ | myPrefix/046b6c7f-0b/ | You can use it with both prefix types.You can place it at the beginning of the format string to get a randomized prefix, which is sometimes necessary for attaining extremely high throughput with Amazon S3. | ## `partitionKeyFromLambda` and `partitionKeyFromQuery` namespaces For [dynamic partitioning](dynamic-partitioning.md), you must use the following expression format in your S3 bucket prefix: `!{namespace:value}`, where namespace can be either `partitionKeyFromQuery` or `partitionKeyFromLambda`, or both. If you are using inline parsing to create the partitioning keys for your source data, you must specify an S3 bucket prefix value that consists of expressions specified in the following format: `"partitionKeyFromQuery:keyID"`. If you are using an AWS Lambda function to create partitioning keys for your source data, you must specify an S3 bucket prefix value that consists of expressions specified in the following format: `"partitionKeyFromLambda:keyID"`. For more information, see the "Choose Amazon S3 for Your Destination" in [Creating an Amazon Firehose stream](basic-create.md#basic-create.title). ## Semantic rules The following rules apply to `Prefix` and `ErrorOutputPrefix` expressions. + For the `timestamp` namespace, any character that isn't in single quotes is evaluated. In other words, any string escaped with single quotes in the value field is taken literally. + If you specify a prefix that doesn't contain a timestamp namespace expression, Firehose appends the expression `!{timestamp:yyyy/MM/dd/HH/}`to the value in the `Prefix` field. + The sequence `!{` can only appear in `!{namespace:value}` expressions. + `ErrorOutputPrefix` can be null only if `Prefix` contains no expressions. In this case, `Prefix` evaluates to `yyyy/MM/DDD/HH/` and `ErrorOutputPrefix` evaluates to `yyyy/MM/DDD/HH/`. `DDD` represents the day of the year. + If you specify an expression for `ErrorOutputPrefix`, you must include at least one instance of `!{firehose:error-output-type}`. + `Prefix` can't contain `!{firehose:error-output-type}`. + Neither `Prefix` nor `ErrorOutputPrefix` can be greater than 512 characters after they're evaluated. + If the destination is Amazon Redshift, `Prefix` must not contain expressions and `ErrorOutputPrefix` must be null. + When the destination is Amazon OpenSearch Service or Splunk, and no `ErrorOutputPrefix` is specified, Firehose uses the `Prefix` field for failed records. + When the destination is Amazon S3, the `Prefix` and `ErrorOutputPrefix` in the Amazon S3 destination configuration are used for successful records and failed records, respectively. If you use the AWS CLI or the API, you can use `ExtendedS3DestinationConfiguration` to specify an Amazon S3 *backup* configuration with its own `Prefix` and `ErrorOutputPrefix`. + When you use the AWS Management Console and set the destination to Amazon S3, Firehose uses the `Prefix` and `ErrorOutputPrefix` in the destination configuration for successful records and failed records, respectively. If you specify a prefix using expressions, you must specify the error prefix including `!{firehose:error-output-type}`. + When you use `ExtendedS3DestinationConfiguration` with the AWS CLI, the API, or CloudFormation, if you specify a `S3BackupConfiguration`, Firehose doesn't provide a default `ErrorOutputPrefix`. + You cannot use `partitionKeyFromLambda` and `partitionKeyFromQuery` namespaces when creating ErrorOutputPrefix expressions. ## Example prefixes **`Prefix` and `ErrorOutputPrefix` examples** | Input | Evaluated prefix (at 10:30 AM UTC on Aug 27, 2018) | | --- | --- | | `Prefix`: Unspecified `ErrorOutputPrefix`: `myFirehoseFailures/!{firehose:error-output-type}/` | `Prefix`: `2018/08/27/10` `ErrorOutputPrefix`: `myFirehoseFailures/processing-failed/` | | `Prefix`: `!{timestamp:yyyy/MM/dd}` `ErrorOutputPrefix`: Unspecified | Invalid input: ErrorOutputPrefix can't be null when Prefix contains expressions | | `Prefix`: `myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}` `ErrorOutputPrefix`: `myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}` | `Prefix`: `myFirehose/DeliveredYear=2018/anyMonth/rand=5abf82daaa5` `ErrorOutputPrefix`: `myFirehoseFailures/processing-failed/2018/anyMonth/10` | | `Prefix`: `myPrefix/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/` `ErrorOutputPrefix`: `myErrorPrefix/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/!{firehose:error-output-type}` | `Prefix`: `myPrefix/year=2018/month=07/day=06/hour=23/` `ErrorOutputPrefix`: `myErrorPrefix/year=2018/month=07/day=06/hour=23/processing-failed` | | `Prefix`: `myFirehosePrefix/` `ErrorOutputPrefix`: Unspecified | `Prefix`: `myFirehosePrefix/2018/08/27/` `ErrorOutputPrefix`: `myFirehosePrefix/processing-failed/2018/08/27/` |