Configure Amazon S3 object name format - Amazon Data Firehose

Configure Amazon S3 object name format

When Firehose delivers data to Amazon S3, S3 object key name follows the format <evaluated prefix><suffix>, where the suffix has the format <Firehose stream name>-<Firehose stream version>-<year>-<month>-<day>-<hour>-<minute>-<second>-<uuid><file extension> <Firehose stream version> begins with 1 and increases by 1 for every configuration change of Firehose stream. You can change Firehose stream configurations (for example, the name of the S3 bucket, buffering hints, compression, and encryption). You can do so by using the Firehose console or the UpdateDestination API operation.

For <evaluated prefix>, Firehose adds a default time prefix in the format YYYY/MM/dd/HH. This prefix creates a logical hierarchy in the bucket, where each forward slash (/) creates a level in the hierarchy. You can modify this structure by specifying a custom prefix that includes expressions that are evaluated at runtime. For information about how to specify a custom prefix, see Custom Prefixes for Amazon Simple Storage Service Objects.

By default, the time zone used for time prefix and suffix is in UTC, but you can change it to a time zone that you prefer. For example, to use Japan Standard Time instead of UTC, you can configure the time zone to Asia/Tokyo in the AWS Management Console or in API parameter setting (CustomTimeZone). The following list contains time zones that Firehose supports for S3 prefix configuration.

Following is a list of time zones that Firehose supports for S3 prefix configuration.

Africa
Africa/Abidjan Africa/Accra Africa/Addis_Ababa Africa/Algiers Africa/Asmera Africa/Bangui Africa/Banjul Africa/Bissau Africa/Blantyre Africa/Bujumbura Africa/Cairo Africa/Casablanca Africa/Conakry Africa/Dakar Africa/Dar_es_Salaam Africa/Djibouti Africa/Douala Africa/Freetown Africa/Gaborone Africa/Harare Africa/Johannesburg Africa/Kampala Africa/Khartoum Africa/Kigali Africa/Kinshasa Africa/Lagos Africa/Libreville Africa/Lome Africa/Luanda Africa/Lubumbashi Africa/Lusaka Africa/Malabo Africa/Maputo Africa/Maseru Africa/Mbabane Africa/Mogadishu Africa/Monrovia Africa/Nairobi Africa/Ndjamena Africa/Niamey Africa/Nouakchott Africa/Ouagadougou Africa/Porto-Novo Africa/Sao_Tome Africa/Timbuktu Africa/Tripoli Africa/Tunis Africa/Windhoek
America
America/Adak America/Anchorage America/Anguilla America/Antigua America/Aruba America/Asuncion America/Barbados America/Belize America/Bogota America/Buenos_Aires America/Caracas America/Cayenne America/Cayman America/Chicago America/Costa_Rica America/Cuiaba America/Curacao America/Dawson_Creek America/Denver America/Dominica America/Edmonton America/El_Salvador America/Fortaleza America/Godthab America/Grand_Turk America/Grenada America/Guadeloupe America/Guatemala America/Guayaquil America/Guyana America/Halifax America/Havana America/Indianapolis America/Jamaica America/La_Paz America/Lima America/Los_Angeles America/Managua America/Manaus America/Martinique America/Mazatlan America/Mexico_City America/Miquelon America/Montevideo America/Montreal America/Montserrat America/Nassau America/New_York America/Noronha America/Panama America/Paramaribo America/Phoenix America/Port_of_Spain America/Port-au-Prince America/Porto_Acre America/Puerto_Rico America/Regina America/Rio_Branco America/Santiago America/Santo_Domingo America/Sao_Paulo America/Scoresbysund America/St_Johns America/St_Kitts America/St_Lucia America/St_Thomas America/St_Vincent America/Tegucigalpa America/Thule America/Tijuana America/Tortola America/Vancouver America/Winnipeg
Antarctica
Antarctica/Casey Antarctica/DumontDUrville Antarctica/Mawson Antarctica/McMurdo Antarctica/Palmer
Asia
Asia/Aden Asia/Almaty Asia/Amman Asia/Anadyr Asia/Aqtau Asia/Aqtobe Asia/Ashgabat Asia/Ashkhabad Asia/Baghdad Asia/Bahrain Asia/Baku Asia/Bangkok Asia/Beirut Asia/Bishkek Asia/Brunei Asia/Calcutta Asia/Colombo Asia/Dacca Asia/Damascus Asia/Dhaka Asia/Dubai Asia/Dushanbe Asia/Hong_Kong Asia/Irkutsk Asia/Jakarta Asia/Jayapura Asia/Jerusalem Asia/Kabul Asia/Kamchatka Asia/Karachi Asia/Katmandu Asia/Krasnoyarsk Asia/Kuala_Lumpur Asia/Kuwait Asia/Macao Asia/Magadan Asia/Manila Asia/Muscat Asia/Nicosia Asia/Novosibirsk Asia/Phnom_Penh Asia/Pyongyang Asia/Qatar Asia/Rangoon Asia/Riyadh Asia/Saigon Asia/Seoul Asia/Shanghai Asia/Singapore Asia/Taipei Asia/Tashkent Asia/Tbilisi Asia/Tehran Asia/Thimbu Asia/Thimphu Asia/Tokyo Asia/Ujung_Pandang Asia/Ulaanbaatar Asia/Ulan_Bator Asia/Vientiane Asia/Vladivostok Asia/Yakutsk Asia/Yekaterinburg Asia/Yerevan
Atlantic
Atlantic/Azores Atlantic/Bermuda Atlantic/Canary Atlantic/Cape_Verde Atlantic/Faeroe Atlantic/Jan_Mayen Atlantic/Reykjavik Atlantic/South_Georgia Atlantic/St_Helena Atlantic/Stanley
Australia
Australia/Adelaide Australia/Brisbane Australia/Broken_Hill Australia/Darwin Australia/Hobart Australia/Lord_Howe Australia/Perth Australia/Sydney
Europe
Europe/Amsterdam Europe/Andorra Europe/Athens Europe/Belgrade Europe/Berlin Europe/Brussels Europe/Bucharest Europe/Budapest Europe/Chisinau Europe/Copenhagen Europe/Dublin Europe/Gibraltar Europe/Helsinki Europe/Istanbul Europe/Kaliningrad Europe/Kiev Europe/Lisbon Europe/London Europe/Luxembourg Europe/Madrid Europe/Malta Europe/Minsk Europe/Monaco Europe/Moscow Europe/Oslo Europe/Paris Europe/Prague Europe/Riga Europe/Rome Europe/Samara Europe/Simferopol Europe/Sofia Europe/Stockholm Europe/Tallinn Europe/Tirane Europe/Vaduz Europe/Vienna Europe/Vilnius Europe/Warsaw Europe/Zurich
Indian
Indian/Antananarivo Indian/Chagos Indian/Christmas Indian/Cocos Indian/Comoro Indian/Kerguelen Indian/Mahe Indian/Maldives Indian/Mauritius Indian/Mayotte Indian/Reunion
Pacific
Pacific/Apia Pacific/Auckland Pacific/Chatham Pacific/Easter Pacific/Efate Pacific/Enderbury Pacific/Fakaofo Pacific/Fiji Pacific/Funafuti Pacific/Galapagos Pacific/Gambier Pacific/Guadalcanal Pacific/Guam Pacific/Honolulu Pacific/Kiritimati Pacific/Kosrae Pacific/Majuro Pacific/Marquesas Pacific/Nauru Pacific/Niue Pacific/Norfolk Pacific/Noumea Pacific/Pago_Pago Pacific/Palau Pacific/Pitcairn Pacific/Ponape Pacific/Port_Moresby Pacific/Rarotonga Pacific/Saipan Pacific/Tahiti Pacific/Tarawa Pacific/Tongatapu Pacific/Truk Pacific/Wake Pacific/Wallis

You cannot change the suffix field except <file extension>. When you enable data format conversion or compression, Firehose will append a file extension based on the configuration. The following table explains the default file extension appended by Firehose:

Configuration File extension
Data Format Conversion: Parquet .parquet
Data Format Conversion: ORC .orc
Compression: Gzip .gz
Compression: Zip .zip
Compression: Snappy .snappy
Compression: Hadoop-Snappy .hsnappy

You can also specify a file extension that you prefer in the Firehose console or API. File extension must start with a period (.) and can contain allowed characters: 0-9a-z!-_.*‘(). File extension cannot exceed 128 characters.

Note

When you specify a file extension, it will override the default file extension that Firehose adds when data format conversion or compression is enabled.