OrcOutputFormatProps

class aws_cdk.aws_kinesisfirehose.OrcOutputFormatProps(*, block_size=None, bloom_filter_columns=None, bloom_filter_false_positive_probability=None, compression=None, dictionary_key_threshold=None, enable_padding=None, format_version=None, padding_tolerance=None, row_index_stride=None, stripe_size=None)

Bases: object

Props for ORC output format for data record format conversion.

Parameters:

block_size (Optional[Size]) – The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. Firehose uses this value for padding calculations. Default: Size.mebibytes(256)
bloom_filter_columns (Optional[Sequence[str]]) – The column names for which you want Firehose to create bloom filters. Default: no bloom filters are created
bloom_filter_false_positive_probability (Union[int, float, None]) – The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the bloom filter. Default: 0.05
compression (Optional[OrcCompression]) – The compression code to use over data blocks. The possible values are NONE , SNAPPY , and ZLIB. Use SNAPPY for higher decompression speed. Use GZIP if the compression ratio is more important than speed. Default: SNAPPY
dictionary_key_threshold (Union[int, float, None]) – Determines whether dictionary encoding should be applied to a column. If the number of distinct keys (unique values) in a column exceeds this fraction of the total non-null rows in that column, dictionary encoding will be turned off for that specific column. To turn off dictionary encoding, set this threshold to 0. To always use dictionary encoding, set this threshold to 1. Default: 0.8
enable_padding (Optional[bool]) – Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. Default: false
format_version (Optional[OrcFormatVersion]) – The version of the ORC format to write. The possible values are V0_11 and V0_12. Default: V0_12
padding_tolerance (Union[int, float, None]) – A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size. For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task. Kinesis Data Firehose ignores this parameter when EnablePadding is false . Default: 0.05 if enablePadding is true
row_index_stride (Union[int, float, None]) – The number of rows between index entries. Default: 10000
stripe_size (Optional[Size]) – The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB. Default: Size.mebibytes(64)

ExampleMetadata:

infused

Example:

output_format = firehose.OrcOutputFormat(
    format_version=firehose.OrcFormatVersion.V0_11,
    block_size=Size.mebibytes(256),
    compression=firehose.OrcCompression.NONE,
    bloom_filter_columns=["columnA"],
    bloom_filter_false_positive_probability=0.1,
    dictionary_key_threshold=0.7,
    enable_padding=True,
    padding_tolerance=0.2,
    row_index_stride=9000,
    stripe_size=Size.mebibytes(32)
)

Attributes

block_size

The Hadoop Distributed File System (HDFS) block size.

This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. Firehose uses this value for padding calculations.

Default:: Size.mebibytes(256)
Minimum:: Size.mebibytes(64)

bloom_filter_columns

The column names for which you want Firehose to create bloom filters.

Default:: no bloom filters are created
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-bloomfiltercolumns

bloom_filter_false_positive_probability

The Bloom filter false positive probability (FPP).

The lower the FPP, the bigger the bloom filter.

Default:: 0.05
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-bloomfilterfalsepositiveprobability
Maximum:: 1
Minimum:: 0

compression

The compression code to use over data blocks.

The possible values are NONE , SNAPPY , and ZLIB. Use SNAPPY for higher decompression speed. Use GZIP if the compression ratio is more important than speed.

Default:: SNAPPY
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-compression

dictionary_key_threshold

Determines whether dictionary encoding should be applied to a column.

If the number of distinct keys (unique values) in a column exceeds this fraction of the total non-null rows in that column, dictionary encoding will be turned off for that specific column.

To turn off dictionary encoding, set this threshold to 0. To always use dictionary encoding, set this threshold to 1.

Default:: 0.8
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-dictionarykeythreshold
Maximum:: 1
Minimum:: 0

enable_padding

Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries.

This is useful if you intend to copy the data from Amazon S3 to HDFS before querying.

Default:: false
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-enablepadding

format_version

The version of the ORC format to write.

The possible values are V0_11 and V0_12.

Default:: V0_12
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-formatversion

padding_tolerance

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.

The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Kinesis Data Firehose ignores this parameter when EnablePadding is false .

Default:: 0.05 if enablePadding is true
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-paddingtolerance

row_index_stride

The number of rows between index entries.

Default:: 10000
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-rowindexstride
Minimum:: 1000

stripe_size

The number of bytes in each stripe.

The default is 64 MiB and the minimum is 8 MiB.

Default:: Size.mebibytes(64)
See:: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-orcserde.html#cfn-kinesisfirehose-deliverystream-orcserde-stripesizebytes
Minimum:: Size.mebibytes(8)