Kafka / Client / create_channel

create_channel¶

Kafka.Client.create_channel(**kwargs)¶

Creates a Channel that streams records from an Amazon MSK Express cluster topic to Amazon S3 or Apache Iceberg.

Request Syntax

response = client.create_channel(
    ChannelName='string',
    ClusterArn='string',
    EncryptionConfiguration={
        'KmsKeyArn': 'string'
    },
    IcebergDestinationConfiguration={
        'AppendOnly': True|False,
        'Catalog': {
            'CatalogArn': 'string',
            'WarehouseLocation': 'string'
        },
        'DataFreshnessInSeconds': 123,
        'DeadLetterQueueS3': {
            'BucketArn': 'string',
            'ErrorOutputPrefix': 'string',
            'ExpectedBucketOwner': 'string'
        },
        'DestinationTableList': [
            {
                'DestinationDatabaseName': 'string',
                'DestinationTableName': 'string',
                'PartitionSpec': {
                    'PartitionStrategy': 'TIME_HOUR',
                    'SourceList': [
                        {
                            'SourceName': 'string'
                        },
                    ]
                }
            },
        ],
        'SchemaEvolution': {
            'EnableSchemaEvolution': True|False
        },
        'ServiceExecutionRoleArn': 'string',
        'TableCreation': {
            'EnableTableCreation': True|False
        },
        'CompressionType': 'ZSTD'|'SNAPPY'
    },
    S3DestinationConfiguration={
        'DataFreshnessInSeconds': 123,
        'DeadLetterQueueS3': {
            'BucketArn': 'string',
            'ErrorOutputPrefix': 'string',
            'ExpectedBucketOwner': 'string'
        },
        'ServiceExecutionRoleArn': 'string',
        'Storage': {
            'BucketArn': 'string',
            'CompressionType': 'NONE'|'GZIP'|'ZSTD',
            'OutputPrefix': 'string',
            'OutputKeyTemplate': 'string',
            'StorageClass': 'STANDARD'|'INTELLIGENT_TIERING'|'GLACIER_IR',
            'ExpectedBucketOwner': 'string'
        }
    },
    Tags={
        'string': 'string'
    },
    TopicConfigurationList=[
        {
            'RecordConverter': {
                'ValueConverter': 'BYTE_ARRAY'|'JSON'|'JSON_SCHEMA_GSR'|'STRING'
            },
            'RecordSchema': {
                'GsrArn': 'string'
            },
            'TopicArn': 'string'
        },
    ],
    LoggingInfo={
        'CloudWatchLogs': {
            'Enabled': True|False,
            'LogGroup': 'string'
        },
        'Firehose': {
            'DeliveryStream': 'string',
            'Enabled': True|False
        },
        'S3': {
            'Bucket': 'string',
            'Enabled': True|False,
            'Prefix': 'string'
        }
    }
)

Parameters:

ChannelName (string) –
[REQUIRED]

The name of the channel. Must be unique within the cluster.
ClusterArn (string) –
[REQUIRED]

The Amazon Resource Name (ARN) that uniquely identifies the cluster.
EncryptionConfiguration (dict) –
The encryption configuration applied to the channel.
- KmsKeyArn (string) – [REQUIRED]
  
  The Amazon Resource Name (ARN) of the AWS KMS key used to encrypt the data.
IcebergDestinationConfiguration (dict) –
The Apache Iceberg destination for the channel. Mutually exclusive with s3DestinationConfiguration.
- AppendOnly (boolean) – [REQUIRED]
  
  Whether the destination is append-only. Must be true; updates and deletes are not supported.
- Catalog (dict) –
  
  The AWS Glue Data Catalog and S3 Tables warehouse used by the destination.
  - CatalogArn (string) –
    
    The Amazon Resource Name (ARN) of the federated AWS Glue Data Catalog that projects the S3 Tables bucket. If omitted, MSK derives the catalog ARN from warehouseLocation.
  - WarehouseLocation (string) –
    
    The Amazon Resource Name (ARN) of the S3 Tables bucket that backs the Apache Iceberg warehouse.
- DataFreshnessInSeconds (integer) –
  
  The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.
- DeadLetterQueueS3 (dict) – [REQUIRED]
  
  The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.
  - BucketArn (string) – [REQUIRED]
    
    The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.
  - ErrorOutputPrefix (string) –
    
    An optional prefix prepended to every dead-letter Amazon S3 object key.
  - ExpectedBucketOwner (string) –
    
    Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.
- DestinationTableList (list) – [REQUIRED]
  
  The destination Iceberg tables. Currently exactly one table must be specified.
  - (dict) –
    
    Configuration of an Apache Iceberg destination table.
    - DestinationDatabaseName (string) –
      
      The name of the destination namespace (database) in the AWS Glue Data Catalog.
    - DestinationTableName (string) –
      
      The name of the destination Apache Iceberg table.
    - PartitionSpec (dict) –
      
      The partition specification for the destination table.
      - PartitionStrategy (string) – [REQUIRED]
        
        The partitioning strategy applied to records written to the table.
      - SourceList (list) –
        
        The source columns used by the partitioning strategy. For TIME_HOUR, must contain exactly one source column whose value is a timestamp.
        
        (dict) –
        
        A source column used by an Apache Iceberg destination table’s partition specification.
        
        SourceName (string) –
        
        Source name.
- SchemaEvolution (dict) – [REQUIRED]
  
  Configuration controlling whether the destination table’s schema is evolved to match incoming records.
  - EnableSchemaEvolution (boolean) –
    
    Whether to allow MSK to evolve the destination table’s schema. Must be false for the current release.
- ServiceExecutionRoleArn (string) – [REQUIRED]
  
  The Amazon Resource Name (ARN) of the IAM role that MSK assumes to access the destination table, the AWS Glue Data Catalog, and the dead-letter Amazon S3 bucket.
- TableCreation (dict) – [REQUIRED]
  
  Configuration controlling whether MSK creates the destination table if it does not already exist.
  - EnableTableCreation (boolean) –
    
    Whether MSK creates the destination table on the customer’s behalf. Must be true for the current release.
- CompressionType (string) –
  
  The compression codec for Iceberg table data files. Defaults to ZSTD.
S3DestinationConfiguration (dict) –
The Amazon S3 destination for the channel. Mutually exclusive with icebergDestinationConfiguration.
- DataFreshnessInSeconds (integer) –
  
  The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.
- DeadLetterQueueS3 (dict) – [REQUIRED]
  
  The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.
  - BucketArn (string) – [REQUIRED]
    
    The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.
  - ErrorOutputPrefix (string) –
    
    An optional prefix prepended to every dead-letter Amazon S3 object key.
  - ExpectedBucketOwner (string) –
    
    Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.
- ServiceExecutionRoleArn (string) – [REQUIRED]
  
  The Amazon Resource Name (ARN) of the IAM role that MSK assumes to write to the destination Amazon S3 bucket and the dead-letter bucket.
- Storage (dict) – [REQUIRED]
  
  The Amazon S3 bucket, prefix, and storage class for delivered records.
  - BucketArn (string) – [REQUIRED]
    
    The Amazon Resource Name (ARN) of the destination Amazon S3 bucket.
  - CompressionType (string) – [REQUIRED]
    
    The compression codec applied to delivered Amazon S3 objects.
  - OutputPrefix (string) –
    
    An optional prefix prepended to every Amazon S3 object key written by the channel.
  - OutputKeyTemplate (string) –
    
    An optional template that controls the Amazon S3 object key for each delivered record. Supports the placeholders !{partition-id}, !{sequence-number}, and !{kafka-offset}.
  - StorageClass (string) – [REQUIRED]
    
    The Amazon S3 storage class for delivered objects.
  - ExpectedBucketOwner (string) –
    
    Optional 12-digit AWS account ID expected to own the Amazon S3 bucket.
Tags (dict) –
The tags attached to the channel.
- (string) –
  - (string) –
TopicConfigurationList (list) –
[REQUIRED]

The list of topic configurations for the channel. Currently exactly one topic must be specified.
- (dict) –
  
  Configuration of an Apache Kafka topic that feeds a channel.
  - RecordConverter (dict) – [REQUIRED]
    
    Configuration that controls how Apache Kafka record values are deserialized for the destination.
    - ValueConverter (string) – [REQUIRED]
      
      The deserialization format applied to Apache Kafka record values.
  - RecordSchema (dict) –
    
    The schema used to validate records when the value converter requires one (for example, JSON_SCHEMA_GSR).
    - GsrArn (string) – [REQUIRED]
      
      The Amazon Resource Name (ARN) of the AWS Glue Schema Registry schema (not registry) used to validate records for the destination Apache Iceberg table.
  - TopicArn (string) – [REQUIRED]
    
    The Amazon Resource Name (ARN) that uniquely identifies the topic.
LoggingInfo (dict) –
The destinations to which the channel publishes operational logs.
- CloudWatchLogs (dict) –
  
  Details of the CloudWatch Logs destination for Channel logs.
  - Enabled (boolean) – [REQUIRED]
  - LogGroup (string) –
- Firehose (dict) –
  
  Details of the Kinesis Data Firehose delivery stream that is the destination for Channel logs.
  - DeliveryStream (string) –
  - Enabled (boolean) – [REQUIRED]
- S3 (dict) –
  
  Details of the Amazon S3 destination for Channel logs.
  - Bucket (string) –
  - Enabled (boolean) – [REQUIRED]
  - Prefix (string) –

Return type:

dict

Returns:

Response Syntax

{
    'ChannelArn': 'string',
    'ClusterOperationArn': 'string'
}

Response Structure

(dict) –

200 response
- ChannelArn (string) –
  
  The Amazon Resource Name (ARN) that uniquely identifies the channel.
- ClusterOperationArn (string) –
  
  The Amazon Resource Name (ARN) of the cluster operation.

Exceptions

Kafka.Client.exceptions.BadRequestException
Kafka.Client.exceptions.UnauthorizedException
Kafka.Client.exceptions.InternalServerErrorException
Kafka.Client.exceptions.ForbiddenException
Kafka.Client.exceptions.NotFoundException
Kafka.Client.exceptions.ServiceUnavailableException
Kafka.Client.exceptions.TooManyRequestsException
Kafka.Client.exceptions.ConflictException