View a markdown version of this page

Clusters clusterArn Channels - Amazon Managed Streaming for Apache Kafka

Clusters clusterArn Channels

URI

/v1/clusters/clusterArn/channels

HTTP methods

GET

Operation ID: ListChannels

Returns all channels in a cluster.

Path parameters
NameTypeRequiredDescription
clusterArnStringTrue

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

Query parameters
NameTypeRequiredDescription
topicNameFilterStringFalse

Filters results to channels whose topic name matches the specified value.

nextTokenStringFalse

The paginated results marker. When the result of the operation is truncated, the call returns NextToken in the response. To get the next batch, provide this token in your next request.

maxResultsStringFalse

The maximum number of results to return in the response (default maximum 100 results per API call). If there are more results, the response includes a NextToken parameter.

Responses
Status codeResponse modelDescription
200 ListChannelsResponse

Successful response.

400Error

The request isn't valid because the input is incorrect. Correct your input and then submit it again.

401Error

The request is not authorized. The provided credentials couldn't be validated.

403Error

Access forbidden. Check your credentials and then retry your request.

404Error

The resource could not be found due to incorrect input. Correct the input, then retry the request.

429Error

429 response

500Error

There was an unexpected internal server error. Retrying your request might resolve the issue.

503Error

503 response

POST

Operation ID: CreateChannel

Creates a channel using the specified properties in the request on the cluster specified by the Amazon Resource Name (ARN) in the request.

Path parameters
NameTypeRequiredDescription
clusterArnStringTrue

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

Responses
Status codeResponse modelDescription
200 CreateChannelResponse

Successful response.

400Error

The request isn't valid because the input is incorrect. Correct your input and then submit it again.

401Error

The request is not authorized. The provided credentials couldn't be validated.

403Error

Access forbidden. Check your credentials and then retry your request.

404Error

The resource could not be found due to incorrect input. Correct the input, then retry the request.

409Error

This cluster name already exists. Retry your request using another name.

429Error

429 response

500Error

There was an unexpected internal server error. Retrying your request might resolve the issue.

503Error

503 response

OPTIONS

Enable CORS by returning correct headers

Path parameters
NameTypeRequiredDescription
clusterArnStringTrue

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

Responses
Status codeResponse modelDescription
200None

Default response for CORS method

Schemas

Request bodies

{ "s3DestinationConfiguration": { "dataFreshnessInSeconds": integer, "serviceExecutionRoleArn": "string", "deadLetterQueueS3": { "bucketArn": "string", "expectedBucketOwner": "string", "errorOutputPrefix": "string" }, "storage": { "storageClass": enum, "bucketArn": "string", "outputKeyTemplate": "string", "outputPrefix": "string", "expectedBucketOwner": "string", "compressionType": enum } }, "topicConfigurationList": [ { "recordConverter": { "valueConverter": enum }, "recordSchema": { "gsrArn": "string" }, "topicArn": "string" } ], "encryptionConfiguration": { "kmsKeyArn": "string" }, "channelName": "string", "loggingInfo": { "s3": { "bucket": "string", "prefix": "string", "enabled": boolean }, "firehose": { "deliveryStream": "string", "enabled": boolean }, "cloudWatchLogs": { "logGroup": "string", "enabled": boolean } }, "icebergDestinationConfiguration": { "dataFreshnessInSeconds": integer, "serviceExecutionRoleArn": "string", "catalog": { "catalogArn": "string", "warehouseLocation": "string" }, "schemaEvolution": { "enableSchemaEvolution": boolean }, "deadLetterQueueS3": { "bucketArn": "string", "expectedBucketOwner": "string", "errorOutputPrefix": "string" }, "compressionType": enum, "destinationTableList": [ { "destinationTableName": "string", "destinationDatabaseName": "string", "partitionSpec": { "partitionStrategy": enum, "sourceList": [ { "sourceName": "string" } ] } } ], "tableCreation": { "enableTableCreation": boolean }, "appendOnly": boolean }, "tags": { } }

Response bodies

{ "channels": [ { "clusterOperationArn": "string", "channelArn": "string", "creationTime": "string", "destinationType": enum, "channelName": "string", "status": enum } ], "nextToken": "string" }
{ "clusterOperationArn": "string", "channelArn": "string" }
{ "message": "string", "invalidParameter": "string" }

Properties

Catalog

Configuration of the AWS Glue Data Catalog and S3 Tables warehouse used by the Apache Iceberg destination.

PropertyTypeRequiredDescription
catalogArn

string

False

The Amazon Resource Name (ARN) of the federated AWS Glue Data Catalog that projects the S3 Tables bucket. If omitted, MSK derives the catalog ARN from warehouseLocation.

warehouseLocation

string

False

The Amazon Resource Name (ARN) of the S3 Tables bucket that backs the Apache Iceberg warehouse.

ChannelDestinationType

The type of destination configured for the channel.

  • ICEBERG

  • S3

ChannelInfo

Summary information about a channel returned by ListChannels.

PropertyTypeRequiredDescription
channelArn

string

True

The Amazon Resource Name (ARN) that uniquely identifies the channel.

channelName

string

True

The name of the channel.

clusterOperationArn

string

False

The Amazon Resource Name (ARN) of the in-flight cluster operation. Returned only while the channel is in CREATING, UPDATING, or DELETING.

creationTime

string

True

The time when the channel was created.

destinationType

ChannelDestinationType

True

The type of destination configured for the channel.

status

ChannelStatus

True

The current lifecycle state of the channel.

ChannelLoggingInfo

Configuration for the destinations to which the channel publishes operational logs.

PropertyTypeRequiredDescription
cloudWatchLogs

CloudWatchLogs

False

Configuration for delivering logs to Amazon CloudWatch Logs.

firehose

Firehose

False

Configuration for delivering logs to an Amazon Data Firehose delivery stream.

s3

S3

False

Configuration for delivering logs to an Amazon S3 bucket.

ChannelStatus

The lifecycle state of a channel.

  • CREATING

  • ACTIVE

  • UPDATING

  • DELETING

  • FAILED

  • SUSPENDING

  • SUSPENDED

CloudWatchLogs

Details of the CloudWatch Logs destination for broker logs.

PropertyTypeRequiredDescription
enabled

boolean

True

Specifies whether broker logs get sent to the specified CloudWatch Logs destination.

logGroup

string

False

The CloudWatch log group that is the destination for broker logs.

CreateChannelRequest

Creates a Channel that streams records from an Amazon MSK Express cluster topic to Amazon S3 or Apache Iceberg.

PropertyTypeRequiredDescription
channelName

string

True

The name of the channel. Must be unique within the cluster.

encryptionConfiguration

EncryptionConfiguration

False

The encryption configuration applied to the channel.

icebergDestinationConfiguration

IcebergDestinationConfiguration

False

The Apache Iceberg destination for the channel. Mutually exclusive with s3DestinationConfiguration.

loggingInfo

ChannelLoggingInfo

False

The destinations to which the channel publishes operational logs.

s3DestinationConfiguration

S3DestinationConfiguration

False

The Amazon S3 destination for the channel. Mutually exclusive with icebergDestinationConfiguration.

tags

object

False

The tags attached to the channel.

topicConfigurationList

Array of type TopicConfiguration

True

The list of topic configurations for the channel. Currently exactly one topic must be specified.

CreateChannelResponse

Returns the channel ARN and the cluster-operation ARN that tracks the asynchronous create.

PropertyTypeRequiredDescription
channelArn

string

True

The Amazon Resource Name (ARN) that uniquely identifies the channel.

clusterOperationArn

string

False

The Amazon Resource Name (ARN) of the cluster operation.

DeadLetterQueueS3

Configuration of the Amazon S3 bucket where records that fail to deliver are stored.

PropertyTypeRequiredDescription
bucketArn

string

True

The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.

errorOutputPrefix

string

False

An optional prefix prepended to every dead-letter Amazon S3 object key.

expectedBucketOwner

string

False

Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.

DestinationTable

Configuration of an Apache Iceberg destination table.

PropertyTypeRequiredDescription
destinationDatabaseName

string

False

The name of the destination namespace (database) in the AWS Glue Data Catalog.

destinationTableName

string

False

The name of the destination Apache Iceberg table.

partitionSpec

PartitionSpec

False

The partition specification for the destination table.

EncryptionConfiguration

The AWS KMS encryption configuration applied to data at rest.

PropertyTypeRequiredDescription
kmsKeyArn

string

True

The Amazon Resource Name (ARN) of the AWS KMS key used to encrypt the data.

Error

Returns information about an error.

PropertyTypeRequiredDescription
invalidParameter

string

False

The parameter that caused the error.

message

string

False

The description of the error.

Firehose

Firehose details for BrokerLogs.

PropertyTypeRequiredDescription
deliveryStream

string

False

The Kinesis Data Firehose delivery stream that is the destination for broker logs.

enabled

boolean

True

Specifies whether broker logs get sent to the specified Kinesis Data Firehose delivery stream.

IcebergCompressionType

Compression codec for Iceberg table data files. Defaults to ZSTD.

  • ZSTD

  • SNAPPY

IcebergDestinationConfiguration

Configuration of an Apache Iceberg destination for a channel.

PropertyTypeRequiredDescription
appendOnly

boolean

True

Whether the destination is append-only. Must be true; updates and deletes are not supported.

catalog

Catalog

False

The AWS Glue Data Catalog and S3 Tables warehouse used by the destination.

compressionType

IcebergCompressionType

False

The compression codec for Iceberg table data files. Defaults to ZSTD.

dataFreshnessInSeconds

integer

False

The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.

deadLetterQueueS3

DeadLetterQueueS3

True

The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.

destinationTableList

Array of type DestinationTable

True

The destination Iceberg tables. Currently exactly one table must be specified.

schemaEvolution

SchemaEvolution

True

Configuration controlling whether the destination table's schema is evolved to match incoming records.

serviceExecutionRoleArn

string

True

The Amazon Resource Name (ARN) of the IAM role that MSK assumes to access the destination table, the AWS Glue Data Catalog, and the dead-letter Amazon S3 bucket.

tableCreation

TableCreation

True

Configuration controlling whether MSK creates the destination table if it does not already exist.

ListChannelsResponse

Returns the list of channels in the cluster.

PropertyTypeRequiredDescription
channels

Array of type ChannelInfo

False

The list of channels in the cluster.

nextToken

string

False

If the response from ListChannels is truncated, this token is included. Send it as the nextToken parameter on a subsequent ListChannels call to retrieve the next page.

PartitionSource

A source column used by an Apache Iceberg destination table's partition specification.

PropertyTypeRequiredDescription
sourceName

string

False

The name of the source column.

PartitionSpec

Partition specification for an Apache Iceberg destination table.

PropertyTypeRequiredDescription
partitionStrategy

PartitionStrategy

True

The partitioning strategy applied to records written to the table.

sourceList

Array of type PartitionSource

False

The source columns used by the partitioning strategy. For TIME_HOUR, must contain exactly one source column whose value is a timestamp.

PartitionStrategy

The partitioning strategy used to partition records in the destination Apache Iceberg table.

  • TIME_HOUR

RecordConverter

Configuration that controls how Apache Kafka record values are deserialized for the destination.

PropertyTypeRequiredDescription
valueConverter

ValueConverter

True

The deserialization format applied to Apache Kafka record values.

RecordSchema

Schema configuration that controls how Apache Kafka record values are validated.

PropertyTypeRequiredDescription
gsrArn

string

True

The Amazon Resource Name (ARN) of the AWS Glue Schema Registry schema (not registry) used to validate records for the destination Apache Iceberg table.

S3

The details of the Amazon S3 destination for broker logs.

PropertyTypeRequiredDescription
bucket

string

False

The name of the S3 bucket that is the destination for broker logs.

enabled

boolean

True

Specifies whether broker logs get sent to the specified Amazon S3 destination.

prefix

string

False

The S3 prefix that is the destination for broker logs.

S3CompressionType

The compression codec applied to delivered Amazon S3 objects.

  • NONE

  • GZIP

  • ZSTD

S3DestinationConfiguration

Configuration of an Amazon S3 destination for a channel.

PropertyTypeRequiredDescription
dataFreshnessInSeconds

integer

False

The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.

deadLetterQueueS3

DeadLetterQueueS3

True

The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.

serviceExecutionRoleArn

string

True

The Amazon Resource Name (ARN) of the IAM role that MSK assumes to write to the destination Amazon S3 bucket and the dead-letter bucket.

storage

S3Storage

True

The Amazon S3 bucket, prefix, and storage class for delivered records.

S3Storage

Storage configuration for an Amazon S3 destination bucket.

PropertyTypeRequiredDescription
bucketArn

string

True

The Amazon Resource Name (ARN) of the destination Amazon S3 bucket.

compressionType

S3CompressionType

True

The compression codec applied to delivered Amazon S3 objects.

expectedBucketOwner

string

False

Optional 12-digit AWS account ID expected to own the Amazon S3 bucket.

outputKeyTemplate

string

False

An optional template that controls the Amazon S3 object key for each delivered record. Supports the placeholders !{partition-id}, !{sequence-number}, and !{kafka-offset}.

outputPrefix

string

False

An optional prefix prepended to every Amazon S3 object key written by the channel.

storageClass

S3StorageClass

True

The Amazon S3 storage class for delivered objects.

S3StorageClass

The Amazon S3 storage class applied to delivered objects.

  • STANDARD

  • INTELLIGENT_TIERING

  • GLACIER_IR

SchemaEvolution

Configuration controlling whether the Apache Iceberg destination table's schema is evolved as incoming records change.

PropertyTypeRequiredDescription
enableSchemaEvolution

boolean

False

Whether to allow MSK to evolve the destination table's schema. Must be false for the current release.

TableCreation

Configuration controlling whether MSK creates the destination Apache Iceberg table if it does not already exist.

PropertyTypeRequiredDescription
enableTableCreation

boolean

False

Whether MSK creates the destination table on the customer's behalf. Must be true for the current release.

TopicConfiguration

Configuration of an Apache Kafka topic that feeds a channel.

PropertyTypeRequiredDescription
recordConverter

RecordConverter

True

Configuration that controls how Apache Kafka record values are deserialized for the destination.

recordSchema

RecordSchema

False

The schema used to validate records when the value converter requires one (for example, JSON_SCHEMA_GSR).

topicArn

string

True

The Amazon Resource Name (ARN) that uniquely identifies the topic.

ValueConverter

The deserialization format applied to Apache Kafka record values.

  • BYTE_ARRAY

  • JSON

  • JSON_SCHEMA_GSR

  • STRING

See also

For more information about using this API in one of the language-specific AWS SDKs and references, see the following:

ListChannels

CreateChannel