create-channel¶

Description¶

Creates a Channel that streams records from an Amazon MSK Express cluster topic to Amazon S3 or Apache Iceberg.

Synopsis¶

  create-channel
--channel-name <value>
--cluster-arn <value>
[--encryption-configuration <value>]
[--iceberg-destination-configuration <value>]
[--s3-destination-configuration <value>]
[--tags <value>]
--topic-configuration-list <value>
[--logging-info <value>]
[--cli-input-json | --cli-input-yaml]
[--generate-cli-skeleton <value>]
[--debug]
[--endpoint-url <value>]
[--no-verify-ssl]
[--no-paginate]
[--output <value>]
[--query <value>]
[--profile <value>]
[--region <value>]
[--version <value>]
[--color <value>]
[--no-sign-request]
[--ca-bundle <value>]
[--cli-read-timeout <value>]
[--cli-connect-timeout <value>]
[--cli-binary-format <value>]
[--no-cli-pager]
[--cli-auto-prompt]
[--no-cli-auto-prompt]
[--cli-error-format <value>]

Options¶

--channel-name (string) [required]

The name of the channel. Must be unique within the cluster.

--cluster-arn (string) [required]

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

--encryption-configuration (structure)

The encryption configuration applied to the channel.

KmsKeyArn -> (string) [required]

The Amazon Resource Name (ARN) of the AWS KMS key used to encrypt the data.

Shorthand Syntax:

KmsKeyArn=string

JSON Syntax:

{
  "KmsKeyArn": "string"
}

--iceberg-destination-configuration (structure)

The Apache Iceberg destination for the channel. Mutually exclusive with s3DestinationConfiguration.

AppendOnly -> (boolean) [required]

Whether the destination is append-only. Must be true; updates and deletes are not supported.

Catalog -> (structure)

The AWS Glue Data Catalog and S3 Tables warehouse used by the destination.

CatalogArn -> (string)

The Amazon Resource Name (ARN) of the federated AWS Glue Data Catalog that projects the S3 Tables bucket. If omitted, MSK derives the catalog ARN from warehouseLocation.

WarehouseLocation -> (string)

The Amazon Resource Name (ARN) of the S3 Tables bucket that backs the Apache Iceberg warehouse.

DataFreshnessInSeconds -> (integer)

The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.

DeadLetterQueueS3 -> (structure) [required]

The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.

BucketArn -> (string) [required]

The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.

ErrorOutputPrefix -> (string)

An optional prefix prepended to every dead-letter Amazon S3 object key.

ExpectedBucketOwner -> (string)

Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.

DestinationTableList -> (list) [required]

The destination Iceberg tables. Currently exactly one table must be specified.

(structure)

Configuration of an Apache Iceberg destination table.

DestinationDatabaseName -> (string)

The name of the destination namespace (database) in the AWS Glue Data Catalog.

DestinationTableName -> (string)

The name of the destination Apache Iceberg table.

PartitionSpec -> (structure)

The partition specification for the destination table.

PartitionStrategy -> (string) [required]

The partitioning strategy applied to records written to the table.

Possible values:

TIME_HOUR

SourceList -> (list)

The source columns used by the partitioning strategy. For TIME_HOUR, must contain exactly one source column whose value is a timestamp.

(structure)

A source column used by an Apache Iceberg destination table’s partition specification.

SourceName -> (string)

Source name.

SchemaEvolution -> (structure) [required]

Configuration controlling whether the destination table’s schema is evolved to match incoming records.

EnableSchemaEvolution -> (boolean)

Whether to allow MSK to evolve the destination table’s schema. Must be false for the current release.

ServiceExecutionRoleArn -> (string) [required]

The Amazon Resource Name (ARN) of the IAM role that MSK assumes to access the destination table, the AWS Glue Data Catalog, and the dead-letter Amazon S3 bucket.

TableCreation -> (structure) [required]

Configuration controlling whether MSK creates the destination table if it does not already exist.

EnableTableCreation -> (boolean)

Whether MSK creates the destination table on the customer’s behalf. Must be true for the current release.

CompressionType -> (string)

The compression codec for Iceberg table data files. Defaults to ZSTD.

Possible values:

ZSTD

SNAPPY

JSON Syntax:

{
  "AppendOnly": true|false,
  "Catalog": {
    "CatalogArn": "string",
    "WarehouseLocation": "string"
  },
  "DataFreshnessInSeconds": integer,
  "DeadLetterQueueS3": {
    "BucketArn": "string",
    "ErrorOutputPrefix": "string",
    "ExpectedBucketOwner": "string"
  },
  "DestinationTableList": [
    {
      "DestinationDatabaseName": "string",
      "DestinationTableName": "string",
      "PartitionSpec": {
        "PartitionStrategy": "TIME_HOUR",
        "SourceList": [
          {
            "SourceName": "string"
          }
          ...
        ]
      }
    }
    ...
  ],
  "SchemaEvolution": {
    "EnableSchemaEvolution": true|false
  },
  "ServiceExecutionRoleArn": "string",
  "TableCreation": {
    "EnableTableCreation": true|false
  },
  "CompressionType": "ZSTD"|"SNAPPY"
}

--s3-destination-configuration (structure)

The Amazon S3 destination for the channel. Mutually exclusive with icebergDestinationConfiguration.

DataFreshnessInSeconds -> (integer)

The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.

DeadLetterQueueS3 -> (structure) [required]

The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.

BucketArn -> (string) [required]

The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.

ErrorOutputPrefix -> (string)

An optional prefix prepended to every dead-letter Amazon S3 object key.

ExpectedBucketOwner -> (string)

Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.

ServiceExecutionRoleArn -> (string) [required]

The Amazon Resource Name (ARN) of the IAM role that MSK assumes to write to the destination Amazon S3 bucket and the dead-letter bucket.

Storage -> (structure) [required]

The Amazon S3 bucket, prefix, and storage class for delivered records.

BucketArn -> (string) [required]

The Amazon Resource Name (ARN) of the destination Amazon S3 bucket.

CompressionType -> (string) [required]

The compression codec applied to delivered Amazon S3 objects.

Possible values:

NONE

GZIP

ZSTD

OutputPrefix -> (string)

An optional prefix prepended to every Amazon S3 object key written by the channel.

OutputKeyTemplate -> (string)

An optional template that controls the Amazon S3 object key for each delivered record. Supports the placeholders !{partition-id}, !{sequence-number}, and !{kafka-offset}.

StorageClass -> (string) [required]

The Amazon S3 storage class for delivered objects.

Possible values:

STANDARD

INTELLIGENT_TIERING

GLACIER_IR

ExpectedBucketOwner -> (string)

Optional 12-digit AWS account ID expected to own the Amazon S3 bucket.

Shorthand Syntax:

DataFreshnessInSeconds=integer,DeadLetterQueueS3={BucketArn=string,ErrorOutputPrefix=string,ExpectedBucketOwner=string},ServiceExecutionRoleArn=string,Storage={BucketArn=string,CompressionType=string,OutputPrefix=string,OutputKeyTemplate=string,StorageClass=string,ExpectedBucketOwner=string}

JSON Syntax:

{
  "DataFreshnessInSeconds": integer,
  "DeadLetterQueueS3": {
    "BucketArn": "string",
    "ErrorOutputPrefix": "string",
    "ExpectedBucketOwner": "string"
  },
  "ServiceExecutionRoleArn": "string",
  "Storage": {
    "BucketArn": "string",
    "CompressionType": "NONE"|"GZIP"|"ZSTD",
    "OutputPrefix": "string",
    "OutputKeyTemplate": "string",
    "StorageClass": "STANDARD"|"INTELLIGENT_TIERING"|"GLACIER_IR",
    "ExpectedBucketOwner": "string"
  }
}

--tags (map)

The tags attached to the channel.

key -> (string)

value -> (string)

Shorthand Syntax:

KeyName1=string,KeyName2=string

JSON Syntax:

{"string": "string"
  ...}

--topic-configuration-list (list) [required]

The list of topic configurations for the channel. Currently exactly one topic must be specified.

(structure)

Configuration of an Apache Kafka topic that feeds a channel.

RecordConverter -> (structure) [required]

Configuration that controls how Apache Kafka record values are deserialized for the destination.

ValueConverter -> (string) [required]

The deserialization format applied to Apache Kafka record values.

Possible values:

BYTE_ARRAY

JSON

JSON_SCHEMA_GSR

STRING

RecordSchema -> (structure)

The schema used to validate records when the value converter requires one (for example, JSON_SCHEMA_GSR).

GsrArn -> (string) [required]

The Amazon Resource Name (ARN) of the AWS Glue Schema Registry schema (not registry) used to validate records for the destination Apache Iceberg table.

TopicArn -> (string) [required]

The Amazon Resource Name (ARN) that uniquely identifies the topic.

Shorthand Syntax:

RecordConverter={ValueConverter=string},RecordSchema={GsrArn=string},TopicArn=string ...

JSON Syntax:

[
  {
    "RecordConverter": {
      "ValueConverter": "BYTE_ARRAY"|"JSON"|"JSON_SCHEMA_GSR"|"STRING"
    },
    "RecordSchema": {
      "GsrArn": "string"
    },
    "TopicArn": "string"
  }
  ...
]

--logging-info (structure)

The destinations to which the channel publishes operational logs.

CloudWatchLogs -> (structure)

Details of the CloudWatch Logs destination for Channel logs.

Enabled -> (boolean) [required]

LogGroup -> (string)

Firehose -> (structure)

Details of the Kinesis Data Firehose delivery stream that is the destination for Channel logs.

DeliveryStream -> (string)

Enabled -> (boolean) [required]

S3 -> (structure)

Details of the Amazon S3 destination for Channel logs.

Bucket -> (string)

Enabled -> (boolean) [required]

Prefix -> (string)

Shorthand Syntax:

CloudWatchLogs={Enabled=boolean,LogGroup=string},Firehose={DeliveryStream=string,Enabled=boolean},S3={Bucket=string,Enabled=boolean,Prefix=string}

JSON Syntax:

{
  "CloudWatchLogs": {
    "Enabled": true|false,
    "LogGroup": "string"
  },
  "Firehose": {
    "DeliveryStream": "string",
    "Enabled": true|false
  },
  "S3": {
    "Bucket": "string",
    "Enabled": true|false,
    "Prefix": "string"
  }
}

--cli-input-json | --cli-input-yaml (string) Reads arguments from the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, those values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally. This may not be specified along with --cli-input-yaml.

--generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. Similarly, if provided yaml-input it will print a sample input YAML that can be used with --cli-input-yaml. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command. The generated JSON skeleton is not stable between versions of the AWS CLI and there are no backwards compatibility guarantees in the JSON skeleton generated.

Global Options¶

--debug (boolean)

Turn on debug logging.

--endpoint-url (string)

Override command’s default URL with the given URL.

--no-verify-ssl (boolean)

By default, the AWS CLI uses SSL when communicating with AWS services. For each SSL connection, the AWS CLI will verify SSL certificates. This option overrides the default behavior of verifying SSL certificates.

--no-paginate (boolean)

Disable automatic pagination. If automatic pagination is disabled, the AWS CLI will only make one call, for the first page of results.

--output (string)

The formatting style for command output.

json
text
table
yaml
yaml-stream
off

--query (string)

A JMESPath query to use in filtering the response data.

--profile (string)

Use a specific profile from your credential file.

--region (string)

The region to use. Overrides config/env settings.

--version (string)

Display the version of this tool.

--color (string)

Turn on/off color output.

on
off
auto

--no-sign-request (boolean)

Do not sign requests. Credentials will not be loaded if this argument is provided.

--ca-bundle (string)

The CA certificate bundle to use when verifying SSL certificates. Overrides config/env settings.

--cli-read-timeout (int)

The maximum socket read time in seconds. If the value is set to 0, the socket read will be blocking and not timeout. The default value is 60 seconds.

--cli-connect-timeout (int)

The maximum socket connect time in seconds. If the value is set to 0, the socket connect will be blocking and not timeout. The default value is 60 seconds.

--cli-binary-format (string)

The formatting style to be used for binary blobs. The default format is base64. The base64 format expects binary blobs to be provided as a base64 encoded string. The raw-in-base64-out format preserves compatibility with AWS CLI V1 behavior and binary values must be passed literally. When providing contents from a file that map to a binary blob fileb:// will always be treated as binary and use the file contents directly regardless of the cli-binary-format setting. When using file:// the file contents will need to properly formatted for the configured cli-binary-format.

base64
raw-in-base64-out

--no-cli-pager (boolean)

Disable cli pager for output.

--cli-auto-prompt (boolean)

Automatically prompt for CLI input parameters.

--no-cli-auto-prompt (boolean)

Disable automatically prompt for CLI input parameters.

--cli-error-format (string)

The formatting style for error output. By default, errors are displayed in enhanced format.

legacy
json
yaml
text
table
enhanced

Output¶

ChannelArn -> (string)

The Amazon Resource Name (ARN) that uniquely identifies the channel.

ClusterOperationArn -> (string)

The Amazon Resource Name (ARN) of the cluster operation.

Table of Contents

Feedback

User Guide

create-channel¶

Description¶

Synopsis¶

Options¶

Global Options¶

Output¶