View a markdown version of this page

V1 Replicators - Amazon Managed Streaming for Apache Kafka
URIHTTP methodsSchemasPropertiesSee also

V1 Replicators

URI

/replication/v1/replicators

HTTP methods

GET

Operation ID: ListReplicators

Lists the replicators.

Query parameters
NameTypeRequiredDescription
replicatorNameFilterStringFalse

Returns replicators starting with given name.

nextTokenStringFalse

If the response of ListReplicators is truncated, it returns a NextToken in the response. This NextToken should be sent in the subsequent request to ListReplicators.

maxResultsStringFalse

The maximum number of results to return in the response. If there are more results, the response includes a NextToken parameter.

Responses
Status codeResponse modelDescription
200 ListReplicatorsResponse

HTTP Status Code 200: OK.

400None

HTTP Status Code 400: Bad request due to incorrect input. Correct your request and then retry it.

401None

HTTP Status Code 401: Unauthorized request. The provided credentials couldn't be validated.

403None

HTTP Status Code 403: Access forbidden. Correct your credentials and then retry your request.

404None

HTTP Status Code 404: Resource not found due to incorrect input. Correct your request and then retry it.

429None

HTTP Status Code 429: Limit exceeded. Resource limit reached.

500None

HTTP Status Code 500: Unexpected internal server error. Retrying your request might resolve the issue.

503None

HTTP Status Code 503: Service Unavailable. Retrying your request in some time might resolve the issue.

POST

Operation ID: CreateReplicator

Creates the replicator.

Note: Enhanced consumer offset syncing is only supported when the topicNameConfiguration type is IDENTICAL.

Responses
Status codeResponse modelDescription
200 CreateReplicatorResponse

HTTP Status Code 200: OK.

400None

HTTP Status Code 400: Bad request due to incorrect input. Correct your request and then retry it.

401None

HTTP Status Code 401: Unauthorized request. The provided credentials couldn't be validated.

403None

HTTP Status Code 403: Access forbidden. Correct your credentials and then retry your request.

404None

HTTP Status Code 404: Resource not found due to incorrect input. Correct your request and then retry it.

409None

HTTP Status Code 409: Conflict. This replicator name already exists. Retry your request with another name.

429None

HTTP Status Code 429: Limit exceeded. Resource limit reached.

500None

HTTP Status Code 500: Unexpected internal server error. Retrying your request might resolve the issue.

503None

HTTP Status Code 503: Service Unavailable. Retrying your request in some time might resolve the issue.

OPTIONS

Enable CORS by returning correct headers

Responses
Status codeResponse modelDescription
200None

Default response for CORS method

Schemas

Request bodies

{
  "replicatorName": "string",
  "logDelivery": {
    "replicatorLogDelivery": {
      "s3": {
        "bucket": "string",
        "prefix": "string",
        "enabled": boolean
      },
      "firehose": {
        "deliveryStream": "string",
        "enabled": boolean
      },
      "cloudWatchLogs": {
        "logGroup": "string",
        "enabled": boolean
      }
    }
  },
  "serviceExecutionRoleArn": "string",
  "replicationInfoList": [
    {
      "consumerGroupReplication": {
        "consumerGroupsToExclude": [
          "string"
        ],
        "consumerGroupOffsetSyncMode": enum,
        "detectAndCopyNewConsumerGroups": boolean,
        "consumerGroupsToReplicate": [
          "string"
        ],
        "synchroniseConsumerGroupOffsets": boolean
      },
      "targetCompressionType": enum,
      "topicReplication": {
        "copyAccessControlListsForTopics": boolean,
        "detectAndCopyNewTopics": boolean,
        "topicNameConfiguration": {
          "type": enum
        },
        "copyTopicConfigurations": boolean,
        "startingPosition": {
          "type": enum
        },
        "topicsToReplicate": [
          "string"
        ],
        "topicsToExclude": [
          "string"
        ]
      },
      "sourceKafkaClusterArn": "string",
      "targetKafkaClusterArn": "string",
      "sourceKafkaClusterId": "string",
      "targetKafkaClusterId": "string"
    }
  ],
  "description": "string",
  "kafkaClusters": [
    {
      "amazonMskCluster": {
        "mskClusterArn": "string"
      },
      "vpcConfig": {
        "securityGroupIds": [
          "string"
        ],
        "subnetIds": [
          "string"
        ]
      },
      "encryptionInTransit": {
        "encryptionType": enum,
        "rootCaCertificate": "string"
      },
      "apacheKafkaCluster": {
        "bootstrapBrokerString": "string",
        "apacheKafkaClusterId": "string"
      },
      "clientAuthentication": {
        "saslScram": {
          "secretArn": "string",
          "mechanism": enum
        }
      }
    }
  ],
  "tags": {
  }
}

Response bodies

{
  "nextToken": "string",
  "replicators": [
    {
      "replicatorArn": "string",
      "replicatorName": "string",
      "creationTime": "string",
      "kafkaClustersSummary": [
        {
          "kafkaClusterAlias": "string",
          "amazonMskCluster": {
            "mskClusterArn": "string"
          },
          "apacheKafkaCluster": {
            "bootstrapBrokerString": "string",
            "apacheKafkaClusterId": "string"
          }
        }
      ],
      "replicatorState": enum,
      "isReplicatorReference": boolean,
      "replicationInfoSummaryList": [
        {
          "sourceKafkaClusterAlias": "string",
          "targetKafkaClusterAlias": "string"
        }
      ],
      "replicatorResourceArn": "string",
      "currentVersion": "string"
    }
  ]
}
{
  "replicatorArn": "string",
  "replicatorName": "string",
  "replicatorState": enum
}

Properties

AmazonMskCluster

Details of an Amazon MSK Cluster.

PropertyTypeRequiredDescription
mskClusterArn

string

True

The Amazon Resource Name (ARN) of an Amazon MSK cluster.

ApacheKafkaCluster

Details of an Apache Kafka cluster.

PropertyTypeRequiredDescription
apacheKafkaClusterId

string

True

The globally unique cluster ID of the Apache Kafka cluster. For information on how to retrieve the cluster ID, see https://github.com/apache/kafka/blob/trunk/bin/kafka-cluster.sh

bootstrapBrokerString

string

True

The bootstrap broker string of the Apache Kafka cluster.

CloudWatchLogs

CloudWatch Logs details for ReplicatorLogDelivery.

PropertyTypeRequiredDescription
enabled

boolean

True

Whether log delivery to CloudWatch Logs is enabled.

logGroup

string

False

The CloudWatch log group that is the destination for log delivery.

ConsumerGroupOffsetSyncMode

The consumer group offset synchronization mode. With LEGACY, offsets are synchronized when producers write to the source cluster. With ENHANCED, consumer offsets are synchronized regardless of producer location. ENHANCED requires a corresponding replicator that replicates data from the target cluster to the source cluster.

  • LEGACY

  • ENHANCED

ConsumerGroupReplication

Details about consumer group replication.

PropertyTypeRequiredDescription
consumerGroupOffsetSyncMode

ConsumerGroupOffsetSyncMode

False

The consumer group offset synchronization mode. With LEGACY, offsets are synchronized when producers write to the source cluster. With ENHANCED, consumer offsets are synchronized regardless of producer location. ENHANCED requires a corresponding replicator that replicates data from the target cluster to the source cluster.

consumerGroupsToExclude

Array of type string

MaxLength: 256

False

List of regular expression patterns indicating the consumer groups that should not be replicated.

consumerGroupsToReplicate

Array of type string

MaxLength: 256

True

List of regular expression patterns indicating the consumer groups to copy.

detectAndCopyNewConsumerGroups

boolean

False

Enables synchronization of consumer groups to target cluster.

synchroniseConsumerGroupOffsets

boolean

False

Enables synchronization of consumer group offsets to target cluster. The translated offsets will be written to topic __consumer_offsets.

CreateReplicatorRequest

Request body for replicator.

PropertyTypeRequiredDescription
description

string

MaxLength: 1024

False

A summary description of the replicator.

kafkaClusters

Array of type KafkaCluster

True

Kafka Clusters to use in setting up sources / targets for replication.

logDelivery

LogDelivery

False

Configuration for delivering replicator logs to customer destinations.

replicationInfoList

Array of type ReplicationInfo

True

A list of replication configurations, where each configuration targets a given source cluster to target cluster replication flow.

replicatorName

string

Pattern: ^[0-9A-Za-z][0-9A-Za-z-]{0,}$

MinLength: 1

MaxLength: 128

True

The name of the replicator. Alpha-numeric characters with '-' are allowed.

serviceExecutionRoleArn

string

True

The ARN of the IAM role used by the replicator to access resources in the customer's account (e.g source and target clusters)

tags

object

False

List of tags to attach to created Replicator.

CreateReplicatorResponse

Returns information about the created replicator.

PropertyTypeRequiredDescription
replicatorArn

string

False

The Amazon Resource Name (ARN) of the replicator.

replicatorName

string

False

Name of the replicator provided by the customer.

replicatorState

ReplicatorState

False

State of the replicator.

Firehose

Firehose details for ReplicatorLogDelivery.

PropertyTypeRequiredDescription
deliveryStream

string

False

The Firehose delivery stream that is the destination for log delivery.

enabled

boolean

True

Whether log delivery to Firehose is enabled.

KafkaCluster

Information about Kafka Cluster to be used as source / target for replication.

PropertyTypeRequiredDescription
amazonMskCluster

AmazonMskCluster

False

Details of an Amazon MSK Cluster.

apacheKafkaCluster

ApacheKafkaCluster

False

Details of an Apache Kafka cluster. Exactly one of amazonMskCluster and apacheKafkaCluster is required.

clientAuthentication

KafkaClusterClientAuthentication

False

Details of the client authentication used by the Apache Kafka cluster.

encryptionInTransit

KafkaClusterEncryptionInTransit

False

Details of encryption in transit to the Apache Kafka cluster.

vpcConfig

KafkaClusterClientVpcConfig

False

Details of an Amazon VPC which has network connectivity to the Apache Kafka cluster.

KafkaClusterClientAuthentication

Details of the client authentication used by the Apache Kafka cluster.

PropertyTypeRequiredDescription
saslScram

KafkaClusterSaslScramAuthentication

True

Details for SASL/SCRAM client authentication.

KafkaClusterClientVpcConfig

Details of an Amazon VPC which has network connectivity to the Apache Kafka cluster.

PropertyTypeRequiredDescription
securityGroupIds

Array of type string

False

The security groups to attach to the ENIs for the broker nodes.

subnetIds

Array of type string

True

The list of subnets in the client VPC to connect to.

KafkaClusterEncryptionInTransit

Details of encryption in transit to the Apache Kafka cluster.

PropertyTypeRequiredDescription
encryptionType

KafkaClusterEncryptionInTransitType

True

The type of encryption in transit to the Apache Kafka cluster.

rootCaCertificate

string

False

Amazon Resource Name (ARN) of the Secrets Manager secret containing the root CA certificate for TLS encryption.

KafkaClusterEncryptionInTransitType

The type of encryption in transit to the Apache Kafka cluster.

  • TLS

KafkaClusterSaslScramAuthentication

Details for SASL/SCRAM client authentication.

PropertyTypeRequiredDescription
mechanism

KafkaClusterSaslScramMechanism

True

The SCRAM mechanism to use for authentication.

secretArn

string

True

Amazon Resource Name (ARN) of the Secrets Manager secret containing the username and password.

KafkaClusterSaslScramMechanism

The SCRAM mechanism to use for authentication.

  • SHA256

  • SHA512

KafkaClusterSummary

Summarized information about Kafka Cluster used as source / target for replication.

PropertyTypeRequiredDescription
amazonMskCluster

AmazonMskCluster

False

Details of an Amazon MSK Cluster.

apacheKafkaCluster

ApacheKafkaCluster

False

Details of an Apache Kafka Cluster.

kafkaClusterAlias

string

False

The alias of the Kafka cluster. Used to prefix names of replicated topics.

ListReplicatorsResponse

The response contains an array containing replicator information and a NextToken if the response is truncated.

PropertyTypeRequiredDescription
nextToken

string

False

If the response of ListReplicators is truncated, it returns a NextToken in the response. This NextToken should be sent in the subsequent request to ListReplicators.

replicators

Array of type ReplicatorSummary

False

List containing information of each of the replicators in the account.

LogDelivery

Configuration for log delivery to customer destinations.

PropertyTypeRequiredDescription
replicatorLogDelivery

ReplicatorLogDelivery

False

Configuration for replicator log delivery.

ReplicationInfo

Specifies configuration for replication between a source and target Kafka cluster.

PropertyTypeRequiredDescription
consumerGroupReplication

ConsumerGroupReplication

True

Configuration relating to consumer group replication.

sourceKafkaClusterArn

string

False

The ARN of the source Kafka cluster.

sourceKafkaClusterId

string

False

The cluster ID of the source Apache Kafka cluster. Specify either sourceKafkaClusterArn or sourceKafkaClusterId, but not both.

targetCompressionType

TargetCompressionType

True

The compression type to use when producing records to target cluster.

targetKafkaClusterArn

string

False

The ARN of the target Kafka cluster.

targetKafkaClusterId

string

False

The cluster ID of the target Apache Kafka cluster. Specify either targetKafkaClusterArn or targetKafkaClusterId, but not both.

topicReplication

TopicReplication

True

Configuration relating to topic replication.

ReplicationInfoSummary

Summarized information of replication between clusters.

PropertyTypeRequiredDescription
sourceKafkaClusterAlias

string

False

The alias of the source Kafka cluster.

targetKafkaClusterAlias

string

False

The alias of the target Kafka cluster.

ReplicationStartingPosition

Specifies the position in the topics to start replicating from.

PropertyTypeRequiredDescription
type

ReplicationStartingPositionType

False

The type of replication starting position.

ReplicationStartingPositionType

The type of replication starting position.

  • LATEST

  • EARLIEST

ReplicationTopicNameConfiguration

Configuration for specifying replicated topic names will be the same as their corresponding upstream topics or prefixed with source cluster alias.

PropertyTypeRequiredDescription
type

ReplicationTopicNameConfigurationType

False

The type of replication topic name configuration, identical to upstream topic name or prefixed with source cluster alias.

ReplicationTopicNameConfigurationType

The type of replication topic name configuration, identical to upstream topic name or prefixed with source cluster alias.

  • IDENTICAL

  • PREFIXED_WITH_SOURCE_CLUSTER_ALIAS

ReplicatorLogDelivery

Configuration for replicator log delivery.

PropertyTypeRequiredDescription
cloudWatchLogs

CloudWatchLogs

False

Configuration for CloudWatch Logs delivery.

firehose

Firehose

False

Configuration for Firehose delivery.

s3

S3

False

Configuration for S3 delivery.

ReplicatorState

State of a replicator.

  • RUNNING

  • CREATING

  • UPDATING

  • DELETING

  • FAILED

ReplicatorSummary

Information about a replicator.

PropertyTypeRequiredDescription
creationTime

string

False

The time the replicator was created.

currentVersion

string

False

The current version of the replicator.

isReplicatorReference

boolean

False

Whether this resource is a replicator reference.

kafkaClustersSummary

Array of type KafkaClusterSummary

False

Kafka Clusters used in setting up sources / targets for replication.

replicationInfoSummaryList

Array of type ReplicationInfoSummary

False

A list of summarized information of replications between clusters.

replicatorArn

string

False

The Amazon Resource Name (ARN) of the replicator.

replicatorName

string

False

The name of the replicator.

replicatorResourceArn

string

False

The Amazon Resource Name (ARN) of the replicator resource in the region where the replicator was created.

replicatorState

ReplicatorState

False

State of the replicator.

S3

S3 details for ReplicatorLogDelivery.

PropertyTypeRequiredDescription
bucket

string

False

The S3 bucket that is the destination for log delivery.

enabled

boolean

True

Whether log delivery to S3 is enabled.

prefix

string

False

The S3 prefix that is the destination for log delivery.

TargetCompressionType

The type of compression to use producing records to the target cluster.

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

TopicReplication

Details about topic replication.

PropertyTypeRequiredDescription
copyAccessControlListsForTopics

boolean

False

Whether to periodically configure remote topic ACLs to match their corresponding upstream topics.

copyTopicConfigurations

boolean

False

Whether to periodically configure remote topics to match their corresponding upstream topics.

detectAndCopyNewTopics

boolean

False

Whether to periodically check for new topics and partitions.

startingPosition

ReplicationStartingPosition

False

Specifies the position in the topics to start replicating from.

topicNameConfiguration

ReplicationTopicNameConfiguration

False

Configuration for specifying replicated topic names will be the same as their corresponding upstream topics or prefixed with source cluster alias.

topicsToExclude

Array of type string

MaxLength: 249

False

List of regular expression patterns indicating the topics that should not be replicated.

topicsToReplicate

Array of type string

MaxLength: 249

True

List of regular expression patterns indicating the topics to copy.

See also

For more information about using this API in one of the language-specific AWS SDKs and references, see the following:

ListReplicators

CreateReplicator