Read data from Amazon Kinesis Data Streams
A consumer is an application that processes all data from a Kinesis data stream. When a consumer uses enhanced fan-out, it gets its own 2 MB/sec allotment of read throughput, allowing multiple consumers to read data from the same stream in parallel, without contending for read throughput with other consumers. To use the enhanced fan-out capability of shards, see Develop custom consumers with dedicated throughput (enhanced fan-out).
By default, shards in a stream provide 2 MB/sec of read throughput per shard. This throughput gets shared across all the consumers that are reading from a given shard. In other words, the default 2 MB/sec of throughput per shard is fixed, even if there are multiple consumers that are reading from the shard. To use this default throughput of shards see, Develop custom consumers with shared throughput.
The following table compares default throughput to enhanced fan-out. Message propagation delay is defined as the time taken in milliseconds for a payload sent using the payload-dispatching APIs (like PutRecord and PutRecords) to reach the consumer application through the payload-consuming APIs (like GetRecords and SubscribeToShard).
Characteristics | Unregistered Consumers without Enhanced Fan-Out | Registered Consumers with Enhanced Fan-Out |
---|---|---|
Shard Read Throughput | Fixed at a total of 2 MB/sec per shard. If there are multiple consumers reading from the same shard, they all share this throughput. The sum of the throughputs they receive from the shard doesn't exceed 2 MB/sec. | Scales as consumers register to use enhanced fan-out. Each consumer registered to use enhanced fan-out receives its own read throughput per shard, up to 2 MB/sec, independently of other consumers. |
Message propagation delay | An average of around 200 ms if you have one consumer reading from the stream. This average goes up to around 1000 ms if you have five consumers. | Typically an average of 70 ms whether you have one consumer or five consumers. |
Cost | N/A | There is a data retrieval cost and a consumer-shard hour cost. For more
information, see Amazon Kinesis Data Streams Pricing |
Record delivery model | Pull model over HTTP using GetRecords. | Kinesis Data Streams pushes the records to you over HTTP/2 using SubscribeToShard. |
Topics
- Use the Data Viewer in the Kinesis console
- Query your data streams in the Kinesis console
- Develop consumers using AWS Lambda
- Develop consumers using Amazon Managed Service for Apache Flink
- Develop consumers using Amazon Data Firehose
- Use the Kinesis Client Library
- Develop custom consumers with shared throughput
- Develop custom consumers with dedicated throughput (enhanced fan-out)
- Migrate consumers from KCL 1.x to KCL 2.x
- Read data from Kinesis Data Streams using other AWS services
- Read from Kinesis Data Streams using third-party integrations
- Troubleshoot Kinesis Data Streams consumers
- Optimize Amazon Kinesis Data Streams consumers