QLDB stream records in Kinesis

An Amazon QLDB stream writes three types of data records to a given Amazon Kinesis Data Streams resource: control, block summary, and revision details. All three record types are written in the binary representation of the Amazon Ion format.

Control records indicate the start and completion of your QLDB streams. Whenever a revision is committed to your journal, a QLDB stream writes all of the associated journal block data in block summary and revision details records.

The three record types are polymorphic. They all consist of a common top-level record that contains the QLDB stream ARN, the record type, and the record payload. This top-level record has the following format.

{
  qldbStreamArn: string,
  recordType: string,
  payload: {
    //control | block summary | revision details record
  }
}

The recordType field can have one of three values:

The following sections describe the format and contents of each individual payload record.

Control records

A QLDB stream writes control records to indicate its start and completion events. The following are examples of control records with sample data for each controlRecordType:

Block summary records

A block summary record represents a journal block in which your document revisions are committed. A block is an object that is committed to your QLDB journal during a transaction.

The payload of a block summary record contains the block address, timestamp, and other metadata of the transaction that committed the block. It also includes summary attributes of the revisions in the block and the PartiQL statements that committed them. The following is an example of a block summary record with sample data.

{
  qldbStreamArn:"arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy",
  recordType:"BLOCK_SUMMARY",
  payload:{
    blockAddress:{
      strandId:"ElYL30RGoqrFCbbaQn3K6m",
      sequenceNo:60807
    },
    transactionId:"9RWohCo7My4GGkxRETAJ6M",
    blockTimestamp:2019-09-18T17:00:14.601000001Z,
    blockHash:{{6Pk9KDYJd38ci09oaHxx0D2grtgh4QBBqbDS6i9quX8=}},
    entriesHash:{{r5YoH6+NXDXxgoRzPREGAWJfn73KlZE0eTfbTxZWUDU=}},
    previousBlockHash:{{K3ti0Agk7DEponywKcQCPRYVHb5RuyxdmQFTfrloptA=}},
    entriesHashList:[
      {{pbzvz6ofJC7mD2jvgfyrY/VtR01zIZHoWy8T1VcxlGo=}},
      {{k2brC23DLMercmiOWHiURaGwHu0mQtLzdNPuviE2rcs=}},
      {{hvw1EV8k4oOkIO36kblO/+UUSFUQqCanKuDGr0aP9nQ=}},
      {{ZrLbkyzDcpJ9KWsZMZqRuKUkG/czLIJ4US+K5E31b+Q=}}
    ],
    transactionInfo:{
      statements:[
        {
          statement:"SELECT * FROM Person WHERE GovId = ?",
          startTime:2019-09-18T17:00:14.587Z,
          statementDigest:{{p4Dn0DiuYD3Xm9UQQ75YLwmoMbSfJmopOmTfMnXs26M=}}
        },
        {
          statement:"INSERT INTO Person ?",
          startTime:2019-09-18T17:00:14.594Z,
          statementDigest:{{klMLkLfa5VJqk6JUPtHkQpOsDdG4HmuUaq/VaApQflU=}}
        },
        {
          statement:"INSERT INTO VehicleRegistration ?",
          startTime:2019-09-18T17:00:14.598Z,
          statementDigest:{{B0gO9BWVNrzRYFoe7t+GVLpJ6uZcLKf5t/chkfRhspI=}}
        }
      ],
      documents:{
        '7z2OpEBgVCvCtwvx4a2JGn':{
          tableName:"Person",
          tableId:"LSkFkQvkIOjCmpTZpkfpn9",
          statements:[1]
        },
        'K0FpsSLpydLDr7hi6KUzqk':{
          tableName:"VehicleRegistration",
          tableId:"Ad3A07z0ZffC7Gpso7BXyO",
          statements:[2]
        }
      }
    },
    revisionSummaries:[
      {
        hash:{{uDthuiqSy4FwjZssyCiyFd90XoPSlIwomHBdF/OrmkE=}},
        documentId:"7z2OpEBgVCvCtwvx4a2JGn"
      },
      {
        hash:{{qJID/amu0gN3dpG5Tg0FfIFTh/U5yFkfT+g/O6k5sPM=}},
        documentId:"K0FpsSLpydLDr7hi6KUzqk"
      }
    ]
  }
}

In the revisionSummaries field, some revisions might not have a documentId. These are internal-only system revisions that don't contain user data. A QLDB stream includes these revisions in their respective block summary records because the hashes of these revisions are part of the journal's full hash chain. The full hash chain is required for cryptographic verification.

Only the revisions that do have a document ID are published in separate revision details records, as described in the following section.

Revision details records

A revision details record represents a document revision that is committed to your journal. The payload contains all of the attributes from the committed view of the revision, along with the associated table name and table ID. The following is an example of a revision record with sample data.

{
  qldbStreamArn:"arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy",
  recordType:"REVISION_DETAILS",
  payload:{
    tableInfo:{
      tableName:"VehicleRegistration",
      tableId:"Ad3A07z0ZffC7Gpso7BXyO"
    },
    revision:{
      blockAddress:{
        strandId:"ElYL30RGoqrFCbbaQn3K6m",
        sequenceNo:60807
      },
      hash:{{qJID/amu0gN3dpG5Tg0FfIFTh/U5yFkfT+g/O6k5sPM=}},
      data:{
        VIN:"1N4AL11D75C109151",
        LicensePlateNumber:"LEWISR261LL",
        State:"WA",
        City:"Seattle",
        PendingPenaltyTicketAmount:90.25,
        ValidFromDate:2017-08-21,
        ValidToDate:2020-05-11,
        Owners:{
          PrimaryOwner:{PersonId:"7z2OpEBgVCvCtwvx4a2JGn"},
          SecondaryOwners:[]
        }
      },
      metadata:{
        id:"K0FpsSLpydLDr7hi6KUzqk",
        version:0,
        txTime:2019-09-18T17:00:14.602Z,
        txId:"9RWohCo7My4GGkxRETAJ6M"
      }
    }
  }
}

Handling duplicate and out-of-order records

QLDB streams can publish duplicate and out-of-order records to Kinesis Data Streams. So, a consumer application might need to implement its own logic to identify and handle such scenarios. The block summary and revision details records include fields that you can use for this purpose. Combined with the features of downstream services, these fields can indicate both a unique identity and a strict order for the records.

For example, consider a stream that integrates QLDB with an OpenSearch index to provide full text search capabilities over documents. In this use case, you need to avoid indexing stale (out-of-order) revisions of a document. To enforce ordering and deduplication, you can use the document ID and external version fields in OpenSearch, along with the document ID and version fields in a revision details record.

For an example of deduplication logic in a sample application that integrates QLDB with Amazon OpenSearch Service, see the GitHub repository aws-samples/amazon-qldb-streaming-amazon-opensearch-service-sample-python.