View a markdown version of this page

Data storage - AWS CDK Blocks

Data storage

This section covers AWS CDK Blocks for persisting and retrieving data.

Choosing a data Block

Block Best for Avoid when

KVStore

Caches and session stores, and app records that you read by key, list in order inside a group, or read by another field through an index

You need JOINs, transactions, or complex SQL

Database

Relational data, complex queries, JOINs, transactions, migrations, with no virtual private cloud (VPC) to manage. Put it on an aurora cluster for triggers, extensions, Row Level Security, or large transactions

Your workload is mostly key-value lookups, which KVStore serves with no cluster to manage

FileBucket

File uploads, user-generated content, large binary objects

You need structured queries over file metadata (pair with KVStore for metadata)

Feature flags and runtime config for the whole app go in AppSetting, not in a data CDK Block. KVStore is for data your app writes. For more information, see Configuration.

KVStore

Key-value and document storage backed by DynamoDB. Pick one of three forms by how you read the data:

  • No options: a string key and a string value (or a type argument), for caches, sessions, and other data where you always know the key.

  • A schema and indexes: typed values on a string key, for data that you also read by another field.

  • A schema and a key ({ partitionKey, sortKey? }): typed records on an object key, for data that you list or query as a collection, such as the todos of one user. The key is an object with those fields, each a string or a number.

A store fixes its key when you create it, so declare a key for any records that you list. The following example declares a key and lists the todos of one user:

const todos = new KVStore(scope, 'todos', { schema: z.object({ userId: z.string(), todoId: z.string(), title: z.string(), done: z.boolean() }), key: { partitionKey: 'userId', sortKey: 'todoId' }, }); await todos.put({ userId: 'alice', todoId: '001' }, { title: 'Write the docs', done: false }); for await (const { value } of todos.query({ where: { userId: { equals: 'alice' } }, limit: 20 })) { console.log(value.title); }

Every call takes the key first, as in put(key, value, options). query() reads one partition in sort order, by the key or by an index. query() and scan() return { key, value, expiresAt } entries and a cursor for the next page. KVStore also has batch calls and conditional writes, and no transaction API.

get() is strongly consistent by default. Pass { consistentRead: false } to use half the read capacity where a stale value is acceptable. On AWS, query(), scan(), getBatch(), and index reads are eventually consistent.

Locally, KVStore persists data in the .bb-data/ directory. The local implementation is always consistent and does not throttle requests, so test those behaviors in a sandbox. The local implementation applies DynamoDB’s limits as AWS does: the 400 KB item size, measured as DynamoDB measures it, the size limits on key fields, at most 32 levels of nested lists and maps, and no repeated key within one batch request. On AWS, KVStore checks the key, nesting, and batch limits before it calls DynamoDB. Each of these limits throws a KVStoreErrors error in both places, so match those names rather than DynamoDB’s ValidationException. A value that DynamoDB can’t store, such as a Date or NaN, fails locally as it does on AWS. On AWS, it provisions a DynamoDB table with a global secondary index for each entry in indexes. To wrap an existing table, pass table: KVStore.fromExisting(tableName) and a key that matches the table.

KVStore has these durability options:

  • Expiry: write an item with put(key, value, { ttlSeconds }) or { expiresAt }. Every read hides an expired item, and DynamoDB deletes it later. ttl names a number field of the value to hold the expiry.

  • pointInTimeRecovery: continuous backups. On by default with the production preset and off with the sandbox preset.

  • encryption: the AWS managed key (the default), a dedicated customer managed key, or a key you already own.

  • protection: 'disposable', 'retained', or 'locked'. By default, the production preset keeps and protects tables, and the sandbox preset deletes them with the stack.

Upgrading from DistributedTable or an earlier KVStore

DistributedTable is now part of KVStore. To keep a DistributedTable’s deployed table and items, declare it as `new KVStore(scope, id, { schema, key, indexes, ttl }) with the same ID. Then pass the key first to put(), read { key, value } entries from query(), and use KVStoreErrors.

A KVStore deployed with @aws-cdk-blocks/bb-kv-store 0.4.0 or earlier stores items in another format, so the deploy fails if a new KVStore takes over its table. To upgrade, change the class of that store to LegacyKVStore and keep its ID and options. Add a KVStore with a new ID, copy the data to it, and then remove the LegacyKVStore. In a sandbox, you can destroy and redeploy the stack instead. In the options of the new store, don’t use ttl: true, and use protection instead of removalPolicy and deletionProtection. Replace ifValueEquals on an object value with ifFieldEquals. For more information about the upgrade, see the KVStore upgrade steps on GitHub.

For more information, see bb-kv-store on GitHub.

Database

SQL on PostgreSQL, with migrations, transactions, foreign keys, and joins. Each Database is a PostgreSQL schema on a cluster. With no options, the CDK Block owns an Amazon Aurora DSQL cluster, which scales to zero and needs no VPC. Write parameterized queries with the sql tagged template, which turns every ${value} into a bound parameter. The query code is the same on every cluster.

// Owns an Aurora DSQL cluster. Its migrations are in cdk-blocks/migrations/shop/. const shop = new Database(scope, 'shop'); export const api = new ApiNamespace(scope, 'api', (context) => ({ async addToOrder(orderId: number, amount: number) { const { userId } = await auth.requireAuth(context); await shop.transaction(async (tx) => { const order = await tx.queryOne<{ total: number }>( sql`SELECT total FROM orders WHERE id = ${orderId} AND customer_id = ${userId} FOR UPDATE`, ); if (!order) throw new ApiError('Order not found', 404); await tx.execute(sql`UPDATE orders SET total = ${order.total + amount} WHERE id = ${orderId}`); }, { retryOnConflict: true, signal: context.request.signal }); }, }));
Upgrading from the earlier Database and DistributedDatabase

Database replaces the earlier Database and DistributedDatabase Blocks.

A DistributedDatabase keeps its Aurora DSQL cluster and its rows. Declare it as a Database with the same ID, in the same scope and stack, and CloudFormation treats it as the same cluster. Database reads the _migrations table that DistributedDatabase wrote, so keep the migration files and their names. No applied file runs a second time.

Before you deploy, run cdk diff. Check that the cluster (AWS::DSQL::Cluster) shows as changed ([~]), not as removed and added ([-] then [+]). A removed cluster means the ID, the scope, or the stack name is not the one the DistributedDatabase had. Under the sandbox preset’s destroy removal policy, the deploy then deletes that cluster with its rows.

An app that deployed the Database of @aws-blocks/bb-data keeps its Aurora cluster, its secret, and its VPC. Give a LegacyDatabaseCluster the ID and options of the old Block, then pass it as the cluster of a Database with a new ID. That class is deprecated and frozen, and asUser() is an error on it. The old package name keeps its original scope here because the package no longer exists.

asUser() and asRole() replace withRLS(), and crud() has no replacement. An app that used fromExisting() keeps its migration files and _migrations table. Pass DatabaseCluster.fromExisting({ name: 'db-url' }) as cluster, point migrationsPath at the same directory, and no file runs again.

For more information, see Upgrading an older app on GitHub.

Clusters

To put several Database Blocks on one cluster, or to use full PostgreSQL, pass a DatabaseCluster as the cluster option. Each CDK Block on the cluster gets its own schema, named after its ID:

const main = new DatabaseCluster(scope, 'main', { type: 'aurora' }); const notes = new Database(scope, 'notes', { cluster: main });

The cluster type is one of these:

  • 'dsql': Aurora DSQL, the engine that a Database without cluster owns. A transaction can change at most 3,000 rows and 10 MiB of data. Triggers, extensions, and Row Level Security are not available.

  • 'aurora': Amazon Aurora PostgreSQL Serverless v2 with the express configuration. It has all of PostgreSQL, runs outside any VPC with IAM authentication, and scales to zero by default (minCapacity: 0). The first request after five idle minutes then waits about 20 seconds. Set minCapacity: 0.5 to keep it warm, and maxCapacity (default 2) to raise the ceiling.

Schemas separate CDK Blocks by name, not by permission: every CDK Block on a cluster connects as one database role that can read and write every schema on it. Put a CDK Block whose data must stay separate on its own cluster.

The data stays on the cluster that a CDK Block first deploys to. If you rename a Database or DatabaseCluster ID, or change the cluster option, the CDK Block points at a new, empty cluster or schema, and nothing copies the rows. With the production preset, a cluster that leaves the stack is retained. With the sandbox preset, it is deleted in the same deploy.

Migrations

Create tables in migration files. Your app’s own statements can only read and write rows. Each CDK Block runs the .sql files in cdk-blocks/migrations/<id>/, relative to the project root, in name order and once each. migrationsPath sets another directory. Locally, a CDK Block applies pending files before its first query. On AWS, they run during the deploy, before any function in the stack is updated.

-- cdk-blocks/migrations/shop/001_init.sql CREATE TABLE orders ( id BIGINT GENERATED BY DEFAULT AS IDENTITY (CACHE 65536) PRIMARY KEY, customer_id TEXT NOT NULL, total INTEGER NOT NULL ); CREATE INDEX ASYNC orders_customer_id ON orders (customer_id);

On a dsql cluster, write the DSQL forms that the Aurora DSQL documentation shows, such as CREATE INDEX ASYNC. CDK Blocks checks every file with dsql-lint at npm run dev and at cdk synth, and the error names the file and the form to use instead. A deploy that stops in a file resumes it where it stopped, so don’t edit a file that has partly run. Put the change in a new file. On an aurora cluster, write plain PostgreSQL. Each file runs in one transaction and, on AWS, must finish within ten minutes.

Queries and transactions

query() returns every row, queryOne() the first row or null, and execute() returns { rowCount }. Pass a row type, as in query<T>(), or pass a Standard Schema such as a Zod object as { schema } to validate every row. A bigint value, such as an identity column, comes back as a JavaScript number.

transaction(fn) commits when fn resolves and rolls back when it throws. An ApiError thrown in fn reaches the client unchanged. Aurora DSQL never waits on a lock: when two transactions change the same row, one fails at commit with SerializationFailure. With retryOnConflict: true, the CDK Block runs fn again after a random wait, up to maxRetries times (default 8), so keep side effects such as sending email out of fn. Pass context.request.signal as signal so that no retry commits after the request times out.

For the Kysely query builder, install kysely and pass new DatabaseDialect(db) from @aws-cdk-blocks/bb-database/kysely as its dialect. –0— writes a Tables interface from the CDK Block’s migration files.

Row level access

On an aurora cluster, Row Level Security policies that you write in a migration can limit each user to their own rows. db.asUser(user) returns the CDK Block with each transaction run for that user: it first sets request.jwt.claims to { sub: user.userId }, plus any claims you pass as the second argument. Pass the result of requireAuth() as user.

-- cdk-blocks/migrations/notes/001_notes.sql CREATE TABLE notes (id SERIAL PRIMARY KEY, owner_id TEXT NOT NULL, body TEXT NOT NULL); ALTER TABLE notes ENABLE ROW LEVEL SECURITY; CREATE POLICY notes_owner ON notes USING (owner_id = current_setting('request.jwt.claims', true)::jsonb ->> 'sub');

With this policy, +notes.asUser(user).query(sql`SELECT id, body FROM notes)+` returns only that user’s notes, and the same statement on the CDK Block itself returns none. For policies written for a database role, such as TO authenticated, use asRole(role, claims?). Both throw DatabaseConfigurationError on a dsql cluster.

Existing PostgreSQL databases

To use a PostgreSQL database that you already run, with its tables and data, pass cluster: DatabaseCluster.fromExisting({ name: 'db-url', ca }) and migrationsPath: './migrations'. The CDK Block provisions nothing, and a Lambda function outside any VPC must be able to reach the database over IPv4 with TLS. The CDK Block always verifies the server certificate, against ca (PEM text) or the Node.js trust store.

Never write the connection string in code. –0— reads it from a hidden prompt, checks the database, and stores it for local mode. Add --stage production to store it for a stage as the SSM SecureString /<stackName>-db-url. You can pass your own Secret as connectionString instead of name.

Locally, the CDK Block connects to that database and runs no migration files, so use a development database. –0— applies the pending files in ./migrations to it, and a deploy applies them to the stage, one transaction per file. –1— records a file, such as a dump of the existing schema, without running it.

Local mode

Locally, each cluster is one PGlite database (PostgreSQL compiled to WebAssembly) in .bb-data/, with one schema per CDK Block. A dsql cluster applies the DSQL checks and limits, so SQL that Aurora DSQL rejects fails on your machine. Statements run with the grants of the deployed app, and Row Level Security applies as on AWS. PGlite runs one statement or transaction at a time. To test the retry path, call simulateConflict(db) from @aws-cdk-blocks/bb-database/testing.

Database errors

Common request errors reach the client as named 4xx errors instead of a generic 500. Check for them with isBlocksError() on the server or the client, for example isBlocksError(e, DatabaseErrors.UniqueConstraintViolation).

Error Cause HTTP status

DatabaseErrors.PermissionDenied

A Row Level Security policy rejected a statement from asUser() or asRole()

403

DatabaseErrors.CheckViolation

A value fails a CHECK constraint

400

DatabaseErrors.NotNullViolation

A required column is missing

400

DatabaseErrors.UniqueConstraintViolation

A duplicate key

409

DatabaseErrors.ForeignKeyViolation

The referenced row does not exist, or other rows still reference the row being deleted

409

DatabaseErrors.SerializationFailure

A conflict with a concurrent transaction. Safe to retry.

409

Other failures reach the client as status 500, with the name QueryFailed (any other SQL error, including DDL outside a migration), ConnectionFailed, TransactionFailed, TransactionRowLimitExceeded, or ResultValidationFailed (a row did not match schema).

For more information, see bb-database on GitHub.

FileBucket

File storage for uploads and downloads. Store files by path, generate presigned URLs for direct browser uploads and downloads, and list files by prefix. Versioning is on by default, so you can list and restore earlier versions of a file. Noncurrent versions expire after 90 days. Change this with noncurrentVersionExpirationDays.

Browser uploads and downloads that use presigned URLs need CORS rules after you deploy. A FileBucket has no CORS rules by default, and it does not use your stack’s allowed origins. The local dev server accepts any origin, so a missing rule fails only after deploy. List your frontend origin in corsRules. A sandbox frontend on http://localhost:3000 also calls the deployed bucket, so add that origin for sandboxes only, for example with getDeploymentType():

const origins = ['https://app.example.com']; if (getDeploymentType() === 'sandbox') origins.push('http://localhost:3000'); const uploads = new FileBucket(scope, 'uploads', { corsRules: [{ allowedOrigins: origins, allowedMethods: ['GET', 'PUT'], allowedHeaders: ['Content-Type'], }], });

A rule with the wildcard origin '*' cannot allow PUT, POST, or DELETE. CDK Blocks rejects it.

FileBucket also has these security options:

  • encryption: S3 managed keys (the default), a dedicated customer managed AWS KMS key ('customer-managed'), or a key you already own (FileBucket.fromKmsKey(keyArn)).

  • objectLock: write-once-read-many retention, for example { mode: 'governance', retentionDays: 365 }. Decide when you create the bucket, because you cannot add Object Lock to a deployed bucket or turn it off. The bucket must be retained, and presigned uploads are not supported, so upload with put().

  • accessLogging: S3 server access logs in a separate log bucket. Off by default. To turn it on for every FileBucket, set storageAccessLogging: true in your stack’s defaults. defaults.accessLogging: true also turns it on, unless storageAccessLogging is false.

Locally, FileBucket stores files in the .bb-data/ directory on your filesystem, mirroring the S3 API behavior. On AWS, it provisions an S3 bucket. You can also wrap an existing S3 bucket with FileBucket.fromExisting(bucketName). A wrapped bucket gets none of the FileBucket defaults, such as versioning, encryption, or blocked public access, and FileBucket ignores corsRules for it. Configure those on the bucket itself.

For more information, see bb-file-bucket on GitHub.