# Data key caching example
This example uses [data key caching](data-key-caching.md) with a [local cache](data-caching-details.md#simplecache) to speed up an application in which data generated by multiple devices is encrypted and stored in different Regions.
In this scenario, multiple data producers generate data, encrypt it, and write to a [Kinesis stream](https://aws.amazon.com/kinesis/streams/) in each Region. [AWS Lambda](https://aws.amazon.com/lambda/) functions (consumers) decrypt the streams and write plaintext data to a DynamoDB table in the Region. Data producers and consumers use the AWS Encryption SDK and an [AWS KMS master key provider](concepts.md#master-key-provider). To reduce calls to KMS, each producer and consumer has their own local cache.
You can find the source code for these examples in [Java and Python](sample-cache-example-code.md). The sample also includes a CloudFormation template that defines the resources for the samples.
![\[This diagram shows how data producers and consumers use the AWS KMS, Amazon Kinesis Data Streams, and Amazon DynamoDB.\]](http://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/images/simplecache-example.png)
## Local cache results
The following table shows that a local cache reduces the total calls to KMS (per second per Region) in this example to 1% of its original value.
**Producer requests**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/sample-cache-example.html)
**Consumer requests**
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/sample-cache-example.html)
# Data key caching example code
This code sample creates a simple implementation of data key caching with a [local cache](data-caching-details.md#simplecache) in Java and Python. The code creates two instances of a local cache: one for [data producers](#caching-producer) that are encrypting data and another for [data consumers](#caching-consumer) (AWS Lambda functions) that are decrypting data. For details about the implementation of data key caching in each language, see the [Javadoc](https://aws.github.io/aws-encryption-sdk-java/) and [Python documentation](https://aws-encryption-sdk-python.readthedocs.io/en/latest/) for the AWS Encryption SDK.
Data key caching is available for all [programming languages](programming-languages.md) that the AWS Encryption SDK supports.
For complete and tested examples of using data key caching in the AWS Encryption SDK, see:
+ C/C\$1\$1: [caching\$1cmm.cpp](https://github.com/aws/aws-encryption-sdk-c/blob/master/examples/caching_cmm.cpp)
+ Java: [SimpleDataKeyCachingExample.java](https://github.com/aws/aws-encryption-sdk-java/blob/master/src/examples/java/com/amazonaws/crypto/examples/v2/SimpleDataKeyCachingExample.java)
+ JavaScript Browser: [caching\$1cmm.ts](https://github.com/aws/aws-encryption-sdk-javascript/blob/master/modules/example-browser/src/caching_cmm.ts)
+ JavaScript Node.js: [caching\$1cmm.ts](https://github.com/aws/aws-encryption-sdk-javascript/blob/master/modules/example-node/src/caching_cmm.ts)
+ Python: [data\$1key\$1caching\$1basic.py](https://github.com/aws/aws-encryption-sdk-python/blob/master/examples/src/legacy/data_key_caching_basic.py)
## Producer
The producer gets a map, converts it to JSON, uses the AWS Encryption SDK to encrypt it, and pushes the ciphertext record to a [Kinesis stream](https://aws.amazon.com/kinesis/streams/) in each AWS Region.
The code defines a [caching cryptographic materials manager](data-caching-details.md#caching-cmm) (caching CMM) and associates it with a [local cache](data-caching-details.md#simplecache) and an underlying [AWS KMS master key provider](concepts.md#master-key-provider). The caching CMM caches the data keys (and [related cryptographic materials](data-caching-details.md#cache-entries)) from the master key provider. It also interacts with the cache on behalf of the SDK and enforces security thresholds that you set.
Because the call to the encrypt method specifies a caching CMM, instead of a regular [cryptographic materials manager (CMM)](concepts.md#crypt-materials-manager) or master key provider, the encryption will use data key caching.
------
#### [ Java ]
The following example uses version 2.*x* of the AWS Encryption SDK for Java. Version 3.*x* of the AWS Encryption SDK for Java deprecates the data key caching CMM. With version 3.*x*, you can also use the [AWS KMS Hierarchical keyring](use-hierarchical-keyring.md), an alternative cryptographic materials caching solution.
```
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except
* in compliance with the License. A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.amazonaws.crypto.examples.kinesisdatakeycaching;
import com.amazonaws.encryptionsdk.AwsCrypto;
import com.amazonaws.encryptionsdk.CommitmentPolicy;
import com.amazonaws.encryptionsdk.CryptoResult;
import com.amazonaws.encryptionsdk.MasterKeyProvider;
import com.amazonaws.encryptionsdk.caching.CachingCryptoMaterialsManager;
import com.amazonaws.encryptionsdk.caching.LocalCryptoMaterialsCache;
import com.amazonaws.encryptionsdk.kmssdkv2.KmsMasterKey;
import com.amazonaws.encryptionsdk.kmssdkv2.KmsMasterKeyProvider;
import com.amazonaws.encryptionsdk.multi.MultipleProviderFactory;
import com.amazonaws.util.json.Jackson;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.core.SdkBytes;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.kinesis.KinesisClient;
import software.amazon.awssdk.services.kms.KmsClient;
/**
* Pushes data to Kinesis Streams in multiple Regions.
*/
public class MultiRegionRecordPusher {
private static final long MAX_ENTRY_AGE_MILLISECONDS = 300000;
private static final long MAX_ENTRY_USES = 100;
private static final int MAX_CACHE_ENTRIES = 100;
private final String streamName_;
private final ArrayList kinesisClients_;
private final CachingCryptoMaterialsManager cachingMaterialsManager_;
private final AwsCrypto crypto_;
/**
* Creates an instance of this object with Kinesis clients for all target Regions and a cached
* key provider containing KMS master keys in all target Regions.
*/
public MultiRegionRecordPusher(final Region[] regions, final String kmsAliasName,
final String streamName) {
streamName_ = streamName;
crypto_ = AwsCrypto.builder()
.withCommitmentPolicy(CommitmentPolicy.RequireEncryptRequireDecrypt)
.build();
kinesisClients_ = new ArrayList<>();
AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.builder().build();
// Build KmsMasterKey and AmazonKinesisClient objects for each target region
List masterKeys = new ArrayList<>();
for (Region region : regions) {
kinesisClients_.add(KinesisClient.builder()
.credentialsProvider(credentialsProvider)
.region(region)
.build());
KmsMasterKey regionMasterKey = KmsMasterKeyProvider.builder()
.defaultRegion(region)
.builderSupplier(() -> KmsClient.builder().credentialsProvider(credentialsProvider))
.buildStrict(kmsAliasName)
.getMasterKey(kmsAliasName);
masterKeys.add(regionMasterKey);
}
// Collect KmsMasterKey objects into single provider and add cache
MasterKeyProvider masterKeyProvider = MultipleProviderFactory.buildMultiProvider(
KmsMasterKey.class,
masterKeys
);
cachingMaterialsManager_ = CachingCryptoMaterialsManager.newBuilder()
.withMasterKeyProvider(masterKeyProvider)
.withCache(new LocalCryptoMaterialsCache(MAX_CACHE_ENTRIES))
.withMaxAge(MAX_ENTRY_AGE_MILLISECONDS, TimeUnit.MILLISECONDS)
.withMessageUseLimit(MAX_ENTRY_USES)
.build();
}
/**
* JSON serializes and encrypts the received record data and pushes it to all target streams.
*/
public void putRecord(final Map data) {
String partitionKey = UUID.randomUUID().toString();
Map encryptionContext = new HashMap<>();
encryptionContext.put("stream", streamName_);
// JSON serialize data
String jsonData = Jackson.toJsonString(data);
// Encrypt data
CryptoResult result = crypto_.encryptData(
cachingMaterialsManager_,
jsonData.getBytes(),
encryptionContext
);
byte[] encryptedData = result.getResult();
// Put records to Kinesis stream in all Regions
for (KinesisClient regionalKinesisClient : kinesisClients_) {
regionalKinesisClient.putRecord(builder ->
builder.streamName(streamName_)
.data(SdkBytes.fromByteArray(encryptedData))
.partitionKey(partitionKey));
}
}
}
```
------
#### [ Python ]
```
"""
Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except
in compliance with the License. A copy of the License is located at
https://aws.amazon.com/apache-2-0/
or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
"""
import json
import uuid
from aws_encryption_sdk import EncryptionSDKClient, StrictAwsKmsMasterKeyProvider, CachingCryptoMaterialsManager, LocalCryptoMaterialsCache, CommitmentPolicy
from aws_encryption_sdk.key_providers.kms import KMSMasterKey
import boto3
class MultiRegionRecordPusher(object):
"""Pushes data to Kinesis Streams in multiple Regions."""
CACHE_CAPACITY = 100
MAX_ENTRY_AGE_SECONDS = 300.0
MAX_ENTRY_MESSAGES_ENCRYPTED = 100
def __init__(self, regions, kms_alias_name, stream_name):
self._kinesis_clients = []
self._stream_name = stream_name
# Set up EncryptionSDKClient
_client = EncryptionSDKClient(CommitmentPolicy.REQUIRE_ENCRYPT_REQUIRE_DECRYPT)
# Set up KMSMasterKeyProvider with cache
_key_provider = StrictAwsKmsMasterKeyProvider(kms_alias_name)
# Add MasterKey and Kinesis client for each Region
for region in regions:
self._kinesis_clients.append(boto3.client('kinesis', region_name=region))
regional_master_key = KMSMasterKey(
client=boto3.client('kms', region_name=region),
key_id=kms_alias_name
)
_key_provider.add_master_key_provider(regional_master_key)
cache = LocalCryptoMaterialsCache(capacity=self.CACHE_CAPACITY)
self._materials_manager = CachingCryptoMaterialsManager(
master_key_provider=_key_provider,
cache=cache,
max_age=self.MAX_ENTRY_AGE_SECONDS,
max_messages_encrypted=self.MAX_ENTRY_MESSAGES_ENCRYPTED
)
def put_record(self, record_data):
"""JSON serializes and encrypts the received record data and pushes it to all target streams.
:param dict record_data: Data to write to stream
"""
# Kinesis partition key to randomize write load across stream shards
partition_key = uuid.uuid4().hex
encryption_context = {'stream': self._stream_name}
# JSON serialize data
json_data = json.dumps(record_data)
# Encrypt data
encrypted_data, _header = _client.encrypt(
source=json_data,
materials_manager=self._materials_manager,
encryption_context=encryption_context
)
# Put records to Kinesis stream in all Regions
for client in self._kinesis_clients:
client.put_record(
StreamName=self._stream_name,
Data=encrypted_data,
PartitionKey=partition_key
)
```
------
## Consumer
The data consumer is an [AWS Lambda](https://aws.amazon.com/lambda/) function that is triggered by [Kinesis](https://aws.amazon.com/kinesis/) events. It decrypts and deserializes each record, and writes the plaintext record to an [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) table in the same Region.
Like the producer code, the consumer code enables data key caching by using a caching cryptographic materials manager (caching CMM) in calls to the decrypt method.
The Java code builds a master key provider in *strict mode* with a specified AWS KMS key. Strict mode isn't required when decrypting, but it's a [best practice](best-practices.md#strict-discovery-mode). The Python code uses *discovery mode*, which lets the AWS Encryption SDK use any wrapping key that encrypted a data key to decrypt it.
------
#### [ Java ]
The following example uses version 2.*x* of the AWS Encryption SDK for Java. Version 3.*x* of the AWS Encryption SDK for Java deprecates the data key caching CMM. With version 3.*x*, you can also use the [AWS KMS Hierarchical keyring](use-hierarchical-keyring.md), an alternative cryptographic materials caching solution.
This code creates a master key provider for decrypting in strict mode. The AWS Encryption SDK can use only the AWS KMS keys you specify to decrypt your message.
```
/*
* Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except
* in compliance with the License. A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.amazonaws.crypto.examples.kinesisdatakeycaching;
import com.amazonaws.encryptionsdk.AwsCrypto;
import com.amazonaws.encryptionsdk.CommitmentPolicy;
import com.amazonaws.encryptionsdk.CryptoResult;
import com.amazonaws.encryptionsdk.caching.CachingCryptoMaterialsManager;
import com.amazonaws.encryptionsdk.caching.LocalCryptoMaterialsCache;
import com.amazonaws.encryptionsdk.kmssdkv2.KmsMasterKeyProvider;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.events.KinesisEvent;
import com.amazonaws.services.lambda.runtime.events.KinesisEvent.KinesisEventRecord;
import com.amazonaws.util.BinaryUtils;
import java.io.UnsupportedEncodingException;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.TimeUnit;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
/**
* Decrypts all incoming Kinesis records and writes records to DynamoDB.
*/
public class LambdaDecryptAndWrite {
private static final long MAX_ENTRY_AGE_MILLISECONDS = 600000;
private static final int MAX_CACHE_ENTRIES = 100;
private final CachingCryptoMaterialsManager cachingMaterialsManager_;
private final AwsCrypto crypto_;
private final DynamoDbTable- table_;
/**
* Because the cache is used only for decryption, the code doesn't set the max bytes or max
* message security thresholds that are enforced only on on data keys used for encryption.
*/
public LambdaDecryptAndWrite() {
String kmsKeyArn = System.getenv("CMK_ARN");
cachingMaterialsManager_ = CachingCryptoMaterialsManager.newBuilder()
.withMasterKeyProvider(KmsMasterKeyProvider.builder().buildStrict(kmsKeyArn))
.withCache(new LocalCryptoMaterialsCache(MAX_CACHE_ENTRIES))
.withMaxAge(MAX_ENTRY_AGE_MILLISECONDS, TimeUnit.MILLISECONDS)
.build();
crypto_ = AwsCrypto.builder()
.withCommitmentPolicy(CommitmentPolicy.RequireEncryptRequireDecrypt)
.build();
String tableName = System.getenv("TABLE_NAME");
DynamoDbEnhancedClient dynamodb = DynamoDbEnhancedClient.builder().build();
table_ = dynamodb.table(tableName, TableSchema.fromClass(Item.class));
}
/**
* @param event
* @param context
*/
public void handleRequest(KinesisEvent event, Context context)
throws UnsupportedEncodingException {
for (KinesisEventRecord record : event.getRecords()) {
ByteBuffer ciphertextBuffer = record.getKinesis().getData();
byte[] ciphertext = BinaryUtils.copyAllBytesFrom(ciphertextBuffer);
// Decrypt and unpack record
CryptoResult plaintextResult = crypto_.decryptData(cachingMaterialsManager_,
ciphertext);
// Verify the encryption context value
String streamArn = record.getEventSourceARN();
String streamName = streamArn.substring(streamArn.indexOf("/") + 1);
if (!streamName.equals(plaintextResult.getEncryptionContext().get("stream"))) {
throw new IllegalStateException("Wrong Encryption Context!");
}
// Write record to DynamoDB
String jsonItem = new String(plaintextResult.getResult(), StandardCharsets.UTF_8);
System.out.println(jsonItem);
table_.putItem(Item.fromJSON(jsonItem));
}
}
private static class Item {
static Item fromJSON(String jsonText) {
// Parse JSON and create new Item
return new Item();
}
}
}
```
------
#### [ Python ]
This Python code decrypts with a master key provider in discovery mode. It lets the AWS Encryption SDK use any wrapping key that encrypted a data key to decrypt it. Strict mode, in which you specify the wrapping keys that can be used for decryption, is a [best practice](best-practices.md#strict-discovery-mode).
```
"""
Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except
in compliance with the License. A copy of the License is located at
https://aws.amazon.com/apache-2-0/
or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
"""
import base64
import json
import logging
import os
from aws_encryption_sdk import EncryptionSDKClient, DiscoveryAwsKmsMasterKeyProvider, CachingCryptoMaterialsManager, LocalCryptoMaterialsCache, CommitmentPolicy
import boto3
_LOGGER = logging.getLogger(__name__)
_is_setup = False
CACHE_CAPACITY = 100
MAX_ENTRY_AGE_SECONDS = 600.0
def setup():
"""Sets up clients that should persist across Lambda invocations."""
global encryption_sdk_client
encryption_sdk_client = EncryptionSDKClient(CommitmentPolicy.REQUIRE_ENCRYPT_REQUIRE_DECRYPT)
global materials_manager
key_provider = DiscoveryAwsKmsMasterKeyProvider()
cache = LocalCryptoMaterialsCache(capacity=CACHE_CAPACITY)
# Because the cache is used only for decryption, the code doesn't set
# the max bytes or max message security thresholds that are enforced
# only on on data keys used for encryption.
materials_manager = CachingCryptoMaterialsManager(
master_key_provider=key_provider,
cache=cache,
max_age=MAX_ENTRY_AGE_SECONDS
)
global table
table_name = os.environ.get('TABLE_NAME')
table = boto3.resource('dynamodb').Table(table_name)
global _is_setup
_is_setup = True
def lambda_handler(event, context):
"""Decrypts all incoming Kinesis records and writes records to DynamoDB."""
_LOGGER.debug('New event:')
_LOGGER.debug(event)
if not _is_setup:
setup()
with table.batch_writer() as batch:
for record in event.get('Records', []):
# Record data base64-encoded by Kinesis
ciphertext = base64.b64decode(record['kinesis']['data'])
# Decrypt and unpack record
plaintext, header = encryption_sdk_client.decrypt(
source=ciphertext,
materials_manager=materials_manager
)
item = json.loads(plaintext)
# Verify the encryption context value
stream_name = record['eventSourceARN'].split('/', 1)[1]
if stream_name != header.encryption_context['stream']:
raise ValueError('Wrong Encryption Context!')
# Write record to DynamoDB
batch.put_item(Item=item)
```
------
# Data key caching example: CloudFormation template
This CloudFormation template sets up all the necessary AWS resources to reproduce the [data key caching example](sample-cache-example.md).
------
#### [ JSON ]
```
{
"Parameters": {
"SourceCodeBucket": {
"Type": "String",
"Description": "S3 bucket containing Lambda source code zip files"
},
"PythonLambdaS3Key": {
"Type": "String",
"Description": "S3 key containing Python Lambda source code zip file"
},
"PythonLambdaObjectVersionId": {
"Type": "String",
"Description": "S3 version id for S3 key containing Python Lambda source code zip file"
},
"JavaLambdaS3Key": {
"Type": "String",
"Description": "S3 key containing Python Lambda source code zip file"
},
"JavaLambdaObjectVersionId": {
"Type": "String",
"Description": "S3 version id for S3 key containing Python Lambda source code zip file"
},
"KeyAliasSuffix": {
"Type": "String",
"Description": "Suffix to use for KMS key Alias (ie: alias/KeyAliasSuffix)"
},
"StreamName": {
"Type": "String",
"Description": "Name to use for Kinesis Stream"
}
},
"Resources": {
"InputStream": {
"Type": "AWS::Kinesis::Stream",
"Properties": {
"Name": {
"Ref": "StreamName"
},
"ShardCount": 2
}
},
"PythonLambdaOutputTable": {
"Type": "AWS::DynamoDB::Table",
"Properties": {
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "id",
"KeyType": "HASH"
}
],
"ProvisionedThroughput": {
"ReadCapacityUnits": 1,
"WriteCapacityUnits": 1
}
}
},
"PythonLambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
],
"Policies": [
{
"PolicyName": "PythonLambdaAccess",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DescribeTable",
"dynamodb:BatchWriteItem"
],
"Resource": {
"Fn::Sub": "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${PythonLambdaOutputTable}"
}
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem"
],
"Resource": {
"Fn::Sub": "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${PythonLambdaOutputTable}*"
}
},
{
"Effect": "Allow",
"Action": [
"kinesis:GetRecords",
"kinesis:GetShardIterator",
"kinesis:DescribeStream",
"kinesis:ListStreams"
],
"Resource": {
"Fn::Sub": "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}"
}
}
]
}
}
]
}
},
"PythonLambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Description": "Python consumer",
"Runtime": "python2.7",
"MemorySize": 512,
"Timeout": 90,
"Role": {
"Fn::GetAtt": [
"PythonLambdaRole",
"Arn"
]
},
"Handler": "aws_crypto_examples.kinesis_datakey_caching.consumer.lambda_handler",
"Code": {
"S3Bucket": {
"Ref": "SourceCodeBucket"
},
"S3Key": {
"Ref": "PythonLambdaS3Key"
},
"S3ObjectVersion": {
"Ref": "PythonLambdaObjectVersionId"
}
},
"Environment": {
"Variables": {
"TABLE_NAME": {
"Ref": "PythonLambdaOutputTable"
}
}
}
}
},
"PythonLambdaSourceMapping": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"BatchSize": 1,
"Enabled": true,
"EventSourceArn": {
"Fn::Sub": "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}"
},
"FunctionName": {
"Ref": "PythonLambdaFunction"
},
"StartingPosition": "TRIM_HORIZON"
}
},
"JavaLambdaOutputTable": {
"Type": "AWS::DynamoDB::Table",
"Properties": {
"AttributeDefinitions": [
{
"AttributeName": "id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "id",
"KeyType": "HASH"
}
],
"ProvisionedThroughput": {
"ReadCapacityUnits": 1,
"WriteCapacityUnits": 1
}
}
},
"JavaLambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
],
"Policies": [
{
"PolicyName": "JavaLambdaAccess",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DescribeTable",
"dynamodb:BatchWriteItem"
],
"Resource": {
"Fn::Sub": "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${JavaLambdaOutputTable}"
}
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem"
],
"Resource": {
"Fn::Sub": "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${JavaLambdaOutputTable}*"
}
},
{
"Effect": "Allow",
"Action": [
"kinesis:GetRecords",
"kinesis:GetShardIterator",
"kinesis:DescribeStream",
"kinesis:ListStreams"
],
"Resource": {
"Fn::Sub": "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}"
}
}
]
}
}
]
}
},
"JavaLambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Description": "Java consumer",
"Runtime": "java8",
"MemorySize": 512,
"Timeout": 90,
"Role": {
"Fn::GetAtt": [
"JavaLambdaRole",
"Arn"
]
},
"Handler": "com.amazonaws.crypto.examples.kinesisdatakeycaching.LambdaDecryptAndWrite::handleRequest",
"Code": {
"S3Bucket": {
"Ref": "SourceCodeBucket"
},
"S3Key": {
"Ref": "JavaLambdaS3Key"
},
"S3ObjectVersion": {
"Ref": "JavaLambdaObjectVersionId"
}
},
"Environment": {
"Variables": {
"TABLE_NAME": {
"Ref": "JavaLambdaOutputTable"
},
"CMK_ARN": {
"Fn::GetAtt": [
"RegionKinesisCMK",
"Arn"
]
}
}
}
}
},
"JavaLambdaSourceMapping": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"BatchSize": 1,
"Enabled": true,
"EventSourceArn": {
"Fn::Sub": "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}"
},
"FunctionName": {
"Ref": "JavaLambdaFunction"
},
"StartingPosition": "TRIM_HORIZON"
}
},
"RegionKinesisCMK": {
"Type": "AWS::KMS::Key",
"Properties": {
"Description": "Used to encrypt data passing through Kinesis Stream in this region",
"Enabled": true,
"KeyPolicy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": {
"Fn::Sub": "arn:aws:iam::${AWS::AccountId}:root"
}
},
"Action": [
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:CreateAlias",
"kms:DeleteAlias",
"kms:DescribeKey",
"kms:DisableKey",
"kms:EnableKey",
"kms:PutKeyPolicy",
"kms:ScheduleKeyDeletion",
"kms:UpdateAlias",
"kms:UpdateKeyDescription"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": [
{
"Fn::GetAtt": [
"PythonLambdaRole",
"Arn"
]
},
{
"Fn::GetAtt": [
"JavaLambdaRole",
"Arn"
]
}
]
},
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
}
},
"RegionKinesisCMKAlias": {
"Type": "AWS::KMS::Alias",
"Properties": {
"AliasName": {
"Fn::Sub": "alias/${KeyAliasSuffix}"
},
"TargetKeyId": {
"Ref": "RegionKinesisCMK"
}
}
}
}
}
```
------
#### [ YAML ]
```
Parameters:
SourceCodeBucket:
Type: String
Description: S3 bucket containing Lambda source code zip files
PythonLambdaS3Key:
Type: String
Description: S3 key containing Python Lambda source code zip file
PythonLambdaObjectVersionId:
Type: String
Description: S3 version id for S3 key containing Python Lambda source code zip file
JavaLambdaS3Key:
Type: String
Description: S3 key containing Python Lambda source code zip file
JavaLambdaObjectVersionId:
Type: String
Description: S3 version id for S3 key containing Python Lambda source code zip file
KeyAliasSuffix:
Type: String
Description: 'Suffix to use for KMS CMK Alias (ie: alias/)'
StreamName:
Type: String
Description: Name to use for Kinesis Stream
Resources:
InputStream:
Type: AWS::Kinesis::Stream
Properties:
Name: !Ref StreamName
ShardCount: 2
PythonLambdaOutputTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
-
AttributeName: id
AttributeType: S
KeySchema:
-
AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
PythonLambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
-
PolicyName: PythonLambdaAccess
PolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Action:
- dynamodb:DescribeTable
- dynamodb:BatchWriteItem
Resource: !Sub arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${PythonLambdaOutputTable}
-
Effect: Allow
Action:
- dynamodb:PutItem
Resource: !Sub arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${PythonLambdaOutputTable}*
-
Effect: Allow
Action:
- kinesis:GetRecords
- kinesis:GetShardIterator
- kinesis:DescribeStream
- kinesis:ListStreams
Resource: !Sub arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}
PythonLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Description: Python consumer
Runtime: python2.7
MemorySize: 512
Timeout: 90
Role: !GetAtt PythonLambdaRole.Arn
Handler: aws_crypto_examples.kinesis_datakey_caching.consumer.lambda_handler
Code:
S3Bucket: !Ref SourceCodeBucket
S3Key: !Ref PythonLambdaS3Key
S3ObjectVersion: !Ref PythonLambdaObjectVersionId
Environment:
Variables:
TABLE_NAME: !Ref PythonLambdaOutputTable
PythonLambdaSourceMapping:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: 1
Enabled: true
EventSourceArn: !Sub arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}
FunctionName: !Ref PythonLambdaFunction
StartingPosition: TRIM_HORIZON
JavaLambdaOutputTable:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
-
AttributeName: id
AttributeType: S
KeySchema:
-
AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
JavaLambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
-
PolicyName: JavaLambdaAccess
PolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Action:
- dynamodb:DescribeTable
- dynamodb:BatchWriteItem
Resource: !Sub arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${JavaLambdaOutputTable}
-
Effect: Allow
Action:
- dynamodb:PutItem
Resource: !Sub arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${JavaLambdaOutputTable}*
-
Effect: Allow
Action:
- kinesis:GetRecords
- kinesis:GetShardIterator
- kinesis:DescribeStream
- kinesis:ListStreams
Resource: !Sub arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}
JavaLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Description: Java consumer
Runtime: java8
MemorySize: 512
Timeout: 90
Role: !GetAtt JavaLambdaRole.Arn
Handler: com.amazonaws.crypto.examples.kinesisdatakeycaching.LambdaDecryptAndWrite::handleRequest
Code:
S3Bucket: !Ref SourceCodeBucket
S3Key: !Ref JavaLambdaS3Key
S3ObjectVersion: !Ref JavaLambdaObjectVersionId
Environment:
Variables:
TABLE_NAME: !Ref JavaLambdaOutputTable
CMK_ARN: !GetAtt RegionKinesisCMK.Arn
JavaLambdaSourceMapping:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: 1
Enabled: true
EventSourceArn: !Sub arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/${InputStream}
FunctionName: !Ref JavaLambdaFunction
StartingPosition: TRIM_HORIZON
RegionKinesisCMK:
Type: AWS::KMS::Key
Properties:
Description: Used to encrypt data passing through Kinesis Stream in this region
Enabled: true
KeyPolicy:
Version: 2012-10-17
Statement:
-
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action:
# Data plane actions
- kms:Encrypt
- kms:GenerateDataKey
# Control plane actions
- kms:CreateAlias
- kms:DeleteAlias
- kms:DescribeKey
- kms:DisableKey
- kms:EnableKey
- kms:PutKeyPolicy
- kms:ScheduleKeyDeletion
- kms:UpdateAlias
- kms:UpdateKeyDescription
Resource: '*'
-
Effect: Allow
Principal:
AWS:
- !GetAtt PythonLambdaRole.Arn
- !GetAtt JavaLambdaRole.Arn
Action: kms:Decrypt
Resource: '*'
RegionKinesisCMKAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: !Sub alias/${KeyAliasSuffix}
TargetKeyId: !Ref RegionKinesisCMK
```
------