# Getting started with example applications and patterns
The following resources can be used to quickly create and deploy serverless apps that implement some common Lambda uses cases. For each of the example apps, we provide instructions to either create and configure resources manually using the AWS Management Console, or to use the AWS Serverless Application Model to deploy the resources using IaC. Follow the console intructions to learn more about configuring the individual AWS resources for each app, or use to AWS SAM to quickly deploy resources as you would in a production environment.
## File Processing
+ **[PDF Encryption Application](file-processing-app.md)**: Create a serverless application that encrypts PDF files when they are uploaded to an Amazon Simple Storage Service bucket and saves them to another bucket, which is useful for securing sensitive documents upon upload.
+ **[Image Analysis Application](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-example-s3.html)**: Create a serverless application that extracts text from images using Amazon Rekognition, which is useful for document processing, content moderation, and automated image analysis.
## Database Integration
+ **[Queue-to-Database Application](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-lambda-tutorial.html)**: Create a serverless application that writes queue messages to an Amazon RDS database, which is useful for processing user registrations and handling order submissions.
+ **[Database Event Handler](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-example-ddb.html)**: Create a serverless application that responds to Amazon DynamoDB table changes, which is useful for audit logging, data replication, and automated workflows.
## Scheduled Tasks
+ **[Database Maintenance Application](scheduled-task-app.md)**: Create a serverless application that automatically deletes entries more than 12 months old from an Amazon DynamoDB table using a cron schedule, which is useful for automated database maintenance and data lifecycle management.
+ **[Create an EventBridge scheduled rule for Lambda functions](https://docs.aws.amazon.com/eventbridge/latest/userguide/run-lambda-schedule.html)**: Use scheduled expressions for rules in EventBridge to trigger a Lambda function on a timed schedule. This format uses cron syntax and can be set with a one-minute granularity.
## Additional resources
Use the following resources to further explore Lambda and serverless application development:
+ **[Serverless Land](https://serverlessland.com/)**: a library of ready-to-use patterns for building serverless apps. It helps developers create applications faster using AWS services like Lambda, API Gateway, and EventBridge. The site offers pre-built solutions and best practices, making it easier to develop serverless systems.
+ **[Lambda sample applications](https://docs.aws.amazon.com/lambda/latest/dg/lambda-samples.html)**: Applications that are available in the GitHub repository for this guide. These samples demonstrate the use of various languages and AWS services. Each sample application includes scripts for easy deployment and cleanup and supporting resources.
+ **[Code examples for Lambda using AWS SDKs](https://docs.aws.amazon.com/lambda/latest/dg/service_code_examples.html)**: Examples that show you how to use Lambda with AWS software development kits (SDKs). These examples include basics, actions, scenarios, and AWS community contributions. Examples cover essential operations, individual service functions, and specific tasks using multiple functions or AWS services.
# Create a serverless file-processing app
One of the most common use cases for Lambda is to perform file processing tasks. For example, you might use a Lambda function to automatically create PDF files from HTML files or images, or to create thumbnails when a user uploads an image.
In this example, you create an app which automatically encrypts PDF files when they are uploaded to an Amazon Simple Storage Service (Amazon S3) bucket. To implement this app, you create the following resources:
+ An S3 bucket for users to upload PDF files to
+ A Lambda function in Python which reads the uploaded file and creates an encrypted, password-protected version of it
+ A second S3 bucket for Lambda to save the encrypted file in
You also create an AWS Identity and Access Management (IAM) policy to give your Lambda function permission to perform read and write operations on your S3 buckets.
![\[\]](http://docs.aws.amazon.com/lambda/latest/dg/images/ExampleApps/file_process_resources.png)
**Tip**
If you’re brand new to Lambda, we recommend that you start with the tutorial [Create your first Lambda function](getting-started.md) before creating this example app.
You can deploy your app manually by creating and configuring resources with the AWS Management Console or the AWS Command Line Interface (AWS CLI). You can also deploy the app by using the AWS Serverless Application Model (AWS SAM). AWS SAM is an infrastructure as code (IaC) tool. With IaC, you don’t create resources manually, but define them in code and then deploy them automatically.
If you want to learn more about using Lambda with IaC before deploying this example app, see [Using Lambda with infrastructure as code (IaC)](foundation-iac.md).
## Create the Lambda function source code files
Create the following files in your project directory:
+ `lambda_function.py` - the Python function code for the Lambda function that performs the file encryption
+ `requirements.txt` - a manifest file defining the dependencies that your Python function code requires
Expand the following sections to view the code and to learn more about the role of each file. To create the files on your local machine, either copy and paste the code below, or download the files from the [aws-lambda-developer-guide GitHub repo](https://github.com/awsdocs/aws-lambda-developer-guide/tree/main/sample-apps/file-processing-python).
### Python function code
Copy and paste the following code into a file named `lambda_function.py`.
```
from pypdf import PdfReader, PdfWriter
import uuid
import os
from urllib.parse import unquote_plus
import boto3
# Create the S3 client to download and upload objects from S3
s3_client = boto3.client('s3')
def lambda_handler(event, context):
# Iterate over the S3 event object and get the key for all uploaded files
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = unquote_plus(record['s3']['object']['key']) # Decode the S3 object key to remove any URL-encoded characters
download_path = f'/tmp/{uuid.uuid4()}.pdf' # Create a path in the Lambda tmp directory to save the file to
upload_path = f'/tmp/converted-{uuid.uuid4()}.pdf' # Create another path to save the encrypted file to
# If the file is a PDF, encrypt it and upload it to the destination S3 bucket
if key.lower().endswith('.pdf'):
s3_client.download_file(bucket, key, download_path)
encrypt_pdf(download_path, upload_path)
encrypted_key = add_encrypted_suffix(key)
s3_client.upload_file(upload_path, f'{bucket}-encrypted', encrypted_key)
# Define the function to encrypt the PDF file with a password
def encrypt_pdf(file_path, encrypted_file_path):
reader = PdfReader(file_path)
writer = PdfWriter()
for page in reader.pages:
writer.add_page(page)
# Add a password to the new PDF
writer.encrypt("my-secret-password")
# Save the new PDF to a file
with open(encrypted_file_path, "wb") as file:
writer.write(file)
# Define a function to add a suffix to the original filename after encryption
def add_encrypted_suffix(original_key):
filename, extension = original_key.rsplit('.', 1)
return f'{filename}_encrypted.{extension}'
```
**Note**
In this example code, a password for the encrypted file (`my-secret-password`) is hardcoded into the function code. In a production application, don't include sensitive information like passwords in your function code. Instead, [create an AWS Secrets Manager secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) and then [use the AWS Parameters and Secrets Lambda extension](with-secrets-manager.md) to retrieve your credentials in your Lambda function.
The python function code contains three functions - the [handler function](python-handler.md) that Lambda runs when your function is invoked, and two separate function named `add_encrypted_suffix` and `encrypt_pdf` that the handler calls to perform the PDF encryption.
When your function is invoked by Amazon S3, Lambda passes a JSON formatted *event* argument to the function that contains details about the event that caused the invocation. In this case, the information includes name of the S3 bucket and the object keys for the uploaded files. To learn more about the format of event object for Amazon S3, see [Process Amazon S3 event notifications with Lambda](with-s3.md).
Your function then uses the AWS SDK for Python (Boto3) to download the PDF files specified in the event object to its local temporary storage directory, before encrypting them using the [https://pypi.org/project/pypdf/](https://pypi.org/project/pypdf/) library.
Finally, the function uses the Boto3 SDK to store the encrypted file in your S3 destination bucket.
### `requirements.txt` manifest file
Copy and paste the following code into a file named `requirements.txt`.
```
boto3
pypdf
```
For this example, your function code has only two dependencies that aren't part of the standard Python library - the SDK for Python (Boto3) and the `pypdf` package the function uses to perform the PDF encryption.
**Note**
A version of the SDK for Python (Boto3) is included as part of the Lambda runtime, so your code would run without adding Boto3 to your function's deployment package. However, to maintain full control of your function's dependencies and avoid possible issues with version misalignment, best practice for Python is to include all function dependencies in your function's deployment package. See [Runtime dependencies in Python](python-package.md#python-package-dependencies) to learn more.
## Deploy the app
You can create and deploy the resources for this example app either manually or by using AWS SAM. In a production environment, we recommend that you use an IaC tool like AWS SAM to quickly and repeatably deploy whole serverless applications without using manual processes.
### Deploy the resources manually
To deploy your app manually:
+ Create source and destination Amazon S3 buckets
+ Create a Lambda function that encrypts a PDF file and saves the encrypted version to an S3 bucket
+ Configure a Lambda trigger that invokes your function when objects are uploaded to your source bucket
Before you begin, make sure that [Python](https://www.python.org/downloads/) is installed on your build machine.
#### Create two S3 buckets
First create two S3 buckets. The first bucket is the source bucket you will upload your PDF files to. The second bucket is used by Lambda to save the encrypted file when you invoke your function.
------
#### [ Console ]
**To create the S3 buckets (console)**
1. Open the [General purpose buckets](https://console.aws.amazon.com/s3/buckets) page of the Amazon S3 console.
1. Select the AWS Region closest to your geographical location. You can change your region using the drop-down list at the top of the screen.
![\[\]](http://docs.aws.amazon.com/lambda/latest/dg/images/console_region_select.png)
1. Choose **Create bucket**.
1. Under **General configuration**, do the following:
1. For **Bucket type**, ensure **General purpose** is selected.
1. For **Bucket name**, enter a globally unique name that meets the Amazon S3 [bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html). Bucket names can contain only lower case letters, numbers, dots (.), and hyphens (-).
1. Leave all other options set to their default values and choose **Create bucket**.
1. Repeat steps 1 to 4 to create your destination bucket. For **Bucket name**, enter `amzn-s3-demo-bucket-encrypted`, where `amzn-s3-demo-bucket` is the name of the source bucket you just created.
------
#### [ AWS CLI ]
Before you begin, make sure that the [AWS CLI is installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your build machine.
**To create the Amazon S3 buckets (AWS CLI)**
1. Run the following CLI command to create your source bucket. The name you choose for your bucket must be globally unique and follow the Amazon S3 [bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html). Names can only contain lower case letters, numbers, dots (.), and hyphens (-). For `region` and `LocationConstraint`, choose the [AWS Region](https://docs.aws.amazon.com/general/latest/gr/lambda-service.html) closest to your geographical location.
```
aws s3api create-bucket --bucket amzn-s3-demo-bucket --region us-east-2 \
--create-bucket-configuration LocationConstraint=us-east-2
```
Later in the tutorial, you must create your Lambda function in the same AWS Region as your source bucket, so make a note of the region you chose.
1. Run the following command to create your destination bucket. For the bucket name, you must use `amzn-s3-demo-bucket-encrypted`, where `amzn-s3-demo-bucket` is the name of the source bucket you created in step 1. For `region` and `LocationConstraint`, choose the same AWS Region you used to create your source bucket.
```
aws s3api create-bucket --bucket amzn-s3-demo-bucket-encrypted --region us-east-2 \
--create-bucket-configuration LocationConstraint=us-east-2
```
------
#### Create an execution role
An execution role is an IAM role that grants a Lambda function permission to access AWS services and resources. To give your function read and write access to Amazon S3, you attach the [AWS managed policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) `AmazonS3FullAccess`.
------
#### [ Console ]
**To create an execution role and attach the `AmazonS3FullAccess` managed policy (console)**
1. Open the [Roles](https://console.aws.amazon.com/iam/home/roles) page in the IAM console.
1. Choose **Create role**.
1. For **Trusted entity type**, select **AWS service**, and for **Use case**, select **Lambda**.
1. Choose **Next**.
1. Add the `AmazonS3FullAccess` managed policy by doing the following:
1. In **Permissions policies**, enter **AmazonS3FullAccess** into the search bar.
1. Select the checkbox next to the policy.
1. Choose **Next**.
1. In **Role details**, for **Role name** enter **LambdaS3Role**.
1. Choose **Create Role**.
------
#### [ AWS CLI ]
**To create an execution role and attach the `AmazonS3FullAccess` managed policy (AWS CLI)**
1. Save the following JSON in a file named `trust-policy.json`. This trust policy allows Lambda to use the role’s permissions by giving the service principal `lambda.amazonaws.com` permission to call the AWS Security Token Service (AWS STS) `AssumeRole` action.
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
1. From the directory you saved the JSON trust policy document in, run the following CLI command to create the execution role.
```
aws iam create-role --role-name LambdaS3Role --assume-role-policy-document file://trust-policy.json
```
1. To attach the `AmazonS3FullAccess` managed policy, run the following CLI command.
```
aws iam attach-role-policy --role-name LambdaS3Role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
```
------
#### Create the function deployment package
To create your function, you create a *deployment package* containing your function code and its dependencies. For this application, your function code uses a separate library for the PDF encryption.
**To create the deployment package**
1. Navigate to the project directory containing the `lambda_function.py` and `requirements.txt` files you created or downloaded from GitHub earlier and create a new directory named `package`.
1. Install the dependencies specified in the `requirements.txt` file in your `package` directory by running the following command.
```
pip install -r requirements.txt --target ./package/
```
1. Create a .zip file containing your application code and its dependencies. In Linux or MacOS, run the following commands from your command line interface.
```
cd package
zip -r ../lambda_function.zip .
cd ..
zip lambda_function.zip lambda_function.py
```
In Windows, use your preferred zip tool to create the `lambda_function.zip` file. Make sure that your `lambda_function.py` file and the folders containing your dependencies are all at the root of the .zip file.
You can also create your deployment package using a Python virtual environment. See [Working with .zip file archives for Python Lambda functions](python-package.md)
#### Create the Lambda function
You now use the deployment package you created in the previous step to deploy your Lambda function.
------
#### [ Console ]
**To create the function (console)**
To create your Lambda function using the console, you first create a basic function containing some ‘Hello world’ code. You then replace this code with your own function code by uploading the.zip file you created in the previous step.
To ensure that your function doesn't time out when encrypting large PDF files, you configure the function's memory and timeout settings. You also set the function's log format to JSON. Configuring JSON formatted logs is necessary when using the provided test script so it can read the function's invocation status from CloudWatch Logs to confirm successful invocation.
1. Open the [Functions page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console.
1. Make sure you're working in the same AWS Region you created your S3 bucket in. You can change your region using the drop-down list at the top of the screen.
![\[\]](http://docs.aws.amazon.com/lambda/latest/dg/images/console_region_select.png)
1. Choose **Create function**.
1. Choose **Author from scratch**.
1. Under **Basic information**, do the following:
1. For **Function name**, enter `EncryptPDF`.
1. For **Runtime** choose **Python 3.12**.
1. For **Architecture**, choose **x86\$164**.
1. Attach the execution role you created in the previous step by doing the following:
1. Expand the **Change default execution role** section.
1. Select **Use an existing role**.
1. Under **Existing role**, select your role (`LambdaS3Role`).
1. Choose **Create function**.
**To upload the function code (console)**
1. In the **Code source** pane, choose **Upload from**.
1. Choose **.zip file**.
1. Choose **Upload**.
1. In the file selector, select your .zip file and choose **Open**.
1. Choose **Save**.
**To configure the function memory and timeout (console)**
1. Select the **Configuration** tab for your function.
1. In the **General configuration** pane, choose **Edit**.
1. Set **Memory** to 256 MB and **Timeout** to 15 seconds.
1. Choose **Save**.
**To configure the log format (console)**
1. Select the **Configuration** tab for your function.
1. Select **Monitoring and operations tools**.
1. In the **Logging configuration** pane, choose **Edit**.
1. For **Logging configuration**, select **JSON**.
1. Choose **Save**.
------
#### [ AWS CLI ]
**To create the function (AWS CLI)**
+ Run the following command from the directory containing your `lambda_function.zip` file.For the `region` parameter, replace `us-east-2` with the region you created your S3 buckets in.
```
aws lambda create-function --function-name EncryptPDF \
--zip-file fileb://lambda_function.zip --handler lambda_function.lambda_handler \
--runtime python3.12 --timeout 15 --memory-size 256 \
--role arn:aws:iam::123456789012:role/LambdaS3Role --region us-east-2 \
--logging-config LogFormat=JSON
```
------
#### Configure an Amazon S3 trigger to invoke the function
For your Lambda function to run when you upload a file to your source bucket, you need to configure a trigger for your function. You can configure the Amazon S3 trigger using either the console or the AWS CLI.
**Important**
This procedure configures the S3 bucket to invoke your function every time that an object is created in the bucket. Be sure to configure this only on the source bucket. If your Lambda function creates objects in the same bucket that invokes it, your function can be [invoked continuously in a loop](https://serverlessland.com/content/service/lambda/guides/aws-lambda-operator-guide/recursive-runaway). This can result in un expected charges being billed to your AWS account.
------
#### [ Console ]
**To configure the Amazon S3 trigger (console)**
1. Open the [Functions page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console and choose your function (`EncryptPDF`).
1. Choose **Add trigger**.
1. Select **S3**.
1. Under **Bucket**, select your source bucket.
1. Under **Event types**, select **All object create events**.
1. Under **Recursive invocation**, select the check box to acknowledge that using the same S3 bucket for input and output is not recommended. You can learn more about recursive invocation patterns in Lambda by reading [Recursive patterns that cause run-away Lambda functions](https://serverlessland.com/content/service/lambda/guides/aws-lambda-operator-guide/recursive-runaway) in Serverless Land.
1. Choose **Add**.
When you create a trigger using the Lambda console, Lambda automatically creates a [resource based policy](https://docs.aws.amazon.com/lambda/latest/dg/access-control-resource-based.html) to give the service you select permission to invoke your function.
------
#### [ AWS CLI ]
**To configure the Amazon S3 trigger (AWS CLI)**
1. Add a [resource based policy](https://docs.aws.amazon.com/lambda/latest/dg/access-control-resource-based.html) to your function that allows your Amazon S3 source bucket to invoke your function when you add a file. A resource-based policy statement gives other AWS services permission to invoke your function. To give Amazon S3 permission to invoke your function, run the following CLI command. Be sure to replace the `source-account` parameter with your own AWS account ID and to use your own source bucket name.
```
aws lambda add-permission --function-name EncryptPDF \
--principal s3.amazonaws.com --statement-id s3invoke --action "lambda:InvokeFunction" \
--source-arn arn:aws:s3:::amzn-s3-demo-bucket \
--source-account 123456789012
```
The policy you define with this command allows Amazon S3 to invoke your function only when an action takes place on your source bucket.
**Note**
Although S3 bucket names are globally unique, when using resource-based policies it is best practice to specify that the bucket must belong to your account. This is because if you delete a bucket, it is possible for another AWS account to create a bucket with the same Amazon Resource Name (ARN).
1. Save the following JSON in a file named `notification.json`. When applied to your source bucket, this JSON configures the bucket to send a notification to your Lambda function every time a new object is added. Replace the AWS account number and AWS Region in the Lambda function ARN with your own account number and region.
```
{
"LambdaFunctionConfigurations": [
{
"Id": "EncryptPDFEventConfiguration",
"LambdaFunctionArn": "arn:aws:lambda:us-east-2:123456789012:function:EncryptPDF",
"Events": [ "s3:ObjectCreated:Put" ]
}
]
}
```
1. Run the following CLI command to apply the notification settings in the JSON file you created to your source bucket. Replace `amzn-s3-demo-bucket` with the name of your own source bucket.
```
aws s3api put-bucket-notification-configuration --bucket amzn-s3-demo-bucket \
--notification-configuration file://notification.json
```
To learn more about the `put-bucket-notification-configuration` command and the `notification-configuration` option, see [put-bucket-notification-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-bucket-notification-configuration.html) in the *AWS CLI Command Reference*.
------
### Deploy the resources using AWS SAM
Before you begin, make sure that [Docker](https://docs.docker.com/get-docker/) and [the latest version of the AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) are installed on your build machine.
1. In your project directory, copy and paste the following code into a file named `template.yaml`. Replace the placeholder bucket names:
+ For the source bucket, replace `amzn-s3-demo-bucket` with any name that complies with the [S3 bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html).
+ For the destination bucket, replace `amzn-s3-demo-bucket-encrypted` with `-encrypted`, where `` is the name you chose for your source bucket.
```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
EncryptPDFFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: EncryptPDF
Architectures: [x86_64]
CodeUri: ./
Handler: lambda_function.lambda_handler
Runtime: python3.12
Timeout: 15
MemorySize: 256
LoggingConfig:
LogFormat: JSON
Policies:
- AmazonS3FullAccess
Events:
S3Event:
Type: S3
Properties:
Bucket: !Ref PDFSourceBucket
Events: s3:ObjectCreated:*
PDFSourceBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: amzn-s3-demo-bucket
EncryptedPDFBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: amzn-s3-demo-bucket-encrypted
```
The AWS SAM template defines the resources you create for your app. In this example, the template defines a Lambda function using the `AWS::Serverless::Function` type and two S3 buckets using the `AWS::S3::Bucket` type. The bucket names specified in the template are placeholders. Before you deploy the app using AWS SAM, you need to edit the template to rename the buckets with globally unique names that meet the [S3 bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html). This step is explained further in [Deploy the resources using AWS SAM](#file-processing-app-deploy-sam).
The definition of the Lambda function resource configures a trigger for the function using the `S3Event` event property. This trigger causes your function to be invoked whenever an object is created in your source bucket.
The function definition also specifies an AWS Identity and Access Management (IAM) policy to be attached to the function's [execution role](lambda-intro-execution-role.md). The [AWS managed policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) `AmazonS3FullAccess` gives your function the permissions it needs to read and write objects to Amazon S3.
1. Run the following command from the directory in which you saved your `template.yaml`, `lambda_function.py`, and `requirements.txt`files.
```
sam build --use-container
```
This command gathers the build artifacts for your application and places them in the proper format and location to deploy them. Specifying the `--use-container` option builds your function inside a Lambda-like Docker container. We use it here so you don't need to have Python 3.12 installed on your local machine for the build to work.
During the build process, AWS SAM looks for the Lambda function code in the location you specified with the `CodeUri` property in the template. In this case, we specified the current directory as the location (`./`).
If a `requirements.txt` file is present, AWS SAM uses it to gather the specified dependencies. By default, AWS SAM creates a .zip deployment package with your function code and dependencies. You can also choose to deploy your function as a container image using the [PackageType](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-packagetype) property.
1. To deploy your application and create the Lambda and Amazon S3 resources specified in your AWS SAM template, run the following command.
```
sam deploy --guided
```
Using the `--guided` flag means that AWS SAM will show you prompts to guide you through the deployment process. For this deployment, accept the default options by pressing Enter.
During the deployment process, AWS SAM creates the following resources in your AWS account:
+ An CloudFormation [stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-concepts.html#cfn-concepts-stacks) named `sam-app`
+ A Lambda function with the name `EncryptPDF`
+ Two S3 buckets with the names you chose when you edited the `template.yaml` AWS SAM template file
+ An IAM execution role for your function with the name format `sam-app-EncryptPDFFunctionRole-2qGaapHFWOQ8`
When AWS SAM finishes creating your resources, you should see the following message:
```
Successfully created/updated stack - sam-app in us-east-2
```
## Test the app
To test your app, upload a PDF file to your source bucket, and confirm that Lambda creates an encrypted version of the file in your destination bucket. In this example, you can either test this manually using the console or the AWS CLI, or by using the provided test script.
For production applications, you can use traditional test methods and techniques, such as unit testing, to confirm the correct functioning of your Lambda function code. Best practice is also to conduct tests like those in the provided test script which perform integration testing with real, cloud-based resources. Integration testing in the cloud confirms that your infrastructure has been correctly deployed and that events flow between different services as expected. To learn more, see [How to test serverless functions and applications](testing-guide.md).
### Testing the app manually
You can test your function manually by adding a PDF file to your Amazon S3 source bucket. When you add your file to the source bucket, your Lambda function should be automatically invoked and should store an encrypted version of the file in your target bucket.
------
#### [ Console ]
**To test your app by uploading a file (console)**
1. To upload a PDF file to your S3 bucket, do the following:
1. Open the [Buckets](https://console.aws.amazon.com/s3/buckets) page of the Amazon S3 console and choose your source bucket.
1. Choose **Upload**.
1. Choose **Add files** and use the file selector to choose the PDF file you want to upload.
1. Choose **Open**, then choose **Upload**.
1. Verify that Lambda has saved an encrypted version of your PDF file in your target bucket by doing the following:
1. Navigate back to the [Buckets](https://console.aws.amazon.com/s3/buckets) page of the Amazon S3 console and choose your destination bucket.
1. In the **Objects** pane, you should now see a file with name format `filename_encrypted.pdf` (where `filename.pdf` was the name of the file you uploaded to your source bucket). To download your encrypted PDF, select the file, then choose **Download**.
1. Confirm that you can open the downloaded file with the password your Lambda function protected it with (`my-secret-password`).
------
#### [ AWS CLI ]
**To test your app by uploading a file (AWS CLI)**
1. From the directory containing the PDF file you want to upload, run the following CLI command. Replace the `--bucket` parameter with the name of your source bucket. For the `--key` and `--body` parameters, use the filename of your test file.
```
aws s3api put-object --bucket amzn-s3-demo-bucket --key test.pdf --body ./test.pdf
```
1. Verify that your function has created an encrypted version of your file and saved it to your target S3 bucket. Run the following CLI command, replacing `amzn-s3-demo-bucket-encrypted` with the name of your own destination bucket.
```
aws s3api list-objects-v2 --bucket amzn-s3-demo-bucket-encrypted
```
If your function runs successfully, you’ll see output similar to the following. Your target bucket should contain a file with the name format `_encrypted.pdf`, where `` is the name of the file you uploaded.
```
{
"Contents": [
{
"Key": "test_encrypted.pdf",
"LastModified": "2023-06-07T00:15:50+00:00",
"ETag": "\"7781a43e765a8301713f533d70968a1e\"",
"Size": 2763,
"StorageClass": "STANDARD"
}
]
}
```
1. To download the file that Lambda saved in your destination bucket, run the following CLI command. Replace the `--bucket` parameter with the name of your destination bucket. For the `--key` parameter, use the filename `_encrypted.pdf`, where `` is the name of the the test file you uploaded.
```
aws s3api get-object --bucket amzn-s3-demo-bucket-encrypted --key test_encrypted.pdf my_encrypted_file.pdf
```
This command downloads the file to your current directory and saves it as `my_encrypted_file.pdf`.
1. Confirm the you can open the downloaded file with the password your Lambda function protected it with (`my-secret-password`).
------
### Testing the app with the automated script
Create the following files in your project directory:
+ `test_pdf_encrypt.py` - a test script you can use to automatically test your application
+ `pytest.ini` - a configuration file for the the test script
Expand the following sections to view the code and to learn more about the role of each file.
#### Automated test script
Copy and paste the following code into a file named `test_pdf_encrypt.py`. Be sure to replace the placeholder bucket names:
+ In the `test_source_bucket_available` function, replace `amzn-s3-demo-bucket` with the name of your source bucket.
+ In the `test_encrypted_file_in_bucket` function, replace `amzn-s3-demo-bucket-encrypted` with `source-bucket-encrypted`, where `source-bucket>` is the name of your source bucket.
+ In the `cleanup` function, replace `amzn-s3-demo-bucket` with the name of your source bucket, and replace `amzn-s3-demo-bucket-encrypted` with the name of your destination bucket.
```
import boto3
import json
import pytest
import time
import os
@pytest.fixture
def lambda_client():
return boto3.client('lambda')
@pytest.fixture
def s3_client():
return boto3.client('s3')
@pytest.fixture
def logs_client():
return boto3.client('logs')
@pytest.fixture(scope='session')
def cleanup():
# Create a new S3 client for cleanup
s3_client = boto3.client('s3')
yield
# Cleanup code will be executed after all tests have finished
# Delete test.pdf from the source bucket
source_bucket = 'amzn-s3-demo-bucket'
source_file_key = 'test.pdf'
s3_client.delete_object(Bucket=source_bucket, Key=source_file_key)
print(f"\nDeleted {source_file_key} from {source_bucket}")
# Delete test_encrypted.pdf from the destination bucket
destination_bucket = 'amzn-s3-demo-bucket-encrypted'
destination_file_key = 'test_encrypted.pdf'
s3_client.delete_object(Bucket=destination_bucket, Key=destination_file_key)
print(f"Deleted {destination_file_key} from {destination_bucket}")
@pytest.mark.order(1)
def test_source_bucket_available(s3_client):
s3_bucket_name = 'amzn-s3-demo-bucket'
file_name = 'test.pdf'
file_path = os.path.join(os.path.dirname(__file__), file_name)
file_uploaded = False
try:
s3_client.upload_file(file_path, s3_bucket_name, file_name)
file_uploaded = True
except:
print("Error: couldn't upload file")
assert file_uploaded, "Could not upload file to S3 bucket"
@pytest.mark.order(2)
def test_lambda_invoked(logs_client):
# Wait for a few seconds to make sure the logs are available
time.sleep(5)
# Get the latest log stream for the specified log group
log_streams = logs_client.describe_log_streams(
logGroupName='/aws/lambda/EncryptPDF',
orderBy='LastEventTime',
descending=True,
limit=1
)
latest_log_stream_name = log_streams['logStreams'][0]['logStreamName']
# Retrieve the log events from the latest log stream
log_events = logs_client.get_log_events(
logGroupName='/aws/lambda/EncryptPDF',
logStreamName=latest_log_stream_name
)
success_found = False
for event in log_events['events']:
message = json.loads(event['message'])
status = message.get('record', {}).get('status')
if status == 'success':
success_found = True
break
assert success_found, "Lambda function execution did not report 'success' status in logs."
@pytest.mark.order(3)
def test_encrypted_file_in_bucket(s3_client):
# Specify the destination S3 bucket and the expected converted file key
destination_bucket = 'amzn-s3-demo-bucket-encrypted'
converted_file_key = 'test_encrypted.pdf'
try:
# Attempt to retrieve the metadata of the converted file from the destination S3 bucket
s3_client.head_object(Bucket=destination_bucket, Key=converted_file_key)
except s3_client.exceptions.ClientError as e:
# If the file is not found, the test will fail
pytest.fail(f"Converted file '{converted_file_key}' not found in the destination bucket: {str(e)}")
def test_cleanup(cleanup):
# This test uses the cleanup fixture and will be executed last
pass
```
The automated test script executes three test functions to confirm correct operation of your app:
+ The test `test_source_bucket_available` confirms that your source bucket has been successfully created by uploading a test PDF file to the bucket.
+ The test `test_lambda_invoked` interrogates the latest CloudWatch Logs log stream for your function to confirm that when you uploaded the test file, your Lambda function ran and reported success.
+ The test `test_encrypted_file_in_bucket` confirms that your destination bucket contains the encrypted `test_encrypted.pdf` file.
After all these tests have run, the script runs an additional cleanup step to delete the `test.pdf` and `test_encrypted.pdf` files from both your source and destination buckets.
As with the AWS SAM template, the bucket names specified in this file are placeholders. Before running the test, you need to edit this file with your app's real bucket names. This step is explained further in [Testing the app with the automated script](#file-processing-app-test-auto)
#### Test script configuration file
Copy and paste the following code into a file named `pytest.ini`.
```
[pytest]
markers =
order: specify test execution order
```
This is needed to specify the order in which the tests in the `test_pdf_encrypt.py` script run.
To run the tests do the following:
1. Ensure that the `pytest` module is installed in your local environment. You can install `pytest` by running the following command:
```
pip install pytest
```
1. Save a PDF file named `test.pdf` in the directory containing the `test_pdf_encrypt.py` and `pytest.ini` files.
1. Open a terminal or shell program and run the following command from the directory containing the test files.
```
pytest -s -v
```
When the test completes, you should see output like the following:
```
============================================================== test session starts =========================================================
platform linux -- Python 3.12.2, pytest-7.2.2, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/pdf_encrypt_app/.hypothesis/examples')
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=
rootdir: /home/pdf_encrypt_app, configfile: pytest.ini
plugins: anyio-3.7.1, hypothesis-6.70.0, localserver-0.7.1, random-order-1.1.0
collected 4 items
test_pdf_encrypt.py::test_source_bucket_available PASSED
test_pdf_encrypt.py::test_lambda_invoked PASSED
test_pdf_encrypt.py::test_encrypted_file_in_bucket PASSED
test_pdf_encrypt.py::test_cleanup PASSED
Deleted test.pdf from amzn-s3-demo-bucket
Deleted test_encrypted.pdf from amzn-s3-demo-bucket-encrypted
=============================================================== 4 passed in 7.32s ==========================================================
```
## Next steps
Now you've created this example app, you can use the provided code as a basis to create other types of file-processing application. Modify the code in the `lambda_function.py` file to implement the file-processing logic for your use case.
Many typical file-processing use cases involve image processing. When using Python, the most popular image-processing libraries like [pillow](https://pypi.org/project/pillow/) typically contain C or C\$1\$1 components. In order to ensure that your function's deployment package is compatible with the Lambda execution environment, it's important to use the correct source distribution binary.
When deploying your resources with AWS SAM, you need to take some extra steps to include the right source distribution in your deployment package. Because AWS SAM won't install dependencies for a different platform than your build machine, specifying the correct source distribution (`.whl` file) in your `requirements.txt` file won't work if your build machine uses an operating system or architecture that's different from the Lambda execution environment. Instead, you should do one of the following:
+ Use the `--use-container` option when running `sam build`. When you specify this option, AWS SAM downloads a container base image that's compatible with the Lambda execution environment and builds your function's deployment package in a Docker container using that image. To learn more, see [Building a Lambda function inside of a provided container](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/using-sam-cli-build.html#using-sam-cli-build-options-container).
+ Build your function's .zip deployment package yourself using the correct source distribution binary and save the .zip file in the directory you specify as the `CodeUri` in the AWS SAM template. To learn more about building .zip deployment packages for Python using binary distributions, see [Creating a .zip deployment package with dependencies](python-package.md#python-package-create-dependencies) and [Creating .zip deployment packages with native libraries](python-package.md#python-package-native-libraries).
# Create an app to perform scheduled database maintenance
You can use AWS Lambda to replace scheduled processes such as automated system backups, file conversions, and maintenance tasks. In this example, you create a serverless application that performs regular scheduled maintenance on a DynamoDB table by deleting old entries. The app uses EventBridge Scheduler to invoke a Lambda function on a cron schedule. When invoked, the function queries the table for items older than one year, and deletes them. The function logs each deleted item in CloudWatch Logs.
To implement this example, first create a DynamoDB table and populate it with some test data for your function to query. Then, create a Python Lambda function with an EventBridge Scheduler trigger and an IAM execution role that gives the function permission to read, and delete, items from your table.
![\[\]](http://docs.aws.amazon.com/lambda/latest/dg/images/ExampleApps/cron_app.png)
**Tip**
If you’re new to Lambda, we recommend that you complete the tutorial [Create your first Lambda function](getting-started.md) before creating this example app.
You can deploy your app manually by creating and configuring resources with the AWS Management Console. You can also deploy the app by using the AWS Serverless Application Model (AWS SAM). AWS SAM is an infrastructure as code (IaC) tool. With IaC, you don’t create resources manually, but define them in code and then deploy them automatically.
If you want to learn more about using Lambda with IaC before deploying this example app, see [Using Lambda with infrastructure as code (IaC)](foundation-iac.md).
## Prerequisites
Before you can create the example app, make sure you have the required command line tools and programs installed.
+ **Python**
To populate the DynamoDB table you create to test your app, this example uses a Python script and a CSV file to write data into the table. Make sure you have Python version 3.8 or later installed on your machine.
+ **AWS SAM CLI**
If you want to create the DynamoDB table and deploy the example app using AWS SAM, you need to install the AWS SAM CLI. Follow the [installation instructions](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) in the *AWS SAM User Guide*.
+ **AWS CLI**
To use the provided Python script to populate your test table, you need to have installed and configured the AWS CLI. This is because the script uses the AWS SDK for Python (Boto3), which needs access to your AWS Identity and Access Management (IAM) credentials. You also need the AWS CLI installed to deploy resources using AWS SAM. Install the CLI by following the [installation instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*.
+ **Docker**
To deploy the app using AWS SAM, Docker must also be installed on your build machine. Follow the instructions in [Install Docker Engine](https://docs.docker.com/engine/install/) on the Docker documentation website.
## Downloading the example app files
To create the example database and the scheduled-maintenance app, you need to create the following files in your project directory:
**Example database files**
+ `template.yaml` - an AWS SAM template you can use to create the DynamoDB table
+ `sample_data.csv` - a CSV file containing sample data to load into your table
+ `load_sample_data.py` - a Python script that writes the data in the CSV file into the table
**Scheduled-maintenance app files**
+ `lambda_function.py` - the Python function code for the Lambda function that performs the database maintenance
+ `requirements.txt` - a manifest file defining the dependencies that your Python function code requires
+ `template.yaml` - an AWS SAM template you can use to deploy the app
**Test file**
+ `test_app.py` - a Python script that scans the table and confirms successful operation of your function by outputting all records older than one year
Expand the following sections to view the code and to learn more about the role of each file in creating and testing your app. To create the files on your local machine, copy and paste the code below.
### AWS SAM template (example DynamoDB table)
Copy and paste the following code into a file named `template.yaml`.
```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for DynamoDB Table with Order_number as Partition Key and Date as Sort Key
Resources:
MyDynamoDBTable:
Type: AWS::DynamoDB::Table
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
TableName: MyOrderTable
BillingMode: PAY_PER_REQUEST
AttributeDefinitions:
- AttributeName: Order_number
AttributeType: S
- AttributeName: Date
AttributeType: S
KeySchema:
- AttributeName: Order_number
KeyType: HASH
- AttributeName: Date
KeyType: RANGE
SSESpecification:
SSEEnabled: true
GlobalSecondaryIndexes:
- IndexName: Date-index
KeySchema:
- AttributeName: Date
KeyType: HASH
Projection:
ProjectionType: ALL
PointInTimeRecoverySpecification:
PointInTimeRecoveryEnabled: true
Outputs:
TableName:
Description: DynamoDB Table Name
Value: !Ref MyDynamoDBTable
TableArn:
Description: DynamoDB Table ARN
Value: !GetAtt MyDynamoDBTable.Arn
```
**Note**
AWS SAM templates use a standard naming convention of `template.yaml`. In this example, you have two template files - one to create the example database and another to create the app itself. Save them in separate sub-directories in your project folder.
This AWS SAM template defines the DynamoDB table resource you create to test your app. The table uses a primary key of `Order_number` with a sort key of `Date`. In order for your Lambda function to find items directly by date, we also define a [Global Secondary Index](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html) named `Date-index`.
To learn more about creating and configuring a DynamoDB table using the `AWS::DynamoDB::Table` resource, see [AWS::DynamoDB::Table](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html) in the *AWS CloudFormation User Guide*.
### Sample database data file
Copy and paste the following code into a file named `sample_data.csv`.
```
Date,Order_number,CustomerName,ProductID,Quantity,TotalAmount
2023-09-01,ORD001,Alejandro Rosalez,PROD123,2,199.98
2023-09-01,ORD002,Akua Mansa,PROD456,1,49.99
2023-09-02,ORD003,Ana Carolina Silva,PROD789,3,149.97
2023-09-03,ORD004,Arnav Desai,PROD123,1,99.99
2023-10-01,ORD005,Carlos Salazar,PROD456,2,99.98
2023-10-02,ORD006,Diego Ramirez,PROD789,1,49.99
2023-10-03,ORD007,Efua Owusu,PROD123,4,399.96
2023-10-04,ORD008,John Stiles,PROD456,2,99.98
2023-10-05,ORD009,Jorge Souza,PROD789,3,149.97
2023-10-06,ORD010,Kwaku Mensah,PROD123,1,99.99
2023-11-01,ORD011,Li Juan,PROD456,5,249.95
2023-11-02,ORD012,Marcia Oliveria,PROD789,2,99.98
2023-11-03,ORD013,Maria Garcia,PROD123,3,299.97
2023-11-04,ORD014,Martha Rivera,PROD456,1,49.99
2023-11-05,ORD015,Mary Major,PROD789,4,199.96
2023-12-01,ORD016,Mateo Jackson,PROD123,2,199.99
2023-12-02,ORD017,Nikki Wolf,PROD456,3,149.97
2023-12-03,ORD018,Pat Candella,PROD789,1,49.99
2023-12-04,ORD019,Paulo Santos,PROD123,5,499.95
2023-12-05,ORD020,Richard Roe,PROD456,2,99.98
2024-01-01,ORD021,Saanvi Sarkar,PROD789,3,149.97
2024-01-02,ORD022,Shirley Rodriguez,PROD123,1,99.99
2024-01-03,ORD023,Sofia Martinez,PROD456,4,199.96
2024-01-04,ORD024,Terry Whitlock,PROD789,2,99.98
2024-01-05,ORD025,Wang Xiulan,PROD123,3,299.97
```
This file contains some example test data to populate your DynamoDB table with in a standard comma-separated values (CSV) format.
### Python script to load sample data
Copy and paste the following code into a file named `load_sample_data.py`.
```
import boto3
import csv
from decimal import Decimal
# Initialize the DynamoDB client
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('MyOrderTable')
print("DDB client initialized.")
def load_data_from_csv(filename):
with open(filename, 'r') as file:
csv_reader = csv.DictReader(file)
for row in csv_reader:
item = {
'Order_number': row['Order_number'],
'Date': row['Date'],
'CustomerName': row['CustomerName'],
'ProductID': row['ProductID'],
'Quantity': int(row['Quantity']),
'TotalAmount': Decimal(str(row['TotalAmount']))
}
table.put_item(Item=item)
print(f"Added item: {item['Order_number']} - {item['Date']}")
if __name__ == "__main__":
load_data_from_csv('sample_data.csv')
print("Data loading completed.")
```
This Python script first uses the AWS SDK for Python (Boto3) to create a connection to your DynamoDB table. It then iterates over each row in the example-data CSV file, creates an item from that row, and writes the item to the DynamoDB table using the boto3 SDK.
### Python function code
Copy and paste the following code into a file named `lambda_function.py`.
```
import boto3
from datetime import datetime, timedelta
from boto3.dynamodb.conditions import Key, Attr
import logging
logger = logging.getLogger()
logger.setLevel("INFO")
def lambda_handler(event, context):
# Initialize the DynamoDB client
dynamodb = boto3.resource('dynamodb')
# Specify the table name
table_name = 'MyOrderTable'
table = dynamodb.Table(table_name)
# Get today's date
today = datetime.now()
# Calculate the date one year ago
one_year_ago = (today - timedelta(days=365)).strftime('%Y-%m-%d')
# Scan the table using a global secondary index
response = table.scan(
IndexName='Date-index',
FilterExpression='#date < :one_year_ago',
ExpressionAttributeNames={
'#date': 'Date'
},
ExpressionAttributeValues={
':one_year_ago': one_year_ago
}
)
# Delete old items
with table.batch_writer() as batch:
for item in response['Items']:
Order_number = item['Order_number']
batch.delete_item(
Key={
'Order_number': Order_number,
'Date': item['Date']
}
)
logger.info(f'deleted order number {Order_number}')
# Check if there are more items to scan
while 'LastEvaluatedKey' in response:
response = table.scan(
IndexName='DateIndex',
FilterExpression='#date < :one_year_ago',
ExpressionAttributeNames={
'#date': 'Date'
},
ExpressionAttributeValues={
':one_year_ago': one_year_ago
},
ExclusiveStartKey=response['LastEvaluatedKey']
)
# Delete old items
with table.batch_writer() as batch:
for item in response['Items']:
batch.delete_item(
Key={
'Order_number': item['Order_number'],
'Date': item['Date']
}
)
return {
'statusCode': 200,
'body': 'Cleanup completed successfully'
}
```
The Python function code contains the [handler function](python-handler.md) (`lambda_handler`) that Lambda runs when your function is invoked.
When the function is invoked by EventBridge Scheduler, it uses the AWS SDK for Python (Boto3) to create a connection to the DynamoDB table on which the scheduled maintenance task is to be performed. It then uses the Python `datetime` library to calculate the date one year ago, before scanning the table for items older than this and deleting them.
Note that responses from DynamoDB query and scan operations are limited to a maximum of 1 MB in size. If the response is larger than 1 MB, DynamoDB paginates the data and returns a `LastEvaluatedKey` element in the response. To ensure that our function processes all the records in the table, we check for the presence of this key and continue performing table scans from the last evaluated position until the whole table has been scanned.
### `requirements.txt` manifest file
Copy and paste the following code into a file named `requirements.txt`.
```
boto3
```
For this example, your function code has only one dependency that isn't part of the standard Python library - the SDK for Python (Boto3) that the function uses to scan and delete items from the DynamoDB table.
**Note**
A version of the SDK for Python (Boto3) is included as part of the Lambda runtime, so your code would run without adding Boto3 to your function's deployment package. However, to maintain full control of your function's dependencies and avoid possible issues with version misalignment, best practice for Python is to include all function dependencies in your function's deployment package. See [Runtime dependencies in Python](python-package.md#python-package-dependencies) to learn more.
### AWS SAM template (scheduled-maintenance app)
Copy and paste the following code into a file named `template.yaml`.
```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for Lambda function and EventBridge Scheduler rule
Resources:
MyLambdaFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: ScheduledDBMaintenance
CodeUri: ./
Handler: lambda_function.lambda_handler
Runtime: python3.11
Architectures:
- x86_64
Events:
ScheduleEvent:
Type: ScheduleV2
Properties:
ScheduleExpression: cron(0 3 1 * ? *)
Description: Run on the first day of every month at 03:00 AM
Policies:
- CloudWatchLogsFullAccess
- Statement:
- Effect: Allow
Action:
- dynamodb:Scan
- dynamodb:BatchWriteItem
Resource: !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/MyOrderTable'
LambdaLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub /aws/lambda/${MyLambdaFunction}
RetentionInDays: 30
Outputs:
LambdaFunctionName:
Description: Lambda Function Name
Value: !Ref MyLambdaFunction
LambdaFunctionArn:
Description: Lambda Function ARN
Value: !GetAtt MyLambdaFunction.Arn
```
**Note**
AWS SAM templates use a standard naming convention of `template.yaml`. In this example, you have two template files - one to create the example database and another to create the app itself. Save them in separate sub-directories in your project folder.
This AWS SAM template defines the resources for your app. We define the Lambda function using the `AWS::Serverless::Function` resource. The EventBridge Scheduler schedule and the trigger to invoke the Lambda function are created by using the `Events` property of this resource using a type of `ScheduleV2`. To learn more about defining EventBridge Scheduler schedules in AWS SAM templates, see [ScheduleV2](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-schedulev2.html) in the *AWS Serverless Application Model Developer Guide*.
In addition to the Lambda function and the EventBridge Scheduler schedule, we also define a CloudWatch log group for your function to send records of deleted items to.
### Test script
Copy and paste the following code into a file named `test_app.py`.
```
import boto3
from datetime import datetime, timedelta
import json
# Initialize the DynamoDB client
dynamodb = boto3.resource('dynamodb')
# Specify your table name
table_name = 'YourTableName'
table = dynamodb.Table(table_name)
# Get the current date
current_date = datetime.now()
# Calculate the date one year ago
one_year_ago = current_date - timedelta(days=365)
# Convert the date to string format (assuming the date in DynamoDB is stored as a string)
one_year_ago_str = one_year_ago.strftime('%Y-%m-%d')
# Scan the table
response = table.scan(
FilterExpression='#date < :one_year_ago',
ExpressionAttributeNames={
'#date': 'Date'
},
ExpressionAttributeValues={
':one_year_ago': one_year_ago_str
}
)
# Process the results
old_records = response['Items']
# Continue scanning if we have more items (pagination)
while 'LastEvaluatedKey' in response:
response = table.scan(
FilterExpression='#date < :one_year_ago',
ExpressionAttributeNames={
'#date': 'Date'
},
ExpressionAttributeValues={
':one_year_ago': one_year_ago_str
},
ExclusiveStartKey=response['LastEvaluatedKey']
)
old_records.extend(response['Items'])
for record in old_records:
print(json.dumps(record))
# The total number of old records should be zero.
print(f"Total number of old records: {len(old_records)}")
```
This test script uses the AWS SDK for Python (Boto3) to create a connection to your DynamoDB table and scan for items older than one year. To confirm if the Lambda function has run successfully, at the end of the test, the function prints the number of records older than one year still in the table. If the Lambda function was successful, the number of old records in the table should be zero.
## Creating and populating the example DynamoDB table
To test your scheduled-maintenance app, you first create a DynamoDB table and populate it with some sample data. You can create the table either manually using the AWS Management Console or by using AWS SAM. We recommend that you use AWS SAM to quickly create and configure the table using a few AWS CLI commands.
------
#### [ Console ]
**To create the DynamoDB table**
1. Open the [Tables](https://console.aws.amazon.com/dynamodbv2/home#tables) page of the DynamoDB console.
1. Choose **Create table**.
1. Create the table by doing the following:
1. Under **Table details**, for **Table name**, enter **MyOrderTable**.
1. For **Partition key**, enter **Order\$1number** and leave the type as **String**.
1. For **Sort key**, enter **Date** and leave the type as **String**.
1. Leave **Table settings** set to **Default settings** and choose **Create table**.
1. When your table has finished creating and its **Status** shows as **Active**, create a global secondary index (GSI) by doing the following. Your app will use this GSI to search for items directly by date to determine what to delete.
1. Choose **MyOrderTable** from the list of tables.
1. Choose the **Indexes** tab.
1. Under **Global secondary indexes**, choose **Create index**.
1. Under **Index details**, enter **Date** for the **Partition key** and leave the **Data type** set to **String**.
1. For **Index name**, enter **Date-index**.
1. Leave all other parameters set to their default values, scroll to the bottom of the page, and choose **Create index**.
------
#### [ AWS SAM ]
**To create the DynamoDB table**
1. Navigate to the folder you saved the `template.yaml` file for the DynamoDB table in. Note that this example uses two `template.yaml` files. Make sure they are saved in separate sub-folders and that you are in the correct folder containing the template to create your DynamoDB table.
1. Run the following command.
```
sam build
```
This command gathers the build artifacts for the resources you want to deploy and places them in the proper format and location to deploy them.
1. To create the DynamoDB resource specified in the `template.yaml` file, run the following command.
```
sam deploy --guided
```
Using the `--guided` flag means that AWS SAM will show you prompts to guide you through the deployment process. For this deployment, enter a `Stack name` of **cron-app-test-db**, and accept the defaults for all other options by using Enter.
When AWS SAM has finished creating the DynamoDB resource, you should see the following message.
```
Successfully created/updated stack - cron-app-test-db in us-west-2
```
1. You can additionally confirm that the DynamoDB table has been created by opening the [Tables](https://console.aws.amazon.com/dynamodbv2/home#tables) page of the DynamoDB console. You should see a table named `MyOrderTable`.
------
After you've created your table, you next add some sample data to test your app. The CSV file `sample_data.csv` you downloaded earlier contains a number of example entries comprised of order numbers, dates, and customer and order information. Use the provided python script `load_sample_data.py` to add this data to your table.
**To add the sample data to the table**
1. Navigate to the directory containing the `sample_data.csv` and `load_sample_data.py` files. If these files are in separate directories, move them so they're saved in the same location.
1. Create a Python virtual environment to run the script in by running the following command. We recommend that you use a virtual environment because in a following step you'll need to install the AWS SDK for Python (Boto3).
```
python -m venv venv
```
1. Activate the virtual environment by running the following command.
```
source venv/bin/activate
```
1. Install the SDK for Python (Boto3) in your virtual environment by running the following command. The script uses this library to connect to your DynamoDB table and add the items.
```
pip install boto3
```
1. Run the script to populate the table by running the following command.
```
python load_sample_data.py
```
If the script runs successfully, it should print each item to the console as it loads it and report `Data loading completed`.
1. Deactivate the virtual environment by running the following command.
```
deactivate
```
1. You can verify that the data has been loaded to your DynamoDB table by doing the following:
1. Open the [Explore items](https://console.aws.amazon.com/dynamodbv2/home#item-explorer) page of the DynamoDB console and select your table (`MyOrderTable`).
1. In the **Items returned** pane, you should see the 25 items from the CSV file that the script added to the table.
## Creating the scheduled-maintenance app
You can create and deploy the resources for this example app step by step using the AWS Management Console or by using AWS SAM. In a production environment, we recommend that you use an Infrustracture-as-Code (IaC) tool like AWS SAM to repeatably deploy serverless applications without using manual processes.
For this example, follow the console instructions to learn how to configure each AWS resource separately, or follow the AWS SAM instructions to quickly deploy the app using AWS CLI commands.
------
#### [ Console ]
**To create the function using the AWS Management Console**
First, create a function containing basic starter code. You then replace this code with your own function code by either copying and pasting the code directly in the Lambda code editor, or by uploading your code as a `.zip` package. For this task, we recommend copying and pasting the code.
1. Open the [Functions page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console.
1. Choose **Create function**.
1. Choose **Author from scratch**.
1. Under **Basic information**, do the following:
1. For **Function name**, enter `ScheduledDBMaintenance`.
1. For **Runtime** choose the latest Python version.
1. For **Architecture**, choose **x86\$164**.
1. Choose **Create function**.
1. After your function is created, you can configure your function with the provided function code.
1. In the **Code source** pane, replace the Hello world code that Lambda created with the Python function code from the `lambda_function.py` file that you saved earlier.
1. In the **DEPLOY** section, choose **Deploy** to update your function's code:
![\[\]](http://docs.aws.amazon.com/lambda/latest/dg/images/getting-started-tutorial/deploy-console.png)
**To configure the function memory and timeout (console)**
1. Select the **Configuration** tab for your function.
1. In the **General configuration** pane, choose **Edit**.
1. Set **Memory** to 256 MB and **Timeout** to 15 seconds. If you are processing a large table with many records, for example in the case of a production environment, you might consider setting **Timeout** to a larger number. This gives your function more time to scan, and clean the database.
1. Choose **Save**.
**To configure the log format (console)**
You can configure Lambda functions to output logs in either unstructured text or JSON format. We recommend that you use JSON format for logs to make it easier to search and filter log data. To learn more about Lambda log configuration options, see [Configuring advanced logging controls for Lambda functions](monitoring-logs.md#monitoring-cloudwatchlogs-advanced).
1. Select the **Configuration** tab for your function.
1. Select **Monitoring and operations tools**.
1. In the **Logging configuration** pane, choose **Edit**.
1. For **Logging configuration**, select **JSON**.
1. Choose **Save**.
**To set Up IAM permissions**
To give your function the permissions it needs to read and delete DynamoDB items, you need to add a policy to your function's [execution role](lambda-intro-execution-role.md) defining the necessary permissions.
1. Open the **Configuration** tab, then choose **Permissions** from the left navigation bar.
1. Choose the role name under **Execution role**.
1. In the IAM console, choose **Add permissions**, then **Create inline policy**.
1. Use the JSON editor and enter the following policy:
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:Scan",
"dynamodb:DeleteItem",
"dynamodb:BatchWriteItem"
],
"Resource": "arn:aws:dynamodb:*:*:table/MyOrderTable"
}
]
}
```
1. Name the policy **DynamoDBCleanupPolicy**, then create it.
**To set up EventBridge Scheduler as a trigger (console)**
1. Open the [EventBridge console](https://console.aws.amazon.com/events/home).
1. In the left navigation pane, choose **Schedulers** under the **Scheduler** section.
1. Choose **Create schedule**.
1. Configure the schedule by doing the following:
1. Under **Schedule name**, enter a name for your schedule (for example, **DynamoDBCleanupSchedule**).
1. Under **Schedule pattern**, choose **Recurring schedule**.
1. For **Schedule type** leave the default as **Cron-based schedule**, then enter the following schedule details:
+ **Minutes**: **0**
+ **Hours**: **3**
+ **Day of month**: **1**
+ **Month**: **\$1**
+ **Day of the week**: **?**
+ **Year**: **\$1**
When evaluated, this cron expression runs on the first day of every month at 03:00 AM.
1. For **Flexible time window**, select **Off**.
1. Choose **Next**.
1. Configure the trigger for your Lambda function by doing the following:
1. In the **Target detail** pane, leave **Target API** set to **Templated targets**, then select **AWS Lambda Invoke**.
1. Under **Invoke**, select your Lambda function (`ScheduledDBMaintenance`) from the dropdown list.
1. Leave the **Payload** empty and choose **Next**.
1. Scroll down to **Permissions** and select **Create a new role for this schedule**. When you create a new EventBridge Scheduler schedule using the console, EventBridge Scheduler creates a new policy with the required permissions the schedule needs to invoke your function. For more information about managing your schedule permissions, see [Cron-based schedules](https://docs.aws.amazon.com/scheduler/latest/UserGuide/schedule-types.html#cron-based). in the *EventBridge Scheduler User Guide*.
1. Choose **Next**.
1. Review your settings and choose **Create schedule** to complete creation of the schedule and Lambda trigger.
------
#### [ AWS SAM ]
**To deploy the app using AWS SAM**
1. Navigate to the folder you saved the `template.yaml` file for the app in. Note that this example uses two `template.yaml` files. Make sure they are saved in separate sub-folders and that you are in the correct folder containing the template to create the app.
1. Copy the `lambda_function.py` and `requirements.txt` files you downloaded earlier to the same folder. The code location specified in the AWS SAM template is `./`, meaning the current location. AWS SAM will search in this folder for the Lambda function code when you try to deploy the app.
1. Run the following command.
```
sam build --use-container
```
This command gathers the build artifacts for the resources you want to deploy and places them in the proper format and location to deploy them. Specifying the `--use-container` option builds your function inside a Lambda-like Docker container. We use it here so you don't need to have Python 3.12 installed on your local machine for the build to work.
1. To create the Lambda and EventBridge Scheduler resources specified in the `template.yaml` file, run the following command.
```
sam deploy --guided
```
Using the `--guided` flag means that AWS SAM will show you prompts to guide you through the deployment process. For this deployment, enter a `Stack name` of **cron-maintenance-app**, and accept the defaults for all other options by using Enter.
When AWS SAM has finished creating the Lambda and EventBridge Scheduler resources, you should see the following message.
```
Successfully created/updated stack - cron-maintenance-app in us-west-2
```
1. You can additionally confirm that the Lambda function has been created by opening the [Functions](https://console.aws.amazon.com/lambda/home#/functions) page of the Lambda console. You should see a function named `ScheduledDBMaintenance`.
------
## Testing the app
To test that your schedule correctly triggers your function, and that your function correctly cleans records from the database, you can temporarily modify your schedule to run once at a specific time. You can then run `sam deploy` again to reset your recurrence schedule to run once a month.
**To run the application using the AWS Management Console**
1. Navigate back to the EventBridge Scheduler console page.
1. Choose your schedule, then choose **Edit**.
1. In the **Schedule pattern** section, under **Recurrence**, choose **One-time schedule**.
1. Set your invocation time to a few minutes from now, review your settings, then choose **Save**.
After the schedule runs and invokes its target, you run the `test_app.py` script to verify that your function successfully removed all old records from the DynamoDB table.
**To verify that old records are deleted using a Python script**
1. In your command line, navigate to the folder where you saved `test_app.py`.
1. Run the script.
```
python test_app.py
```
If successful, you will see the following output.
```
Total number of old records: 0
```
## Next steps
You can now modify the EventBridge Scheduler schedule to meet your particular application requirements. EventBridge Scheduler supports the following schedule expressions: cron, rate, and one-time schedules.
For more information about EventBridge Scheduler schedule expressions, see [Schedule types](https://docs.aws.amazon.com/scheduler/latest/UserGuide/schedule-types.html) in the *EventBridge Scheduler User Guide*. [Access Management](https://docs.aws.amazon.com/IAM/latest/UserGuide/access.html) in the *IAM User Guide*