Export AWS Backup reports from across an organization in AWS Organizations as a CSV file - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resources

Export AWS Backup reports from across an organization in AWS Organizations as a CSV file

Created by Aromal Raj Jayarajan (AWS) and Purushotham G K (AWS)

Code repository: aws-backup-report-generator

Environment: PoC or pilot

Technologies: DevOps; Infrastructure

AWS services: AWS Backup; AWS Identity and Access Management; AWS Lambda; Amazon S3; Amazon EventBridge

Summary

This pattern shows how to export AWS Backup job reports from across an organization in AWS Organizations as a CSV file. The solution uses AWS Lambda and Amazon EventBridge to categorize AWS Backup job reports based on their status, which can help when configuring status-based automations.

AWS Backup helps organizations centrally manage and automate data protection across AWS services, in the cloud, and on premises. However, for AWS Backup jobs configured within AWS Organizations, consolidated reporting is available only in the AWS Management Console of each organization’s management account. Bringing this reporting outside of the management account can reduce the effort required for auditing and increase the scope for automations, notifications, and alerting.

Prerequisites and limitations

Prerequisites

Limitations

The solution provided in this pattern identifies AWS resources that are configured for AWS Backup jobs only. The report can’t identify AWS resources that aren’t configured for backup through AWS Backup.

Architecture

Target technology stack

  • AWS Backup

  • AWS CloudFormation

  • Amazon EventBridge

  • AWS Lambda

  • AWS Security Token Service (AWS STS)

  • Amazon Simple Storage Service (Amazon S3)

  • AWS Identity and Access Management (IAM)

Target architecture

The following diagram shows an example workflow for exporting AWS Backup job reports from across an organization in AWS Organizations as a CSV file.

Using EventBridge, Lambda, AWS STS, and IAM to export AWS Backup job reports from across an organization in CSV format.

The diagram shows the following workflow:

  1. A scheduled EventBridge event rule invokes a Lambda function in the member (reporting) AWS account.

  2. The Lambda function then uses AWS STS to assume an IAM role that has the permissions required to connect to the management account.

  3. The Lambda function then does the following:

    • Requests the consolidated AWS Backup jobs report from the AWS Backup service

    • Categorizes the results based on AWS Backup job status

    • Converts the response to a CSV file

    • Uploads the results to an Amazon S3 bucket in the reporting account within folders that are labeled based on their creation date

Tools

Tools

  • AWS Backup is a fully managed service that helps you centralize and automate data protection across AWS services, in the cloud, and on premises.

  • AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.

  • Amazon EventBridge is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.

  • AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

  • AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

  • Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

Code

The code for this pattern is available in the GitHub aws-backup-report-generator repository.

Best practices

Epics

TaskDescriptionSkills required

Clone the GitHub repository.

Clone the GitHub aws-backup-report-generator repository by running the following command in a terminal window:

git clone https://github.com/aws-samples/aws-backup-report-generator.git

For more information, see Cloning a repository in the GitHub Docs.

AWS DevOps, DevOps engineer

Deploy the solution components in the member (reporting) AWS account.

  1. In the member (reporting) account, sign in to the AWS Management Console and then open the CloudFormation console.

  2. Choose Create stack, and then choose With new resources (standard).

  3. On the Create stack page, in the Specify template section, choose Upload a template file.

  4. Select Choose file. Then, navigate to the root folder of the cloned GitHub repository on your local workstation and choose template-reporting.yaml.

  5. Choose Open, and then choose Next.

  6. On the Specify stack details page, for Stack name, enter a name for your CloudFormation stack.

  7. For ManagementAccountID, enter the AWS account ID for your organization’s management account in AWS Organizations.

  8. Choose Next.

  9. On the Configure Stack Options page, choose Next.

  10. On the Review page, select the check box to acknowledge that you reviewed the configuration.

  11. Choose Create stack. The stack shows CREATE_COMPLETE status when the solution components are deployed in the member (reporting) account.

DevOps engineer, AWS DevOps
TaskDescriptionSkills required

Make sure that the EventBridge rule runs prior to testing.

Make sure that the EventBridge rule runs by waiting at least 24 hours, or by increasing the report frequency in the CloudFormation template’s template-reporting.yml file.

To increase the report frequency

  1. Open the template-reporting.yml file in the cloned repository.

  2. In the event rule with the logical ID ‘LambdaSchedule’, find the ‘ScheduleExpression’.

  3. Edit the ‘ScheduleExpression’ key so that it includes a valid cron expression. For example, the following cron expression schedules the event rule to run every five minutes: “cron (* /5 * * * *)”

AWS DevOps, DevOps engineer

Check the Amazon S3 bucket for the generated report.

  1. In the member (reporting) account, sign in to the AWS Management Console and then open the CloudFormation console.

  2. In the Stacks pane, select the name of the stack you created. Then, choose the Resources tab.

  3. In the Resources pane, in the Logical ID column, find BackupReportS3Bucket. Then, open the associated Amazon S3 bucket in a new tab by selecting the link in the Physical ID column next to that logical ID.

  4. Make sure that the bucket contains a report that’s generated in the following format: BackupReports/<yyyy>/<mm>/<dd>/BackupReport-<BACKUP JOB STATUS>-<dd>-<Mon>-<yyyy>.csv

AWS DevOps, DevOps engineer
TaskDescriptionSkills required

Delete the solution components from the member (reporting) account.

  1. In the member (reporting) account, open the solution’s Amazon S3 bucket. For instructions, see steps 2-4 in the Check the S3 bucket for the generated report story of the Test the solution section of this pattern.

  2. Delete the contents of the bucket and empty the bucket. For instructions, see Emptying a bucket in the Amazon S3 User Guide.

  3. In the member (reporting) account, sign in to the AWS Management Console and then open the CloudFormation console.

  4. In the Stacks pane, select the checkbox next to the name of the stack you created. Then, choose Delete.

AWS DevOps, DevOps engineer

Delete the solution components from the management account.

  1. In the management account, sign in to the AWS Management Console and then open the CloudFormation console.

  2. In the Stacks pane, select the checkbox next to the name of the stack you created. Then, choose Delete.

AWS DevOps, DevOps engineer

Related resources