# Migration & modernization
**Topics**
+ [Migration](migration-pattern-list.md)
+ [Modernization](modernization-pattern-list.md)
+ [Mainframes](mainframe-pattern-list.md)
# Migration
**Topics**
+ [Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel and Python](create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel-and-python.md)
+ [Get started with automated portfolio discovery](get-started-with-automated-portfolio-discovery.md)
+ [Migrate on-premises Cloudera workloads to Cloudera Data Platform on AWS](migrate-on-premises-cloudera-workloads-to-cloudera-data-platform-on-aws.md)
+ [Resolve connection errors after migrating Microsoft SQL Server to the AWS Cloud](resolve-connection-errors-after-migrating-microsoft-sql-server-to-the-aws-cloud.md)
+ [Restart the AWS Replication Agent automatically without disabling SELinux after rebooting a RHEL source server](restart-the-aws-replication-agent-automatically-without-disabling-selinux-after-rebooting-a-rhel-source-server.md)
+ [Re-architect](migration-rearchitect-pattern-list.md)
+ [Rehost](migration-rehost-pattern-list.md)
+ [Relocate](migration-relocate-pattern-list.md)
+ [Replatform](migration-replatform-pattern-list.md)
+ [Migration patterns by workload](migration-migration-patterns-by-workload-pattern-list.md)
+ [More patterns](migration-more-patterns-pattern-list.md)
# Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel and Python
*Venkata Naveen Koppula, Amazon Web Services*
## Summary
This pattern outlines steps for automatically creating AWS CloudFormation templates for [AWS Database Migration Service](https://aws.amazon.com/dms/) (AWS DMS) using Microsoft Excel and Python.
Migrating databases using AWS DMS often involves creation of AWS CloudFormation templates to provision AWS DMS tasks. Previously, creating AWS CloudFormation templates required knowledge of the JSON or YAML programming language. With this tool, you only need basic knowledge of Excel and how to run a Python script using a terminal or command window.
As input, the tool takes an Excel workbook that includes the names of the tables to be migrated, Amazon Resource Names (ARNs) of AWS DMS endpoints, and AWS DMS replication instances. The tool then generates AWS CloudFormation templates for the required AWS DMS tasks.
For detailed steps and background information, see the blog post [Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel](https://aws.amazon.com/blogs/database/create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel/) in the AWS Database blog.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Microsoft Excel version 2016 or later
+ Python version 2.7 or later
+ The **xlrd** Python module (installed at a command prompt with the command: **pip install xlrd**)
+ AWS DMS source and target endpoints and AWS DMS replication instance
**Limitations**
+ The names of schemas, tables, and associated columns are transformed into lowercase characters at the destination endpoints.
+ This tool doesn’t address the creation of AWS DMS endpoints and replication instances.
+ Currently, the tool supports only one schema for each AWS DMS task.
## Architecture
**Source technology stack**
+ An on-premises database
+ Microsoft Excel
**Target technology stack**
+ AWS CloudFormation templates
+ A database in the AWS Cloud
**Architecture**
![\[Workflow to use Excel and Python to automatically create CloudFormation templates for AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/778c7c1e-2647-496f-8afd-52ff1ef02489/images/8fe1550d-8966-41aa-a480-5f7bef20629f.png)
## Tools
+ [Pycharm IDE](https://aws.amazon.com/pycharm/), or any integrated development environment (IDE) that supports Python version 3.6
+ Microsoft Office 2016 (for Microsoft Excel)
## Epics
### Configure the network, AWS DMS replication instance, and endpoints
| Task | Description | Skills required |
| --- | --- | --- |
| If necessary, request a service quota increase. | Request a service quota increase for the AWS DMS tasks if needed. | General AWS |
| Configure the AWS Region, virtual private clouds (VPCs), CIDR ranges, Availability Zones, and subnets. | | General AWS |
| Configure the AWS DMS replication instance. | The AWS DMS replication instance can connect to both on-premises and AWS databases. | General AWS |
| Configure AWS DMS endpoints. | Configure endpoints for both the source and target databases. | General AWS |
### Prepare the worksheets for AWS DMS tasks and tags
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the tables list. | List all tables involved in the migration. | Database |
| Prepare the tasks worksheet. | Prepare the Excel worksheet using the tables list you configured. | General AWS, Microsoft Excel |
| Prepare the tags worksheet. | Detail the AWS resource tags to attach to the AWS DMS tasks. | General AWS, Microsoft Excel |
### Download and run the tool
| Task | Description | Skills required |
| --- | --- | --- |
| Download and extract the template generation tool from the GitHub repository. | GitHub repository: https://github.com/aws-samples/dms-cloudformation-templates-generator/ | |
| Run the tool. | Follow the detailed instructions in the blog post listed under "References and help." | |
## Related resources
+ [Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel (blog post)](https://aws.amazon.com/blogs/database/create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel/)
+ [DMS CloudFormation Templates Generator (GitHub repository)](https://github.com/aws-samples/dms-cloudformation-templates-generator/tree/v1.0)
+ [Python documentation](https://www.python.org/)
+ [xlrd description and download](https://pypi.org/project/xlrd/)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/)
+ [AWS CloudFormation documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/)
# Get started with automated portfolio discovery
*Pratik Chunawala and Rodolfo Jr. Cerrada, Amazon Web Services*
## Summary
Assessing the portfolio and collecting metadata is a critical challenge when migrating applications and servers to the Amazon Web Services (AWS) Cloud, especially for large migrations that have more than 300 servers. Using an automated portfolio discovery tool can help you collect information about your applications, such as the number of users, frequency of use, dependencies, and information about the application’s infrastructure. This information is essential when planning migration waves so that you can properly prioritize and group applications with similar traits. Using a discovery tool streamlines communication between the portfolio team and the application owners because the portfolio team can validate the results of the discovery tool rather than manually collecting the metadata. This pattern discusses key considerations for selecting an automated discovery tool and information about how to deploy and test one in your environment.
This pattern includes a template, which is a starting point for building your own checklist of high-level activities. Next to the checklist is template for a responsible, accountable, consulted, informed (RACI) matrix. You can use this RACI matrix to determine who is responsible for each task in your checklist.
## Epics
### Select a discovery tool
| Task | Description | Skills required |
| --- | --- | --- |
| Determine whether a discovery tool is appropriate for your use case. | A discovery tool might not be the best solution for your use case. Consider the amount of time required to select, procure, prepare, and deploy a discovery tool. It can take 4–8 weeks to set up the scanning appliance for an agentless discovery tool in your environment or to install agents to all in-scope workloads. Once deployed, you must allow 4–12 weeks for the discovery tool to collect metadata by scanning the application workloads and performing application stack analysis. If you are migrating fewer than 100 servers, you might be able to manually collect the metadata and analyze dependencies faster than the time required to deploy and collect metadata with an automated discovery tool. | Migration lead, Migration engineer |
| Select a discovery tool. | Review the **Considerations for selecting an automated discovery tool** in the [Additional information](#get-started-with-automated-portfolio-discovery-additional) section. Determine the appropriate criteria for selecting a discovery tool for your use case, and then evaluate each tool against those criteria. For a comprehensive list of automated discovery tools, see [Discovery, Planning, and Recommendation migration tools](https://aws.amazon.com/prescriptive-guidance/migration-tools/migration-discovery-tools/). | Migration lead, Migration engineer |
### Prepare for installation
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the pre-deployment checklist. | Create a checklist of the tasks you must complete before deploying the tool. For an example, see [Predeployment Checklist](https://docs.flexera.com/foundationcloudscape/ug/Content/helplibrary/FCGS_Predeployment.htm) on the Flexera documentation website. | Build lead, Migration engineer, Migration lead, Network administrator |
| Prepare the network requirements. | Provision the ports, protocols, IP addresses, and routing necessary for the tool to run and access the target servers. For more information, see the installation guide for your discovery tool. For an example, see [Deployment Requirements](https://docs.flexera.com/foundationcloudscape/help/RCDeployReq.htm) on the Flexera documentation website. | Migration engineer, Network administrator, Cloud architect |
| Prepare the account and credential requirements. | Identify the credentials you need to access the target servers and to install all of the tool’s components. | Cloud administrator, General AWS, Migration engineer, Migration lead, Network administrator, AWS administrator |
| Prepare the appliances on which you will install the tool. | Ensure that the appliances on which you will install the tool components meet the specifications and platform requirements for the tool. | Migration engineer, Migration lead, Network administrator |
| Prepare the change orders. | According to the change management process in your organization, prepare the any change orders needed, and ensure these change orders are approved. | Build lead, Migration lead |
| Send requirements to stakeholders. | Send the pre-deployment checklist and network requirements to the stakeholders. Stakeholders should review, evaluate, and prepare the necessary requirements before proceeding with the deployment. | Build lead, Migration lead |
### Deploy the tool
| Task | Description | Skills required |
| --- | --- | --- |
| Download the installer. | Download the installer or the virtual machine image. Virtual machine images typically come in Open Virtualization Format (OVF). | Build lead, Migration lead |
| Extract the files. | If you are using an installer, you must download and run the installer on an on-premises server. | Build lead, Migration lead |
| Deploy the tool on the servers. | Deploy the discovery tool on the target, on-premises servers as follows:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/get-started-with-automated-portfolio-discovery.html) | Build lead, Migration lead, Network administrator |
| Log in to the discovery tool. | Follow the on-screen prompts, and log in to get started with the tool. | Migration lead, Build lead |
| Activate the product. | Enter your license key. | Build lead, Migration lead |
| Configure the tool. | Enter any credentials necessary to access the target servers, such as credentials for Windows, VMware, Simple Network Management Protocol (SNMP), and Secure Shell Protocol (SSH), or databases. | Build lead, Migration lead |
### Test the tool
| Task | Description | Skills required |
| --- | --- | --- |
| Select test servers. | Identify a small set of non-production subnets or IP addresses that you can use to test the discovery tool. This helps you validate the scans quickly, identify and troubleshoot any errors quickly, and isolate your tests from production environments. | Build lead, Migration lead, Network administrator |
| Start scanning the selected test servers. | For an agentless discovery tool, enter the subnets or IP addresses for the selected test servers in the discovery tool console, and start the scan.For an agent-based discovery tool, install the agent on the selected test servers. | Build lead, Migration lead, Network administrator |
| Review the scan results. | Review the scan results for the test servers. If any errors are found, troubleshoot and fix the errors. Document the errors and solutions. You reference this information in the future, and you can add this information to your portfolio runbook. | Build lead, Migration lead, Network administrator |
| Rescan the test servers. | Once the rescan is complete, repeat the scan until there are no errors. | Build lead, Migration lead, Network administrator |
## Related resources
**AWS resources**
+ [Application portfolio assessment guide for AWS Cloud migration](https://docs.aws.amazon.com/prescriptive-guidance/latest/application-portfolio-assessment-guide/introduction.html)
+ [Discovery, Planning, and Recommendation migration tools](https://aws.amazon.com/prescriptive-guidance/migration-tools/migration-discovery-tools/)
**Deployment guides for commonly selected discovery tools**
+ [Deploy the RN150 virtual appliance](https://docs.flexera.com/foundationcloudscape/ug/Content/helplibrary/FCGS_QSG_DeployRN150.htm) (Flexera documentation)
+ [Gatherer Installation](https://www.modelizeit.com/documentation/ADC-Gatherer-Install.html) (modelizeIT documentation)
+ [On-Prem Analysis Server Installation](https://www.modelizeit.com/documentation/RejuvenApptor-Install.html) (modelizeIT documentation)
## Additional information
**Considerations for selecting an automated discovery tool**
Each discovery tool has benefits and limitations. When selecting the appropriate tool for your use case, consider the following:
+ Select a discovery tool that can collect most, if not all, of the metadata you need to achieve your portfolio assessment goal.
+ Identify any metadata you need to gather manually because the tool doesn’t support it.
+ Provide the discovery tool requirements to stakeholders so they can review and assess the tool based on their internal security and compliance requirements, such as server, network, and credential requirements.
+ Does the tool require that you install an agent in the in-scope workload?
+ Does the tool require that you set up a virtual appliance in your environment?
+ Determine your data residency requirements. Some organizations don’t want to store their data outside of their environment. To address this, you might need to install some components of the tool in the on-premises environment.
+ Make sure the tool supports the operating system (OS) and OS version of the in-scope workload.
+ Determine whether your portfolio includes mainframe, mid-range, and legacy servers. Most of the discovery tools can detect these workloads as dependencies, but some tools might not be able to get device details, such as utilization and server dependencies. Device42 and modernizeIT discovery tools both support mainframe and mid-range servers.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/8c9d84de-e84a-4b0c-bcaa-389cd90be1f0/attachments/attachment.zip)
# Migrate on-premises Cloudera workloads to Cloudera Data Platform on AWS
*Battulga Purevragchaa and Nidhi Gupta, Amazon Web Services*
*Nijjwol Lamsal, Cloudera, Inc.*
## Summary
This pattern describes the high-level steps for migrating your on-premises Cloudera Distributed Hadoop (CDH), Hortonworks Data Platform (HDP), and Cloudera Data Platform (CDP) workloads to CDP Public Cloud on AWS. We recommend that you partner with Cloudera Professional Services and a systems integrator (SI) to implement these steps.
There are many reasons Cloudera customers want to move their on-premises CDH, HDP, and CDP workloads to the cloud. Some typical reasons include:
+ Streamline adoption of new data platform paradigms such as data lakehouse or data mesh
+ Increase business agility, democratize access and inference on existing data assets
+ Lower the total cost of ownership (TCO)
+ Enhance workload elasticity
+ Enable greater scalability; drastically reduce time to provision data services compared with legacy, on-premises install base
+ Retire legacy hardware; significantly reduce hardware refresh cycles
+ Take advantage of pay-as-you-go pricing, which is extended to Cloudera workloads on AWS with the Cloudera licensing model (CCU)
+ Take advantage of faster deployment and improved integration with continuous integration and continuous delivery (CI/CD) platforms
+ Use a single unified platform (CDP) for multiple workloads
Cloudera supports all major workloads, including Machine Learning, Data Engineering, Data Warehouse, Operational Database, Stream Processing (CSP), and data security and governance. Cloudera has offered these workloads for many years on premises, and you can migrate these workloads to the AWS Cloud by using CDP Public Cloud with Workload Manager and Replication Manager.
Cloudera Shared Data Experience (SDX) provides a shared metadata catalog across these workloads to facilitate consistent data management and operations. SDX also includes comprehensive, granular security to protect against threats, and unified governance for audit and search capabilities for compliance with standards such as Payment Card Industry Data Security Standard (PCI DSS) and GDPR.
**CDP migration at a glance**
| | |
| --- |--- |
| Workload | Source workload | CDH, HDP, and CDP Private Cloud |
| --- |--- |--- |
| Source environment | Windows, LinuxOn-premises, colocation, or any non-AWS environment |
| Destination workload | CDP Public Cloud on AWS |
| Destination environment | Deployment model: customer accountOperating model: customer/Cloudera control plane |
| ** **** ****Migration** | Migration strategy (7Rs) | Rehost, replatform, or refactor |
| Is this an upgrade in the workload version? | Yes |
| Migration duration | Deployment: About 1 week to create customer account, virtual private cloud (VPC), and CDP Public Cloud customer-managed environment.Migration duration: 1-4 months, depending on the complexity and size of the workload. |
| **Cost** | Cost of running the workload on AWS | At a high level, the cost of a CDH workload migration to AWS assumes that you will establish a new environment on AWS. It includes accounting for personnel time and effort as well as provisioning computing resources and licensing software for the new environment.The Cloudera cloud consumption-based pricing model gives you the flexibility to take advantage of bursting and automatic scaling capabilities. For more information, see [CDP Public Cloud service rates](https://www.cloudera.com/products/pricing/cdp-public-cloud-service-rates.html) on the Cloudera website.Cloudera Enterprise [Data Hub](https://www.cloudera.com/products/enterprise-data-hub.html) is based on Amazon Elastic Compute Cloud (Amazon EC2) and closely models traditional clusters. Data Hub can be [customized](https://docs.cloudera.com/data-hub/cloud/create-cluster-aws/topics/mc-creating-a-cluster.html), but this will affect costs.[CDP Public Cloud Data Warehouse](https://docs.cloudera.com/data-warehouse/cloud/index.html), [Cloudera Machine Learning](https://docs.cloudera.com/machine-learning/cloud/product/topics/ml-product-overview.html), and [Cloudera Data Engineering (CDE)](https://docs.cloudera.com/data-engineering/cloud/index.html) are container-based and can be configured to scale automatically. |
| ** **** ****Infrastructure agreements and framework** | System requirements | See the [Prerequisites](#migrate-on-premises-cloudera-workloads-to-cloudera-data-platform-on-aws-prereqs) section. |
| SLA | See [Cloudera Service Level Agreement for CDP Public Cloud.](https://www.cloudera.com/legal/terms-and-conditions/cdp-public-cloud-sla.html) |
| DR | See [Disaster Recovery](https://docs.cloudera.com/cdp-reference-architectures/latest/cdp-ra-operations/topics/cdp-ra-abstract.html) in the Cloudera documentation. |
| Licensing and operating model (for target AWS account) | Bring Your Own License (BYOL) model |
| ** ****Compliance** | Security requirements | See [Cloudera Security Overview](https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/security-overview/topics/cm-security-overview.html) in the Cloudera documentation. |
| Other [compliance certifications](https://aws.amazon.com/compliance/programs) | See the information on the Cloudera website about [General Data Protection Regulation (GDPR](https://www.cloudera.com/solutions/lower-business-risks/general-data-protection-regulation.html)) compliance and the [CDP Trust Center](https://www.cloudera.com/products/trust-center.html). |
## Prerequisites and limitations
**Prerequisites**
+ [AWS account requirements](https://docs.cloudera.com/cdp-public-cloud/cloud/requirements-aws/topics/mc-requirements-aws.html), including accounts, resources, services, and permissions, such as AWS Identity and Access Management (IAM) roles and policies setup
+ [Prerequisites for deploying CDP](https://docs.cloudera.com/cdp-public-cloud/cloud/getting-started/topics/cdp-set_up_cdp_prerequisites.html) from the Cloudera website
The migration requires the following roles and expertise:
|
|
| Role | Skills and responsibilities |
| --- |--- |
| Migration lead | Ensures executive support, team collaboration, planning, implementation, and assessment |
| Cloudera SME | Expert skills in CDH, HDP, and CDP administration, system administration, and architecture |
| AWS architect | Skills in AWS services, networking, security, and architectures |
## Architecture
Building to the appropriate architecture is a critical step to ensure that migration and performance meet your expectations. For your migration effort to meet this playbook’s assumptions, your target data environment in the AWS Cloud, either on virtual private cloud (VPC) hosted instances or CDP, must be an equivalent match to your source environment in terms of operating system and software versions as well as major machine specifications.
The following diagram (reproduced with permission from the [Cloudera Shared Data Experience data sheet](https://www.cloudera.com/content/dam/www/marketing/resources/datasheets/cloudera-sdx-datasheet.pdf?daqp=true)) shows the infrastructure components for the CDP environment and how the tiers or infrastructure components interact.
![\[CDP environment components\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bb47435e-2638-425c-ac37-7d55053452ac/images/91d62277-7fde-4ec6-8e2b-86a446e2f6ee.png)
The architecture includes the following CDP components:
+ Data Hub is a service for launching and managing workload clusters powered by Cloudera Runtime. You can use the cluster definitions in Data Hub to provision and access workload clusters for custom use cases and define custom cluster configurations. For more information, see the [Cloudera website](https://docs.cloudera.com/data-hub/cloud/index.html).
+ Data Flow and Streaming addresses the key challenges enterprises face with data in motion. It manages the following:
+ Processing real-time data streaming at high volume and high scale
+ Tracking data provenance and lineage of streaming data
+ Managing and monitoring edge applications and streaming sources
For more information, see [Cloudera DataFlow](https://www.cloudera.com/products/dataflow.html) and [CSP](https://www.cloudera.com/products/stream-processing.html) on the Cloudera website.
+ Data Engineering includes data integration, data quality, and data governance, which help organizations build and maintain data pipelines and workflows. For more information, see the [Cloudera website](https://docs.cloudera.com/data-engineering/cloud/index.html). Learn about [support for spot instances to facilitate cost savings on AWS](https://docs.cloudera.com/data-engineering/cloud/cost-management/topics/cde-spot-instances.html) for Cloudera Data Engineering workloads.
+ Data Warehouse** **enables you to create independent data warehouses and data marts that automatically scale to meet workload demands. This service provides isolated compute instances and automated optimization for each data warehouse and data mart, and helps you save costs while meeting SLAs. For more information, see the [Cloudera website](https://docs.cloudera.com/data-warehouse/cloud/index.html). Learn about [managing costs](https://docs.cloudera.com/data-warehouse/cloud/planning/topics/dw-manage-cloud-costs.html) and [auto-scaling](https://docs.cloudera.com/data-warehouse/cloud/auto-scaling/topics/dw-public-cloud-autoscaling-overview.html) for Cloudera Data Warehouse on AWS.
+ Operational Database in CDP provides a reliable and flexible foundation for scalable, high-performance applications. It delivers a real-time, always available, scalable database that serves traditional structured data alongside new, unstructured data within a unified operational and warehousing platform. For more information, see the [Cloudera website](https://www.cloudera.com/products/operational-db.html).
+ Machine Learning is a cloud-native machine learning platform that merges self-service data science and data engineering capabilities into a single, portable service within an enterprise data cloud. It enables scalable deployment of machine learning and artificial intelligence (AI) on data anywhere. For more information, see the [Cloudera website](https://docs.cloudera.com/machine-learning/cloud/index.html).
**CDP on AWS**
The following diagram (adapted with permission from the Cloudera website) shows the high-level architecture of CDP on AWS. CDP implements its [own security model](https://docs.cloudera.com/runtime/7.1.0/cdp-security-overview/topics/security-management-console-security.html) to manage both accounts and data flow. These are integrated with [IAM](https://aws.amazon.com/iam/) through the use of [cross-account roles](https://docs.cloudera.com/cdp-public-cloud/cloud/requirements-aws/topics/mc-aws-req-credential.html).
![\[CDP on AWS high-level architecture\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bb47435e-2638-425c-ac37-7d55053452ac/images/54420517-38b4-4e82-bd19-9ded50ed009c.png)
The CDP control plane resides in a Cloudera master account in its own VPC. Each customer account has its own sub-account and unique VPC. Cross-account IAM roles and SSL technologies route management traffic to and from the control plane to customer services that reside on internet-routable public subnets within each customer VPC. On the customer’s VPC, the Cloudera Shared Data Experience (SDX) provides enterprise-strength security with unified governance and compliance so you can get insights from your data faster. SDX is a design philosophy incorporated into all Cloudera products. For more information about [SDX](https://docs.cloudera.com/cdp-public-cloud/cloud/overview/topics/cdp-services.html) and the [CDP Public Cloud network architecture for AWS](https://docs.cloudera.com/cdp-public-cloud/cloud/aws-refarch/topics/cdp-pc-aws-refarch-overview.html), see the Cloudera documentation.
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Automation and tooling**
+ For additional tooling, you can use [Cloudera Backup Data Recovery (BDR)](https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_bdr_tutorials.html), [AWS Snowball](https://aws.amazon.com/snowball/), and [AWS Snowmobile](https://aws.amazon.com/snowmobile/) to help migrate data from on-premises CDH, HDP, and CDP to AWS-hosted CDP.
+ For new deployments, we recommend that you use the [AWS Partner Solution for CDP](https://aws.amazon.com/solutions/partners/terraform-modules/cdp-public-cloud/).
## Epics
### Prepare for migration
| Task | Description | Skills required |
| --- | --- | --- |
| Engage the Cloudera team. | Cloudera pursues a standardized engagement model with its customers and can work with your systems integrator (SI) to promote the same approach. Contact the Cloudera customer team so they can provide guidance and the necessary technical resources to get the project started. Contacting the Cloudera team ensures that all necessary teams can prepare for the migration as its date approaches. You can contact Cloudera Professional Services to move your Cloudera deployment from pilot to production quickly, at lower cost, and with peak performance. For a complete list of offerings, see the [Cloudera website](https://www.cloudera.com/about/services-and-support/professional-services.html). | Migration lead |
| Create a CDP Public Cloud environment on AWS for your VPC. | Work with Cloudera Professional Services or your SI to plan and deploy CDP Public Cloud into a VPC on AWS. | Cloud architect, Cloudera SME |
| Prioritize and assess workloads for migration. | Evaluate all your on-premises workloads to determine the workloads that are the easiest to migrate. Applications that aren’t mission-critical are the best to move first, because they will have minimal impact on your customers. Save the mission-critical workloads for last, after you successfully migrate other workloads.Transient (CDP Data Engineering) workloads are easier to migrate than persistent (CDP Data Warehouse) workloads. It’s also important to consider data volume and locations when migrating. Challenges can include replicating data continuously from an on-premises environment to the cloud, and changing the data ingestion pipelines to import data directly to the cloud. | Migration lead |
| Discuss CDH, HDP, CDP, and legacy application migration activities. | Consider and start planning for the following activities with Cloudera Workload Manager:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-on-premises-cloudera-workloads-to-cloudera-data-platform-on-aws.html) | Migration lead |
| Complete the Cloudera Replication Manager requirements and recommendations. | Work with Cloudera Professional Services and your SI to prepare to migrate workloads to your CDP Public Cloud environment on AWS. Understanding the following requirements and recommendations can help you avoid common issues during and after you install the Replication Manager service.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-on-premises-cloudera-workloads-to-cloudera-data-platform-on-aws.html) | Migration lead |
### Migrate CDP to AWS
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the first workload for dev/test environments by using Cloudera Workload Manager. | Your SI can help you migrate your first workload to the AWS Cloud. This should be an application that isn’t customer-facing or mission-critical. Ideal candidates for dev/test migration are applications that have data that the cloud can easily ingest, such as CDP Data Engineering workloads. This is a transient workload that usually has fewer users accessing it, compared with a persistent workload such as a CDP Data Warehouse workload that could have many users who need uninterrupted access. Data Engineering workloads aren’t persistent, which minimizes the business impact if something goes wrong. However, these jobs could be critical for production reporting, so prioritize low-impact Data Engineering workloads first. | Migration lead |
| Repeat migration steps as necessary. | Cloudera Workload Manager helps identify workloads that are best suited for the cloud. It provides metrics such as cloud performance ratings, sizing/capacity plans for the target environment, and replication plans. The best candidates for migration are seasonal workloads, ad hoc reporting, and intermittent jobs that don’t consume many resources.Cloudera Replication Manager moves data from on premises to the cloud, and from the cloud to on premises.Proactively optimize workloads, applications, performance, and infrastructure capacity for data warehousing, data engineering, and machine learning by using Workload Manager. For a complete guide on how to modernize a data warehouse, see the [Cloudera website](https://www.cloudera.com/content/dam/www/marketing/resources/webinars/modern-data-warehouse-fundamentals.png.landing.html). | Cloudera SME |
## Related resources
Cloudera documentation:
+ [Registering classic clusters with CDP, Cloudera Manager, and Replication Manager:](https://docs.cloudera.com/replication-manager/cloud/operations/topics/rm-requirements-for-bdr-cdh-clusters.html)
+ [Management Console](https://docs.cloudera.com/management-console/cloud/overview/topics/mc-management-console.html)
+ [Replication Manager hive replication](https://docs.cloudera.com/replication-manager/cloud/core-concepts/topics/rm-replication-of-data-using-hive.html)
+ [Sentry replication](https://docs.cloudera.com/replication-manager/cloud/core-concepts/topics/rm-sentry-policy-replication.html)
+ [Sentry permissions](https://docs.cloudera.com/replication-manager/cloud/core-concepts/topics/rm-sentry-ranger-permissions.html)
+ [Data Hub cluster planning checklist](https://docs.cloudera.com/data-hub/cloud/cluster-planning/topics/dh-cluster-checklist.html)
+ [Workload Manager architecture](https://docs.cloudera.com/workload-manager/cloud/configuration/topics/wm-public-architecture-wm.html)
+ [Replication Manager requirements](https://docs.cloudera.com/replication-manager/cloud/index.html)
+ [Cloudera Data Platform Observability](https://www.cloudera.com/products/observability.html)
+ [AWS requirements](https://docs.cloudera.com/cdp-public-cloud/cloud/requirements-aws/topics/mc-requirements-aws.html)
AWS documentation:
+ [Cloud Data Migration](https://aws.amazon.com/cloud-data-migration/)
# Resolve connection errors after migrating Microsoft SQL Server to the AWS Cloud
*Premkumar Chelladurai, Amazon Web Services*
## Summary
After you migrate Microsoft SQL Server running on Windows Server 2008 R2, 2012, or 2012 R2 to Amazon Elastic Compute Cloud (Amazon EC2) instances on the Amazon Web Services (AWS) Cloud, the connection to SQL Server fails and the following errors appear:
+ `[Microsoft][ODBC SQL Server Driver][DBNETLIB] General Network error`
+ `ERROR [08S01] [Microsoft][SQL Native Client]Communication link failure. System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)`
+ `TCP Provider: The semaphore timeout period has expired`
This pattern describes how you can resolve these errors by turning off the Windows Scalable Networking Pack (SNP) features at the operating system (OS) and network interface level for SQL Server running on Windows Server 2008 R2, 2012, or 2012 R2.
## Prerequisites and limitations
**Prerequisites**
+ Administrator privileges for Windows Server.
+ If you used AWS Application Migration Service as your migration tool, you require one of the following Windows Server versions:
+ Windows Server 2008 R2 Service Pack 1, 2012, or 2012 R2
+ If you used CloudEndure Migration as your migration tool, you require one of the following Windows Server versions:
+ Windows Server 2003 R2 Service Pack 3, 2008, 2008 R2 Service Pack 1, 2012, or 2012 R2
## Tools
+ [Amazon EC2](https://docs.aws.amazon.com/ec2/index.html) – Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the AWS Cloud. You can use Amazon EC2 to launch as many or as few virtual servers as you need, and you can scale out or scale in.
+ [Windows Server](https://docs.microsoft.com/en-us/windows-server/) – Windows Server is a platform for building an infrastructure of connected applications, networks, and web services.
## Epics
### Turn off SNP features at the OS and elastic network interface levels
| Task | Description | Skills required |
| --- | --- | --- |
| Turn off SNP features at the OS level. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/resolve-connection-errors-after-migrating-microsoft-sql-server-to-the-aws-cloud.html) | AWS administrator, AWS systems administrator, Migration engineer, Cloud administrator |
| Turn off SNP features at the elastic network interface level. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/resolve-connection-errors-after-migrating-microsoft-sql-server-to-the-aws-cloud.html) | AWS administrator, Cloud administrator, AWS systems administrator |
## Related resources
+ [How to troubleshoot advanced network performance features such as RSS and NetDMA](https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/troubleshoot-network-performance-features-rss-netdma)
# Restart the AWS Replication Agent automatically without disabling SELinux after rebooting a RHEL source server
*Anil Kunapareddy, Venkatramana Chintha, and Shanmugam Shanker, Amazon Web Services*
## Summary
AWS Application Migration Service helps simplify, expedite, and automate the migration of your Red Hat Enterprise Linux (RHEL) workload to the Amazon Web Services (AWS) Cloud. To add source servers to Application Migration Service, you install the AWS Replication Agent on the servers.
Application Migration Service provides real-time, asynchronous, block-level replication. This means that you can continue normal IT operations during the entire replication process. These IT operations might require that you reboot or restart your RHEL source server during the migration. If this happens, the AWS Replication Agent will not restart automatically, and your data replication will stop. Typically, you can set Security-Enhanced Linux (SELinux) to **disabled** or **permissive** mode to automatically restart AWS Replication Agent. However, your organization’s security policies might prohibit disabling SELinux, and you might also have to [relabel your files](https://access.redhat.com/solutions/3176).
This pattern describes how to automatically restart the AWS Replication Agent without turning off SELinux when your RHEL source server reboots or restarts during a migration.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ An on-premises RHEL workload that you want to migrate to the AWS Cloud.
+ Application Migration Service initialized from the Application Migration Service console. Initialization is required only the first time you use this service. For instructions, see the [Application Migration Service documentation](https://docs.aws.amazon.com/mgn/latest/ug/mandatory-setup.html).
+ An existing [AWS Identity and Access Management (IAM) policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) for Application Migration Service. For more information, see the [Application Migration Service documentation](https://docs.aws.amazon.com/mgn/latest/ug/mgn-policies.html).
**Versions**
+ RHEL version 7 or later
## Tools
**AWS services**
+ [AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) is a highly automated lift-and-shift (rehost) solution that simplifies, expedites, and reduces the cost of migrating applications to AWS.
**Linux commands**
The following table provides a list of Linux commands that you will run on your RHEL source server. These are also described in the epics and stories for this pattern.
|
|
| Command | Description |
| --- |--- |
| `#systemctl –version` | Identifies the system version. |
| `#systemctl list-units --type=service` | Lists all active services that are available on the RHEL server. |
| `#systemctl list-units --type=service \| grep running` | Lists all services that are currently running on the RHEL server. |
| `#systemctl list-units --type=service \| grep failed` | Lists all services that failed to load after the RHEL server rebooted or restarted. |
| `restorecon -Rv /etc/rc.d/init.d/aws-replication-service` | Changes the context to `aws-replication-service`. |
| `yum install policycoreutils*` | Installs the policy core utilities that are required for the operation of the SELinux system. |
| `ausearch -c "insmod" --raw \| audit2allow -M my-modprobe` | Searches the audit log and creates a module for policies. |
| `semodule -i my-modprobe.pp` | Activates the policy. |
| `cat my-modprobe.te` | Displays the contents of the `my-modprobe.te` file. |
| `semodule -l \| grep my-modprobe` | Checks whether the policy has been loaded to the SELinux module. |
## Epics
### Install the AWS Replication Agent and reboot the RHEL source server
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Application Migration Service user with an access key and a secret access key. | To install the AWS Replication Agent, you must create an Application Migration Service user with the required AWS credentials. For instructions, see the [Application Migration Service documentation](https://docs.aws.amazon.com/mgn/latest/ug/credentials.html). | Migration engineer |
| Install the AWS Replication Agent. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/restart-the-aws-replication-agent-automatically-without-disabling-selinux-after-rebooting-a-rhel-source-server.html) | Migration engineer |
| Restart or reboot the RHEL source server. | Restart or reboot your RHEL source server when its **Data replication status** displays **Stalled **on the [Migration dashboard](https://docs.aws.amazon.com/mgn/latest/ug/migration-dashboard.html). | Migration engineer |
| Check data replication status. | Wait for one hour and then check the **Data replication status** again on the Migration dashboard. It should be in the **Healthy **state. | Migration engineer |
### Check AWS Replication Agent status on the RHEL source server
| Task | Description | Skills required |
| --- | --- | --- |
| Identify the system version. | Open the command line interface for your RHEL source server and run the following command to identify the system version:`#systemctl –version` | Migration engineer |
| List all active services. | To list all active services available on the RHEL server, run the command:`#systemctl list-units --type=service` | Migration engineer |
| List all running services. | To list all services that are currently running on the RHEL server, use the command:`#systemctl list-units --type=service \| grep running` | Migration engineer |
| List all services that failed to load. | To list all services that failed to load after the RHEL server rebooted or restarted, run the command:`#systemctl list-units --type=service \| grep failed` | Migration engineer |
### Create and run the SELinux module
| Task | Description | Skills required |
| --- | --- | --- |
| Change the security context. | In the command line interface for your RHEL source server, run the following command to change the security context to the AWS replication service:`restorecon -Rv /etc/rc.d/init.d/aws-replication-service` | Migration engineer |
| Install core utilities. | To install the core utilities required for the operation of the SELinux system and its policies, run the command:`yum install policycoreutils*` | Migration engineer |
| Search the audit log and create a module for policies. | Run the command:`ausearch -c "insmod" --raw \| audit2allow -M my-modprobe` | Migration engineer |
| Display the contents of the my-modprobe-te file. | The `my-modprobe.te` file is generated by the **audit2allow **command. It includes the SELinux domains, policy source directory, and subdirectories, and specifies the access vector rules and transitions associated with the domains. To display the contents of the file, run the command:`cat my modprobe.te` | Migration engineer |
| Activate the policy. | To insert the module and make the policy package active, run the command:`semodule -i my-modprobe.pp` | Migration engineer |
| Check whether the module has been loaded. | Run the command:`semodule -l \| grep my-modprobe`After the SELinux module is loaded, you will no longer have to set SELinux to **disabled** or **permissive** mode during your migration. | Migration engineer |
| Reboot or restart the RHEL source server and verify the data replication status. | Open the AWS Migration Service console, navigate to **Data replication progress**, and then reboot or restart your RHEL source server. Data replication should now resume automatically after the RHEL source server reboots. | Migration engineer |
## Related resources
+ [Application Migration service documentation](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html)
+ [Technical training materials](https://docs.aws.amazon.com/mgn/latest/ug/mgn-training.html)
+ [Troubleshooting AWS Replication Agent issues](https://docs.aws.amazon.com/mgn/latest/ug/Troubleshooting-Agent-Issues.html)
+ [Application Migration Service policies](https://docs.aws.amazon.com/mgn/latest/ug/mgn-policies.html)
# Re-architect
**Topics**
+ [Convert VARCHAR2(1) data type for Oracle to Boolean data type for Amazon Aurora PostgreSQL](convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.md)
+ [Create application users and roles in Aurora PostgreSQL-Compatible](create-application-users-and-roles-in-aurora-postgresql-compatible.md)
+ [Emulate Oracle DR by using a PostgreSQL-compatible Aurora global database](emulate-oracle-dr-by-using-a-postgresql-compatible-aurora-global-database.md)
+ [Implement SHA1 hashing for PII data when migrating from SQL Server to PostgreSQL](implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.md)
+ [Incrementally migrate from Amazon RDS for Oracle to Amazon RDS for PostgreSQL using Oracle SQL Developer and AWS SCT](incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.md)
+ [Load BLOB files into TEXT by using file encoding in Aurora PostgreSQL-Compatible](load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.md)
+ [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL with AWS SCT and AWS DMS using AWS CLI and CloudFormation](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.md)
+ [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL in SSL mode by using AWS DMS](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.md)
+ [Migrate Oracle SERIALLY\$1REUSABLE pragma packages into PostgreSQL](migrate-oracle-serially-reusable-pragma-packages-into-postgresql.md)
+ [Migrate Oracle external tables to Amazon Aurora PostgreSQL-Compatible](migrate-oracle-external-tables-to-amazon-aurora-postgresql-compatible.md)
+ [Migrate function-based indexes from Oracle to PostgreSQL](migrate-function-based-indexes-from-oracle-to-postgresql.md)
+ [Migrate Oracle native functions to PostgreSQL using extensions](migrate-oracle-native-functions-to-postgresql-using-extensions.md)
+ [Migrate a Db2 database from Amazon EC2 to Aurora MySQL-Compatible by using AWS DMS](migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.md)
+ [Migrate a Microsoft SQL Server database from Amazon EC2 to Amazon DocumentDB by using AWS DMS](migrate-a-microsoft-sql-server-database-from-amazon-ec2-to-amazon-documentdb-by-using-aws-dms.md)
+ [Migrate an on-premises ThoughtSpot Falcon database to Amazon Redshift](migrate-an-on-premises-thoughtspot-falcon-database-to-amazon-redshift.md)
+ [Migrate from Oracle Database to Amazon RDS for PostgreSQL by using Oracle GoldenGate](migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.md)
+ [Migrate an Oracle partitioned table to PostgreSQL by using AWS DMS](migrate-an-oracle-partitioned-table-to-postgresql-by-using-aws-dms.md)
+ [Migrate from Amazon RDS for Oracle to Amazon RDS for MySQL](migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-mysql.md)
+ [Migrate from IBM Db2 on Amazon EC2 to Aurora PostgreSQL-Compatible using AWS DMS and AWS SCT](migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.md)
+ [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using SharePlex and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-shareplex-and-aws-dms.md)
+ [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using materialized views and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-materialized-views-and-aws-dms.md)
+ [Migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-from-oracle-on-amazon-ec2-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md)
+ [Migrate an Oracle database from Amazon EC2 to Amazon RDS for MariaDB using AWS DMS and AWS SCT](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-mariadb-using-aws-dms-and-aws-sct.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-an-on-premises-oracle-database-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for PostgreSQL by using an Oracle bystander and AWS DMS](migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.md)
+ [Migrate an Oracle Database to Amazon Redshift using AWS DMS and AWS SCT](migrate-an-oracle-database-to-amazon-redshift-using-aws-dms-and-aws-sct.md)
+ [Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT](migrate-an-oracle-database-to-aurora-postgresql-using-aws-dms-and-aws-sct.md)
+ [Migrate data from an on-premises Oracle database to Aurora PostgreSQL](migrate-data-from-an-on-premises-oracle-database-to-aurora-postgresql.md)
+ [Migrate from SAP ASE to Amazon RDS for SQL Server using AWS DMS](migrate-from-sap-ase-to-amazon-rds-for-sql-server-using-aws-dms.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS DMS](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-dms.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS SCT data extraction agents](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-sct-data-extraction-agents.md)
+ [Migrate legacy applications from Oracle Pro\$1C to ECPG](migrate-legacy-applications-from-oracle-pro-c-to-ecpg.md)
+ [Migrate virtual generated columns from Oracle to PostgreSQL](migrate-virtual-generated-columns-from-oracle-to-postgresql.md)
+ [Set up Oracle UTL\$1FILE functionality on Aurora PostgreSQL-Compatible](set-up-oracle-utl_file-functionality-on-aurora-postgresql-compatible.md)
+ [Validate database objects after migrating from Oracle to Amazon Aurora PostgreSQL](validate-database-objects-after-migrating-from-oracle-to-amazon-aurora-postgresql.md)
# Convert VARCHAR2(1) data type for Oracle to Boolean data type for Amazon Aurora PostgreSQL
*Naresh Damera, Amazon Web Services*
## Summary
During a migration from Amazon Relational Database Service (Amazon RDS) for Oracle to Amazon Aurora PostgreSQL-Compatible Edition, you might encounter a data mismatch when validating the migration in AWS Database Migration Service (AWS DMS). To prevent this mismatch, you can convert VARCHAR2(1) data type to Boolean data type.
VARCHAR2 data type stores variable-length text strings, and VARCHAR2(1) indicates that the string is 1 character in length or 1 byte. For more information about VARCHAR2, see [Oracle built-in data types](https://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF30020) (Oracle documentation).
In this pattern, in the sample source data table column, the VARCHAR2(1) data is either a **Y**, for *Yes*, or **N**, for *No*. This pattern includes instructions for using AWS DMS and AWS Schema Conversion Tool (AWS SCT) to convert this data type from the **Y** and **N** values in VARCHAR2(1) to **true** or **false** values in Boolean.
**Intended audience**
This pattern is recommended for those who have experience migrating Oracle databases to Aurora PostgreSQL-Compatible by using AWS DMS. As you complete the migration, adhere to the recommendations in [Converting Oracle to Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.ToPostgreSQL.html) (AWS SCT documentation).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ Confirm that your environment is prepared for Aurora, including setting up credentials, permissions, and a security group. For more information, see [Setting up your environment for Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_SettingUp_Aurora.html) (Aurora documentation).
+ A source Amazon RDS for Oracle database that contains a table column with VARCHAR2(1) data.
+ A target Amazon Aurora PostgreSQL-Compatible database instance. For more information, see [Creating a database cluster and connecting to a database on an Aurora PostgreSQL database cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html#CHAP_GettingStarted.AuroraPostgreSQL.CreateDBCluster) (Aurora documentation).
**Product versions**
+ Amazon RDS for Oracle version 12.1.0.2 or later.
+ AWS DMS version 3.1.4 or later. For more information, see [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html) (AWS DMS documentation). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support.
+ AWS Schema Conversion Tool (AWS SCT) version 1.0.632 or later. We recommend that you use the latest version of AWS SCT for the most comprehensive version and feature support.
+ Aurora supports the PostgreSQL versions listed in [Database Engine Versions for Aurora PostgreSQL-Compatible](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Updates.20180305.html) (Aurora documentation).
## Architecture
**Source technology stack**
Amazon RDS for Oracle database instance
**Target technology stack**
Amazon Aurora PostgreSQL-Compatible database instance
**Source and target architecture**
![\[Changing data types from VARCHAR2(1) to Boolean\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5d4dc568-20d8-4883-a942-21c81039d8e6/images/9fd82ae2-56e6-439c-b4cd-9e74fe77b480.png)
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Relational Database Service (Amazon RDS) for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale an Oracle relational database in the AWS Cloud.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format compatible with the target database.
**Other services**
+ [Oracle SQL Developer](https://docs.oracle.com/en/database/oracle/sql-developer/) is an integrated development environment that simplifies the development and management of Oracle databases in both traditional and cloud-based deployments. In this pattern, you use this tool to connect to the Amazon RDS for Oracle database instance and query the data.
+ [pgAdmin](https://www.pgadmin.org/docs/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects. In this pattern, you use this tool to connect to the Aurora database instance and query the data.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Create database migration report. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.html) | DBA, Developer |
| Disable foreign key constraints on the target database. | In PostgreSQL, foreign keys are implemented by using triggers. During the full load phase, AWS DMS loads each table one at a time. We strongly recommend that you disable foreign key constraints during a full load by using one of the following methods:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.html)If disabling foreign key constraints is not feasible, create an AWS DMS migration task for the primary data that is specific to the parent table and child table. | DBA, Developer |
| Disable the primary keys and unique keys on the target database. | Using the following commands, disable the primary keys and constraints on the target database. This helps improve the performance of the initial load task.
ALTER TABLE
DISABLE PRIMARY KEY;
ALTER TABLE
DISABLE CONSTRAINT ; | DBA, Developer |
| Create the initial load task. | In AWS DMS, create the migration task for the initial load. For instructions, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html). For the migration method, choose **Migrate existing data**. This migration method is** **called `Full Load` in the API. Do not start this task yet. | DBA, Developer |
| Edit task settings for the initial load task. | Edit the task settings to add data validation. These validation settings must be created in a JSON file. For instructions and examples, see [Specifying task settings](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html). Add the following validations:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.html)To validate the rest of the data migration, enable data validation in the task. For more information, see [Data validation task settings](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.DataValidation.html). | AWS administrator, DBA |
| Create the ongoing replication task. | In AWS DMS, create the migration task that keeps the target database in sync with the source database. For instructions, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html). For the migration method, choose **Replicate data changes only**. Do not start this task yet. | DBA |
### Test the migration tasks
| Task | Description | Skills required |
| --- | --- | --- |
| Create sample data for testing. | In the source database, create a sample table with data for testing purposes. | Developer |
| Confirm there are no conflicting activities. | Use the `pg_stat_activity` to check for any activity on the server that might affect the migration. For more information, see [The Statistics Collector](https://www.postgresql.org/docs/current/monitoring-stats.html) (PostgreSQL documentation). | AWS administrator |
| Start the AWS DMS migration tasks. | In the AWS DMS console, on the **Dashboard** page, start the initial load and ongoing replication tasks that you created in the previous epic. | AWS administrator |
| Monitor the tasks and table load states. | During the migration, monitor the [task status](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html#CHAP_Tasks.Status) and the [table states](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html#CHAP_Tasks.CustomizingTasks.TableState). When the initial load task is complete, on the **Table statistics** tab:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.html) | AWS administrator |
| Verify the migration results. | Using pgAdmin, query the table on target database. A successful query indicates that the data was migrated successfully. | Developer |
| Add primary keys and foreign keys on the target database. | Create the primary key and foreign key on the target database. For more information, see [ALTER TABLE](https://www.postgresql.org/docs/current/sql-altertable.html) (PostgreSQL website). | DBA |
| Clean up the test data. | On the source and target databases, clean up data that was created for unit testing. | Developer |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Complete the migration. | Repeat the previous epic, *Test the migration tasks*, using the real source data. This migrates the data from the source to the target database. | Developer |
| Validate that the source and target databases are in sync. | Validate that the source and target databases are in sync. For more information and instructions, see [AWS DMS data validation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html). | Developer |
| Stop the source database. | Stop the Amazon RDS for Oracle database. For instructions, see [Stopping an Amazon RDS DB instance temporarily](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_StopInstance.html). When you stop the source database, the initial load and ongoing replication tasks in AWS DMS are automatically stopped. No additional action is required to stop these tasks. | Developer |
## Related resources
**AWS references**
+ [Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-database-to-aurora-postgresql-using-aws-dms-and-aws-sct.html) (AWS Prescriptive Guidance)
+ [Converting Oracle to Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.ToPostgreSQL.html) (AWS SCT documentation)
+ [How AWS DMS Works](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.html) (AWS DMS documentation)
**Other references**
+ [Boolean data type](https://www.postgresqltutorial.com/postgresql-tutorial/postgresql-boolean/) (PostgreSQL documentation)
+ [Oracle built-in data types](https://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF30020) (Oracle documentation)
+ [pgAdmin](https://www.pgadmin.org/) (pgAdmin website)
+ [SQL Developer](https://www.oracle.com/database/technologies/appdev/sql-developer.html) (Oracle website)
**Tutorials and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started With Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [Introduction to AWS DMS](https://www.youtube.com/watch?v=ouia1Sc5QGo) (Video)
+ [Understanding Amazon RDS](https://www.youtube.com/watch?v=eMzCI7S1P9M) (Video)
## Additional information
**Data validation script**
The following data validation script converts **1** to **Y** and **0** to **N**. This helps the AWS DMS task successfully complete and pass the table validation.
```
{
"rule-type": "validation",
"rule-id": "5",
"rule-name": "5",
"rule-target": "column",
"object-locator": {
"schema-name": "ADMIN",
"table-name": "TEMP_CHRA_BOOL",
"column-name": "GRADE"
},
"rule-action": "override-validation-function",
"target-function": "case grade when '1' then 'Y' else 'N' end"
}
```
The `case` statement in the script performs the validation. If validation fails, AWS DMS inserts a record in the **public.awsdms\$1validation\$1failures\$1v1** table on the target database instance. This record includes the table name, error time, and details about the mismatched values in the source and target tables.
If you do not add this data validation script to the AWS DMS task and the data is inserted in the target table, the AWS DMS task shows validation state as **Mismatched Records**.
During AWS SCT conversion, the AWS DMS migration task changes the data type of VARCHAR2(1) data type to Boolean and adds a primary key constraint on the `"NO"` column.
# Create application users and roles in Aurora PostgreSQL-Compatible
*Abhishek Verma, Amazon Web Services*
## Summary
When you migrate to Amazon Aurora PostgreSQL-Compatible Edition, the database users and roles that exist on the source database must be created in the Aurora PostgreSQL-Compatible database. You can create the users and roles in Aurora PostgreSQL-Compatible by using two different approaches:
+ Use similar users and roles in the target as in the source database. In this approach, the data definition languages (DDLs) are extracted for users and roles from the source database. Then they are transformed and applied to the target Aurora PostgreSQL-Compatible database. For example, the blog post [Use SQL to map users, roles, and grants from Oracle to PostgreSQL](https://aws.amazon.com/blogs/database/use-sql-to-map-users-roles-and-grants-from-oracle-to-postgresql) covers using extraction from an Oracle source database engine.
+ Use standardized users and roles that are commonly used during development, administration, and for performing other related operations in the database. This includes read-only, read/write, development, administration, and deployment operations performed by the respective users.
This pattern contains the grants required for users and roles creation in Aurora PostgreSQL-Compatible required for the standardized users and roles approach. The user and role creation steps are aligned to the security policy of granting least privilege to the database users. The following table lists the users, their corresponding roles, and their details on the database.
|
|
| Users | Roles | Purpose |
| --- |--- |--- |
| `APP_read` | `APP_RO` | Used for read-only access on the schema `APP` |
| `APP_WRITE` | `APP_RW` | Used for the write and read operations on the schema `APP` |
| `APP_dev_user` | `APP_DEV` | Used for the development purpose on the schema `APP_DEV`, with read-only access on the schema `APP` |
| `Admin_User` | `rds_superuser` | Used for performing administrator operations on the database |
| `APP` | `APP_DEP` | Used for creating the objects under the `APP` schema and for deployment of objects in the `APP` schema |
## Prerequisites and limitations
**Prerequisites **
+ An active Amazon Web Services (AWS) account
+ A PostgreSQL database, Amazon Aurora PostgreSQL-Compatible Edition database, or Amazon Relational Database Service (Amazon RDS) for PostgreSQL database
**Product versions**
+ All versions of PostgreSQL
## Architecture
**Source technology stack **
+ Any database
**Target technology stack **
+ Amazon Aurora PostgreSQL-Compatible
**Target architecture**
The following diagram shows user roles and the schema architecture in the Aurora PostgreSQL-Compatible database.
![\[User roles and schema architecture for the Aurora PostgreSQL-Comaptible database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/80105a81-e3d1-4258-b3c1-77f3a5e78592/images/b95cb9bc-8bf7-47d1-92e7-66cfb37d7ce7.png)
**Automation and scale**
This pattern contains the users, roles, and schema creation script, which you can run multiple times without any impact to existing users of the source or target database.
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
**Other services**
+ [psql](https://www.postgresql.org/docs/current/app-psql.html) is a terminal-based front-end tool that is installed with every PostgreSQL Database installation. It has a command line interface for running SQL, PL-PGSQL, and operating system commands.
+ [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
## Epics
### Create the users and roles
| Task | Description | Skills required |
| --- | --- | --- |
| Create the deployment user. | The deployment user `APP` will be used to create and modify the database objects during deployments. Use the following scripts to create the deployment user role `APP_DEP` in the schema `APP`. Validate access rights to make sure this user has only the privilege to create objects in the required schema `APP`.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-application-users-and-roles-in-aurora-postgresql-compatible.html) | DBA |
| Create the read-only user. | The read-only user `APP_read` will be used for performing the read-only operation in the schema `APP`. Use the following scripts to create the read-only user. Validate access rights to make sure that this user has privilege only to read the objects in the schema `APP` and for automatically granting read access for any new objects created in the schema `APP`.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-application-users-and-roles-in-aurora-postgresql-compatible.html) | DBA |
| Create the read/write user. | The read/write user `APP_WRITE` will be used to perform read and write operations on the schema `APP`. Use the following scripts to create the read/write user and grant it the `APP_RW` role. Validate access rights to make sure that this user has read and write privileges only on the objects in the schema `APP` and for automatically granting read and write access for any new object created in schema `APP`.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-application-users-and-roles-in-aurora-postgresql-compatible.html) | |
| Create the admin user. | The admin user `Admin_User` will be used to perform admin operations on the database. Examples of these operations are `CREATE ROLE` and `CREATE DATABASE`. `Admin_User` uses the built-in role `rds_superuser` to perform admin operations on the database. Use the following scripts to create and test the privilege for the admin user `Admin_User` in the database.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-application-users-and-roles-in-aurora-postgresql-compatible.html) | DBA |
| Create the development user. | The development user `APP_dev_user` will have rights to create the objects in its local schema `APP_DEV` and read access in the schema `APP`. Use the following scripts to create and test the privileges of the user `APP_dev_user` in the database.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-application-users-and-roles-in-aurora-postgresql-compatible.html) | DBA |
## Related resources
**PostgreSQL documentation**
+ [CREATE ROLE](https://www.postgresql.org/docs/9.1/sql-createrole.html)
+ [CREATE USER](https://www.postgresql.org/docs/8.0/sql-createuser.html)
+ [Predefined Roles](https://www.postgresql.org/docs/14/predefined-roles.html)
## Additional information
**PostgreSQL 14 enhancement**
PostgreSQL 14 provides a set of predefined roles that give access to certain commonly needed, privileged capabilities and information. Administrators (including roles that have the `CREATE ROLE` privilege) can grant these roles or other roles in their environment to users, providing them with access to the specified capabilities and information.
Administrators can grant users access to these roles using the `GRANT` command. For example, to grant the `pg_signal_backend` role to `Admin_User`, you can run the following command.
```
GRANT pg_signal_backend TO Admin_User;
```
The `pg_signal_backend` role is intended to allow administrators to enable trusted, non-superuser roles to send signals to other backends. For more information, see [PostgreSQL 14 enhancement](https://www.postgresql.org/docs/14/predefined-roles.html).
**Fine-tuning access**
In some cases, it might be necessary to provide more granular access to the users (for example, table-based access or column-based access). In such cases, additional roles can be created to grant those privileges to the users. For more information, see [PostgreSQL Grants](https://www.postgresql.org/docs/8.4/sql-grant.html).
# Emulate Oracle DR by using a PostgreSQL-compatible Aurora global database
*HariKrishna Boorgadda, Amazon Web Services*
## Summary
Best practices for Enterprise disaster recovery (DR) basically consist of designing and implementing fault-tolerant hardware and software systems that can survive a disaster (*business continuance*) and resume normal operations (*business resumption*), with minimal intervention and, ideally, with no data loss. Building fault-tolerant environments to satisfy Enterprise DR objectives can be expensive and time consuming and requires a strong commitment by the business.
Oracle Database provides three different approaches to DR that provide the highest level of data protection and availability compared to any other approach for protecting Oracle data.
+ Oracle Zero Data Loss Recovery Appliance
+ Oracle Active Data Guard
+ Oracle GoldenGate
This pattern provides a way to emulate the Oracle GoldenGate DR by using an Amazon Aurora global database. The reference architecture uses Oracle GoldenGate for DR across three AWS Regions. The pattern walks through the replatforming of the source architecture to the cloud-native Aurora global database based on Amazon Aurora PostgreSQL–Compatible Edition.
Aurora global databases are designed for applications with a global footprint. A single Aurora database spans multiple AWS Regions with as many as five secondary Regions. Aurora global databases provide the following features:
+ Physical storage-level replication
+ Low-latency global reads
+ Fast disaster recovery from Region-wide outages
+ Fast cross-Region migrations
+ Low replication lag across Regions
+ Little-to-no performance impact on your database
For more information about Aurora global database features and advantages, see [Using Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html#aurora-global-database-overview). For more information about unplanned and managed failovers, see [Using failover in an Amazon Aurora global database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html#aurora-global-database-failover).
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A Java Database Connectivity (JDBC) PostgreSQL driver for application connectivity
+ An Aurora global database based on Amazon Aurora PostgreSQL-Compatible Edition
+ An Oracle Real Application Clusters (RAC) database migrated to the Aurora global database based on Aurora PostgreSQL–Compatible
**Limitations of Aurora global databases **
+ Aurora global databases aren’t available in all AWS Regions. For a list of supported Regions, see [Aurora global databases with Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.Aurora_Fea_Regions_DB-eng.Feature.GlobalDatabase.html#Concepts.Aurora_Fea_Regions_DB-eng.Feature.GlobalDatabase.apg).
+ For information about features that aren’t supported and other limitations of Aurora global databases, see the [Limitations of Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html#aurora-global-database.limitations).
**Product versions**
+ Amazon Aurora PostgreSQL–Compatible Edition version 10.14 or later
## Architecture
**Source technology stack**** **
+ Oracle RAC four-node database
+ Oracle GoldenGate
**Source architecture**** **
The following diagram shows three clusters with four-node Oracle RAC in different AWS Regions replicated using Oracle GoldenGate.
![\[Oracle RAC in a primary Region and two secondary Regions.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/11d4265b-31af-4ebf-a766-24196193ee01/images/9fc740fc-d339-422e-beaf-1f65690c9d14.png)
**Target technology stack **
+ A three-cluster Amazon Aurora global database based on Aurora PostgreSQL–Compatible, with one cluster in the primary Region, two clusters in different secondary Regions
**Target architecture**
![\[Amazon Aurora in a primary Region and two secondary Regions.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/11d4265b-31af-4ebf-a766-24196193ee01/images/8e3deca9-03f2-437c-9341-795ac17e2b42.png)
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) span multiple AWS Regions, providing low latency global reads and fast recovery from the rare outage that might affect an entire AWS Region.
## Epics
### Add Regions with reader DB instances
| Task | Description | Skills required |
| --- | --- | --- |
| Attach one or more secondary Aurora clusters. | On the AWS Management Console, choose Amazon Aurora. Select the primary cluster, choose **Actions**, and choose **Add region** from the dropdown list. | DBA |
| Select the instance class. | You can change the instance class of the secondary cluster. However, we recommend keeping it the same as the primary cluster instance class. | DBA |
| Add the third Region. | Repeat the steps in this epic to add a cluster in the third Region. | DBA |
### Fail over the Aurora global database
| Task | Description | Skills required |
| --- | --- | --- |
| Remove the primary cluster from the Aurora global database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/emulate-oracle-dr-by-using-a-postgresql-compatible-aurora-global-database.html) | DBA |
| Reconfigure your application to divert write traffic to the newly promoted cluster. | Modify the endpoint in the application with that of the newly promoted cluster. | DBA |
| Stop issuing any write operations to the unavailable cluster. | Stop the application and any data manipulation language (DML) activity to the cluster that you removed. | DBA |
| Create a new Aurora global database. | Now you can create an Aurora global database with the newly promoted cluster as the primary cluster. | DBA |
### Start the primary cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Select the primary cluster to be started from the global database. | On the Amazon Aurora console, in the Global Database setup, choose the primary cluster. | DBA |
| Start the cluster. | On the **Actions** dropdown list, choose **Start**. This process might take some time. Refresh the screen to see the status, or check the **Status** column for the current state of the cluster after the operation is completed. | DBA |
### Clean up the resources
| Task | Description | Skills required |
| --- | --- | --- |
| Delete the remaining secondary clusters. | After the failover pilot is completed, remove the secondary clusters from the global database. | DBA |
| Delete the primary cluster. | Remove the cluster. | DBA |
## Related resources
+ [Using Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html#aurora-global-database-detaching)
+ [Aurora PostgreSQL Disaster Recovery solutions using Amazon Aurora Global Database](https://aws.amazon.com/blogs/database/aurora-postgresql-disaster-recovery-solutions-using-amazon-aurora-global-database/) (blog post)
# Implement SHA1 hashing for PII data when migrating from SQL Server to PostgreSQL
*Rajkumar Raghuwanshi and Jagadish Kantubugata, Amazon Web Services*
## Summary
This pattern describes how to implement Secure Hash Algorithm 1 (SHA1) hashing for email addresses when migrating from SQL Server to either Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL-Compatible. An email address is an example of *personally identifiable information* (PII). PII is information that, when viewed directly or paired with other related data, can be used to reasonably infer the identity of an individual.
This pattern covers the challenges of maintaining consistent hash values across different database collations and character encodings, and provides a solution using PostgreSQL functions and triggers. Although this pattern focuses on SHA1 hashing, it can be adapted for other hashing algorithms supported by PostgreSQL's `pgcrypto` module. Always consider the security implications of your hashing strategy and consult with security experts if handling sensitive data.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Source SQL Server database
+ Target PostgreSQL database (Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible)
+ PL/pgSQL coding expertise
**Limitations**
+ This pattern requires database-level collation changes based on use cases.
+ The performance impact on large datasets has not been evaluated.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
**Product versions**
+ Microsoft SQL Server 2012 or later
## Architecture
**Source technology stack **
+ SQL Server
+ .NET Framework
**Target technology stack **
+ PostgreSQL
+ `pgcrypto` extension
**Automation and scale**
+ Consider implementing the hashing function as a stored procedure for easier maintenance.
+ For large datasets, evaluate performance and consider batch processing or indexing strategies.
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Relational Database Service Amazon RDS for PostgreSQL ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html)helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
**Other tools**
+ [pgAdmin](https://www.pgadmin.org/) is an open source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
+ [SQL Server Management Studio (SSMS)](https://learn.microsoft.com/en-us/ssms/sql-server-management-studio-ssms) is an integrated environment for managing any SQL infrastructure.
## Best practices
+ Use appropriate collation settings for handling special characters on the target database side.
+ Test thoroughly with a variety of email addresses, including addresses with non-ASCII characters.
+ Maintain consistency in uppercase and lowercase handling between the application and database layers.
+ Benchmark performance of queries using the hashed values.
## Epics
### Analyze source hashing implementation
| Task | Description | Skills required |
| --- | --- | --- |
| Review SQL Server code. | To review SQL Server code that generates SHA1 hashes, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | Data engineer, DBA, App developer |
| Document the hashing algorithm and data transformations. | To document the exact hashing algorithm and data transformations, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, Data engineer, DBA |
### Create PostgreSQL hashing function
| Task | Description | Skills required |
| --- | --- | --- |
| Create `pgcrypto` extension. | To create the `pgcrypto` extension, use `pgAdmin/psql` to run the following command:
CREATE EXTENSION pgcrypto;
| DBA, Data engineer |
| Implement a PostgreSQL function. | Implement the following PostgreSQL function to replicate the SQL Server hashing logic. At a high level, this function uses the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html)
CREATE OR REPLACE FUNCTION utility.hex_to_bigint ( par_val character varying, par_upper character varying DEFAULT 'lower'::character varying) RETURNS bigint LANGUAGE 'plpgsql' AS $BODY$ DECLARE retnumber bigint; digest_bytes bytea; BEGIN if lower(par_upper) = 'upper' then digest_bytes := digest(upper(par_val), 'sha1'); else digest_bytes := digest((par_val), 'sha1'); end if; retnumber := ('x' || encode(substring(digest_bytes, length(digest_bytes)-10+1), 'hex'))::bit(64)::bigint; RETURN retnumber; END; $BODY$;
| Data engineer, DBA, App developer |
| Test the function. | To test the function, use sample data from SQL Server to verify matching hash values. Run the following command:
select 'alejandro_rosalez@example.com' as Email, utility.hex_to_bigint('alejandro_rosalez@example.com','upper') as HashValue;
| App developer, DBA, Data engineer |
### Implement triggers for automatic hashing
| Task | Description | Skills required |
| --- | --- | --- |
| Create triggers on relevant tables. | To create triggers on relevant tables to automatically generate hash values on insert or update, run the following command:
CREATE OR REPLACE FUNCTION update_email_hash() RETURNS TRIGGER AS $$ BEGIN NEW.email_hash = utility.hex_to_bigint(NEW.email, 'upper'); RETURN NEW; END; $$ LANGUAGE plpgsql;
CREATE TRIGGER email_hash_trigger BEFORE INSERT OR UPDATE ON users FOR EACH ROW EXECUTE FUNCTION update_email_hash();
| App developer, Data engineer, DBA |
### Migrate existing data
| Task | Description | Skills required |
| --- | --- | --- |
| Develop a migration script or use AWS DMS. | Develop a migration script or use AWS DMS to populate hash values for existing data (including hash values stored as `BIGINT` in the source system.) Complete the following tasks:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | Data engineer, App developer, DBA |
| Use the new PostgreSQL hashing function. | To use the new PostgreSQL hashing function to ensure consistency, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, DBA, DevOps engineer |
### Update application queries
| Task | Description | Skills required |
| --- | --- | --- |
| Identify application queries. | To identify application queries that use hashed values, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, DBA, Data engineer |
| Modify queries. | If necessary, modify queries to use the new PostgreSQL hashing function. Do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, DBA, Data engineer |
### Test and validate
| Task | Description | Skills required |
| --- | --- | --- |
| Perform testing. | To perform thorough testing with a subset of production data, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, Data engineer, DBA |
| Validate that hash values match. | To validate that hash values match between SQL Server and PostgreSQL, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, Data engineer, DBA |
| Verify application functionality. | To verify application functionality by using the migrated data and the new hashing implementation, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.html) | App developer, DBA, Data engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Hash values don’t match. | Verify character encodings and collations between source and target. For more information, see [Manage collation changes in PostgreSQL on Amazon Aurora and Amazon RDS](https://aws.amazon.com/blogs/database/manage-collation-changes-in-postgresql-on-amazon-aurora-and-amazon-rds/) (AWS Blog). |
## Related resources
**AWS Blogs**
+ [Manage collation changes in PostgreSQL on Amazon Aurora and Amazon RDS](https://aws.amazon.com/blogs/database/manage-collation-changes-in-postgresql-on-amazon-aurora-and-amazon-rds/)
+ [Migrate SQL Server to Amazon Aurora PostgreSQL using best practices and lessons learned from the field](https://aws.amazon.com/blogs/database/migrate-sql-server-to-amazon-aurora-postgresql-using-best-practices-and-lessons-learned-from-the-field/)
**Other resources**
+ [PostgreSQL pgcrypto module](https://www.postgresql.org/docs/current/pgcrypto.html) (PostgreSQL documentation)
+ [PostgreSQL trigger functions](https://www.postgresql.org/docs/current/plpgsql-trigger.html) (PostgreSQL documentation)
+ [SQL Server HASHBYTES function](https://docs.microsoft.com/en-us/sql/t-sql/functions/hashbytes-transact-sql) (Microsoft documentation)
# Incrementally migrate from Amazon RDS for Oracle to Amazon RDS for PostgreSQL using Oracle SQL Developer and AWS SCT
*Pinesh Singal, Amazon Web Services*
## Summary
Many migration strategies and approaches run in multiple phases that can last from a few weeks to several months. During this time, you can experience delays because of patching or upgrades in the source Oracle DB instances that you want to migrate to PostgreSQL DB instances. To avoid this situation, we recommend that you incrementally migrate the remaining Oracle database code to PostgreSQL database code.
This pattern provides an incremental migration strategy with no downtime for a multi-terabyte Oracle DB instance that has a high number of transactions performed after your initial migration and that must be migrated to a PostgreSQL database. You can use this pattern’s step-by-step approach to incrementally migrate an Amazon Relational Database Service (Amazon RDS) for Oracle DB instance to an Amazon RDS for PostgreSQL DB instance without signing in to the Amazon Web Services (AWS) Management Console.
The pattern uses [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) to find the differences between two schemas in the source Oracle database. You then use AWS Schema Conversion Tool (AWS SCT) to convert the Amazon RDS for Oracle database schema objects to Amazon RDS for PostgreSQL database schema objects. You can then run a Python script in the Windows Command Prompt to create AWS SCT objects for the incremental changes to the source database objects.
**Note**
Before you migrate your production workloads, we recommend that you run a proof of concept (PoC) for this pattern's approach in a testing or non-production environment.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ An existing Amazon RDS for Oracle DB instance.
+ An existing Amazon RDS for PostgreSQL DB instance.
+ AWS SCT, installed and configured with JDBC drivers for Oracle and PostgreSQL database engines. For more information about this, see [Installing AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html#CHAP_Installing.Procedure) and [Installing the required database drivers](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html#CHAP_Installing.JDBCDrivers) in the AWS SCT documentation.
+ Oracle SQL Developer, installed and configured. For more information about this, see the [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) documentation.
+ The `incremental-migration-sct-sql.zip` file (attached), downloaded to your local computer.
**Limitations **
+ The minimum requirements for your source Amazon RDS for Oracle DB instance are:
+ Oracle versions 10.2 and later (for versions 10.x), 11g (versions 11.2.0.3.v1 and later) and up to 12.2, and 18c for the Enterprise, Standard, Standard One, and Standard Two editions
+ The minimum requirements for your target Amazon RDS for PostgreSQL DB instance are:
+ PostgreSQL versions 9.4 and later (for versions 9.x), 10.x, and 11.x
+ This pattern uses Oracle SQL Developer. Your results might vary if you use other tools to find and export schema differences.
+ The [SQL scripts](https://docs.oracle.com/database/121/AEUTL/sql_rep.htm#AEUTL191) generated by Oracle SQL Developer can raise transformation errors, which means that you need to perform a manual migration.
+ If the AWS SCT source and target test connections fail, make sure that you configure the JDBC driver versions and inbound rules for the virtual private cloud (VPC) security group to accept incoming traffic.
**Product versions**
+ Amazon RDS for Oracle DB instance version 12.1.0.2 (version 10.2 and later)
+ Amazon RDS for PostgreSQL DB instance version 11.5 (version 9.4 and later)
+ Oracle SQL Developer version 19.1 and later
+ AWS SCT version 1.0.632 and later
## Architecture
**Source technology stack **
+ Amazon RDS for Oracle DB instance
**Target technology stack **
+ Amazon RDS for PostgreSQL DB instance
**Source and target architecture**
The following diagram shows the migration of an Amazon RDS for Oracle DB instance to an Amazon RDS for PostgreSQL DB instance.
![\[Migration workflow from Amazon RDS for Oracle to Amazon RDS for PostgreSQL.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/c7eed517-e496-4e8e-a520-c1e43397419e/images/bfbbed5e-db13-4a22-99aa-1a17f00f5faf.png)
The diagram shows the following migration workflow:
1. Open Oracle SQL Developer and connect to the source and target databases.
1. Generate a [diff report ](https://docs.oracle.com/cd/E93130_01/rules_palette/Content/Diff%20Reports/Detailed_Diff_Reports.htm)and then generate the SQL scripts file for the schema difference objects. For more information about diff reports, see [Detailed diff reports](https://docs.oracle.com/cd/E93130_01/rules_palette/Content/Diff%20Reports/Detailed_Diff_Reports.htm) in the Oracle documentation.
1. Configure AWS SCT and run the Python code.
1. The SQL scripts file converts from Oracle to PostgreSQL.
1. Run the SQL scripts file on the target PostgreSQL DB instance.
**Automation and scale**
You can automate this migration by adding additional parameters and security-related changes for multiple functionalities in a single program to your Python script.
## Tools
+ [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) – AWS Schema Conversion Tool (AWS SCT) converts your existing database schema from one database engine to another.
+ [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) – Oracle SQL Developer is an integrated development environment (IDE) that simplifies the development and management of Oracle databases in both traditional and cloud-based deployments.
**Code **
The `incremental-migration-sct-sql.zip` file (attached) contains the complete source code for this pattern.
## Epics
### Create the SQL scripts file for the source database schema differences
| Task | Description | Skills required |
| --- | --- | --- |
| Run Database Diff in Oracle SQL Developer. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.html) | DBA |
| Generate the SQL scripts file. | Choose **Generate Script** to generate the differences in the SQL files. This generates the SQL scripts file that AWS SCT uses to convert your database from Oracle to PostgreSQL. | DBA |
### Use the Python script to create the target DB objects in AWS SCT
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS SCT with the Windows Command Prompt. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.html)
4. Modify the AWS SCT configuration parameters according to your requirements and then copy the SQL scripts file into your working directory in the `input` subdirectory. | DBA |
| Run the Python script. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.html) | DBA |
| Create the objects in Amazon RDS for PostgreSQL | Run the SQL files and create objects in your Amazon RDS for PostgreSQL DB instance. | DBA |
## Related resources
+ [Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html)
+ [PostgreSQL on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html)
+ [Using the AWS SCT user interface](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html)
+ [Using Oracle as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/c7eed517-e496-4e8e-a520-c1e43397419e/attachments/attachment.zip)
# Load BLOB files into TEXT by using file encoding in Aurora PostgreSQL-Compatible
*Bhanu Ganesh Gudivada and Jeevan Shetty, Amazon Web Services*
## Summary
Often during migration, there are cases where you have to process unstructured and structured data that is loaded from files on a local file system. The data might also be in a character set that differs from the database character set.
These files hold the following types of data:
+ **Metadata** – This data describes the file structure.
+ **Semi-structured data** – These are textual strings in a specific format, such as JSON or XML. You might be able to make assertions about such data, such as "will always start with '<' " or "does not contain any newline characters."
+ **Full text** – This data usually contains all types of characters, including newline and quote characters. It might also consist of multibyte characters in UTF-8.
+ **Binary data** – This data might contain bytes or combinations of bytes including, nulls and end-of-file markers.
Loading a mixture of these types of data can be a challenge.
The pattern can be used with on-premises Oracle databases , Oracle databases that are on Amazon Elastic Compute Cloud (Amazon EC2) instances on the Amazon Web Services (AWS) Cloud, and Amazon Relational Database Service (Amazon RDS) for Oracle databases. As an example, this pattern is using Amazon Aurora PostgreSQL-Compatible Edition.
In Oracle Database, with the help of a `BFILE` (binary file) pointer, the `DBMS_LOB` package, and Oracle system functions, you can load from file and convert to CLOB with character encoding. Because PostgreSQL does not support the BLOB data type when migrating to an Amazon Aurora PostgreSQL-Compatible Edition database, these functions must be converted to PostgreSQL-compatible scripts.
This pattern provides two approaches for loading a file into a single database column in an Amazon Aurora PostgreSQL-Compatible database:
+ Approach 1 – You import data from your Amazon Simple Storage Service (Amazon S3) bucket by using the `table_import_from_s3` function of the `aws_s3` extension with the encode option.
+ Approach 2 – You encode to hexadecimal outside of the database, and then you decode to view `TEXT` inside the database.
We recommend using Approach 1 because Aurora PostgreSQL-Compatible has direct integration with the `aws_s3` extension.
This pattern uses the example of loading a flat file that contains an email template, which has multibyte characters and distinct formatting, into an Amazon Aurora PostgreSQL-Compatible database.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ An Amazon RDS instance or an Aurora PostgreSQL-Compatible instance
+ A basic understanding of SQL and Relational Database Management System (RDBMS)
+ An Amazon Simple Storage Service (Amazon S3) bucket.
+ Knowledge of system functions in Oracle and PostgreSQL
+ RPM Package HexDump-XXD-0.1.1 (included with Amazon Linux 2)
**Note**
Amazon Linux 2 is nearing end of support. For more information, see the [Amazon Linux 2 FAQs](http://aws.amazon.com/amazon-linux-2/faqs/).
**Limitations **
+ For the `TEXT` data type, the longest possible character string that can be stored is about 1 GB.
**Product versions**
+ Aurora supports the PostgreSQL versions listed in [Amazon Aurora PostgreSQL updates](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQLReleaseNotes/AuroraPostgreSQL.Updates.html).
## Architecture
**Target technology stack **
+ Aurora PostgreSQL-Compatible
**Target architecture**
*Approach 1 – Using aws\$1s3.table\$1import\$1from\$1s3 *
From an on-premises server, a file containing an email template with multibyte characters and custom formatting is transferred to Amazon S3. The custom database function provided by this pattern uses the `aws_s3.table_import_from_s3` function with `file_encoding` to load files into the database and return query results as the `TEXT` data type.
![\[Four-step process from the on-premises server to the TEXT output from the Aurora database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/cbf63cac-dcea-4e18-ab4f-c4f6296f60e7/images/9c46b385-e8a0-4e50-b856-d522c44d79e3.png)
1. Files are transferred to the staging S3 bucket.
1. Files are uploaded to the Amazon Aurora PostgreSQL-Compatible database.
1. Using the pgAdmin client, the custom function `load_file_into_clob` is deployed to the Aurora database.
1. The custom function internally uses `table_import_from_s3` with file\$1encoding. The output from the function is obtained by using `array_to_string` and `array_agg` as `TEXT` output.
*Approach 2 – Encoding to hexadecimal outside of the database and decoding to view TEXT inside the database*
A file from an on-premises server or a local file system is converted into a hex dump. Then the file is imported into PostgreSQL as a `TEXT` field.
![\[Three-step process using Hex dump.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/cbf63cac-dcea-4e18-ab4f-c4f6296f60e7/images/563038ca-f890-4874-85df-d0f82d99800a.png)
1. Convert the file to a hex dump in the command line by using the `xxd -p` option.
1. Upload the hex dump files into Aurora PostgreSQL-Compatible by using the `\copy` option, and then decode the hex dump files to binary.
1. Encode the binary data to return as `TEXT`.
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
**Other tools**
+ [pgAdmin4](https://www.pgadmin.org/) is an open source administration and development platform for PostgreSQL. pgAdmin4 can be used on Linux, Unix, mac OS, and Windows to manage PostgreSQL.
## Epics
### Approach 1: Import data from Amazon S3 to Aurora PostgreSQL-Compatible
| Task | Description | Skills required |
| --- | --- | --- |
| Launch an EC2 instance. | For instructions on launching an instance, see [Launch your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html). | DBA |
| Install the PostgreSQL client pgAdmin tool. | Download and install [pgAdmin](https://www.pgadmin.org/download/). | DBA |
| Create an IAM policy. | Create an AWS Identity and Access Management (IAM) policy named `aurora-s3-access-pol` that grants access to the S3 bucket where the files will be stored. Use the following code, replacing `` with the name of your S3 bucket.
| DBA |
| Create an IAM role for object import from Amazon S3 to Aurora PostgreSQL-Compatible. | Use the following code to create an IAM role named `aurora-s3-import-role` with the [AssumeRole](https://docs.amazonaws.cn/en_us/STS/latest/APIReference/API_AssumeRole.html) trust relationship. `AssumeRole` allows Aurora to access other AWS services on your behalf.
| DBA |
| Associate the IAM role to the cluster. | To associate the IAM role with the Aurora PostgreSQL-Compatible database cluster, run the following AWS CLI command. Change `` to the ID of the AWS account that hosts the Aurora PostgreSQL-Compatible database. This enables the Aurora PostgreSQL-Compatible database to access the S3 bucket.
| DBA |
| Upload the example to Amazon S3. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.html) | DBA, App owner |
| Deploy the custom function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.html) | App owner, DBA |
| Run the custom function for importing the data into the database. | Run the following SQL command, replacing the items in angle brackets with the appropriate values.
The command loads the file from Amazon S3 and returns the output as `TEXT`. | App owner, DBA |
### Approach 2: Convert the template file into a hex dump in a local Linux system
| Task | Description | Skills required |
| --- | --- | --- |
| Convert the template file into a hex dump. | The Hexdump utility displays the contents of binary files in hexadecimal, decimal, octal, or ASCII. The `hexdump` command is part of the `util-linux` package and comes pre-installed in Linux distributions. The Hexdump RPM package is part of Amazon Linux 2 as well. (: Amazon Linux 2 is nearing end of support. For more information, see the [Amazon Linux 2 FAQs](http://aws.amazon.com/amazon-linux-2/faqs/).)To convert the file contents into a hex dump, run the following shell command.
xxd -p | tr -d '\n' >
Replace the path and file with the appropriate values, as shown in the following example.
| DBA |
| Load the hexdump file into the database schema. | Use the following commands to load the hexdump file into the Aurora PostgreSQL-Compatible database.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.html) | DBA |
## Related resources
**References**
+ [Using a PostgreSQL database as a target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
+ [Oracle Database 19c to Amazon Aurora with PostgreSQL Compatibility (12.4) Migration Playbook](https://d1.awsstatic.com/whitepapers/Migration/oracle-database-amazon-aurora-postgresql-migration-playbook-12.4.pdf)
+ [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html)
+ [Associating an IAM role with an Amazon Aurora MySQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.Authorizing.IAM.AddRoleToDBCluster.html)
+ [pgAdmin](https://www.pgadmin.org/)
**Tutorials**
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [Migrate from Oracle to Amazon Aurora](https://aws.amazon.com/getting-started/projects/migrate-oracle-to-amazon-aurora/)
## Additional information
**load\$1file\$1into\$1clob custom function**
```
CREATE OR REPLACE FUNCTION load_file_into_clob(
s3_bucket_name text,
s3_bucket_region text,
file_name text,
file_delimiter character DEFAULT '&'::bpchar,
file_encoding text DEFAULT 'UTF8'::text)
RETURNS text
LANGUAGE 'plpgsql'
COST 100
VOLATILE PARALLEL UNSAFE
AS $BODY$
DECLARE
blob_data BYTEA;
clob_data TEXT;
l_table_name CHARACTER VARYING(50) := 'file_upload_hex';
l_column_name CHARACTER VARYING(50) := 'template';
l_return_text TEXT;
l_option_text CHARACTER VARYING(150);
l_sql_stmt CHARACTER VARYING(500);
BEGIN
EXECUTE format ('CREATE TEMPORARY TABLE %I (%I text, id_serial serial)', l_table_name, l_column_name);
l_sql_stmt := 'select ''(format text, delimiter ''''' || file_delimiter || ''''', encoding ''''' || file_encoding || ''''')'' ';
EXECUTE FORMAT(l_sql_stmt)
INTO l_option_text;
EXECUTE FORMAT('SELECT aws_s3.table_import_from_s3($1,$2,$6, aws_commons.create_s3_uri($3,$4,$5))')
INTO l_return_text
USING l_table_name, l_column_name, s3_bucket_name, file_name,s3_bucket_region,l_option_text;
EXECUTE format('select array_to_string(array_agg(%I order by id_serial),E''\n'') from %I', l_column_name, l_table_name)
INTO clob_data;
drop table file_upload_hex;
RETURN clob_data;
END;
$BODY$;
```
**Email template**
```
######################################################################################
## ##
## johndoe Template Type: email ##
## File: johndoe.salary.event.notification.email.vm ##
## Author: Aimée Étienne Date 1/10/2021 ##
## Purpose: Email template used by EmplmanagerEJB to inform a johndoe they ##
## have been given access to a salary event ##
## Template Attributes: ##
## invitedUser - PersonDetails object for the invited user ##
## salaryEvent - OfferDetails object for the event the user was given access ##
## buyercollege - CompDetails object for the college owning the salary event ##
## salaryCoordinator - PersonDetails of the salary coordinator for the event ##
## idp - Identity Provider of the email recipient ##
## httpWebRoot - HTTP address of the server ##
## ##
######################################################################################
$!invitedUser.firstname $!invitedUser.lastname,
Ce courriel confirme que vous avez ete invite par $!salaryCoordinator.firstname $!salaryCoordinator.lastname de $buyercollege.collegeName a participer a l'evenement "$salaryEvent.offeringtitle" sur johndoeMaster Sourcing Intelligence.
Votre nom d'utilisateur est $!invitedUser.username
Veuillez suivre le lien ci-dessous pour acceder a l'evenement.
${httpWebRoot}/myDashboard.do?idp=$!{idp}
Si vous avez oublie votre mot de passe, utilisez le lien "Mot de passe oublie" situe sur l'ecran de connexion et entrez votre nom d'utilisateur ci-dessus.
Si vous avez des questions ou des preoccupations, nous vous invitons a communiquer avec le coordonnateur de l'evenement $!salaryCoordinator.firstname $!salaryCoordinator.lastname au ${salaryCoordinator.workphone}.
*******
johndoeMaster Sourcing Intelligence est une plateforme de soumission en ligne pour les equipements, les materiaux et les services.
Si vous avez des difficultes ou des questions, envoyez un courriel a support@johndoeMaster.com pour obtenir de l'aide.
```
# Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL with AWS SCT and AWS DMS using AWS CLI and CloudFormation
*Pinesh Singal, Amazon Web Services*
## Summary
This pattern shows how to migrate a multi-terabyte [Amazon Relational Database Service (Amazon RDS) for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) DB instance to an [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) DB instance by using the AWS Command Line Interface (AWS CLI). The approach provides minimal downtime and doesn’t require signing in to the AWS Management Console.
This pattern helps avoid manual configurations and individual migrations by using the AWS Schema Conversion Tool (AWS SCT) and AWS Database Migration Service (AWS DMS) consoles. The solution sets up a one-time configuration for multiple databases and performs the migrations by using AWS SCT and AWS DMS in the AWS CLI.
The pattern uses AWS SCT to convert database schema objects from Amazon RDS for Oracle to Amazon RDS for PostgreSQL, and then uses AWS DMS to migrate the data. Using Python scripts in AWS CLI, you create AWS SCT objects and AWS DMS tasks with an CloudFormation template.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ An existing Amazon RDS for Oracle DB instance.
+ An existing Amazon RDS for PostgreSQL DB instance.
+ An Amazon Elastic Compute Cloud (Amazon EC2) instance or local machine with Windows or Linux OS for running scripts.
+ An understanding of the following AWS DMS migration task types: `full-load`, `cdc`, `full-load-and-cdc`. For more information, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html) in the AWS DMS documentation.
+ AWS SCT, installed and configured with Java Database Connectivity (JDBC) drivers for Oracle and PostgreSQL database engines. For more information, see [Installing and configuring AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html#CHAP_Installing.Procedure) in the AWS SCT documentation.
+ The `AWSSchemaConversionToolBatch.jar` file from the installed AWS SCT folder, copied to your working directory.
+ The `cli-sct-dms-cft.zip` file (attached), downloaded and extracted in your working directory.
+ The most recent AWS DMS replication instance engine version. For more information, see [How do I create an AWS DMS replication instance](https://aws.amazon.com/premiumsupport/knowledge-center/create-aws-dms-replication-instance/) in the AWS Support documentation and [AWS DMS release notes](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html).
+ AWS CLI version 2, installed and configured with your access key ID, secret access key, and default AWS Region name for the EC2 instance or OS where the scripts are run. For more information, see [Installing or updating to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [Configuring settings for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) in the AWS CLI documentation.
+ Familiarity with CloudFormation templates. For more information, see [How CloudFormation works](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-overview.html) in the CloudFormation documentation.
+ Python version 3, installed and configured on the EC2 instance or OS where the scripts are run. For more information, see the [Python documentation](https://docs.python.org/3/).
**Limitations **
+ The minimum requirements for your source Amazon RDS for Oracle DB instance are:
+ Oracle versions 12c (12.1.0.2, 12.2.0.1), 18c (18.0.0.0), and 19c (19.0.0.0) for the Enterprise, Standard, Standard One, and Standard Two editions.
+ Although Amazon RDS supports Oracle 18c (18.0.0.0), this version is on a deprecation path because Oracle no longer provides patches for 18c after the end-of-support date. For more information, see [Amazon RDS for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html#Oracle.Concepts.Deprecate.11204) in the Amazon RDS documentation.
+ Amazon RDS for Oracle 11g is no longer supported.
+ The minimum requirements for your target Amazon RDS for PostgreSQL DB instance are:
+ PostgreSQL versions 9 (9.5 and 9.6), 10.x, 11.x, 12.x, and 13.x
**Product versions**
+ Amazon RDS for Oracle DB instance version 12.1.0.2 and later
+ Amazon RDS for PostgreSQL DB instance version 11.5 and later
+ AWS CLI version 2
+ The latest version of AWS SCT
+ The latest version of Python 3
## Architecture
**Source technology stack **
+ Amazon RDS for Oracle
**Target technology stack **
+ Amazon RDS for PostgreSQL
**Source and target architecture **
The following diagram shows the migration of an Amazon RDS for Oracle DB instance to an Amazon RDS for PostgreSQL DB instance using AWS DMS and Python scripts.
![\[Migrating RDS for Oracle DB instance to RDS for PostgreSQL DB instance using AWS DMS and Python.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5e041494-2e64-4f09-b6ec-0e0cba3a4972/images/77022e13-46fb-4aa8-ab49-85b0ca4c317a.png)
The diagram shows the following migration workflow:
1. The Python script uses AWS SCT to connect to the source and target DB instances.
1. The user starts AWS SCT with the Python script, converts the Oracle code to PostgreSQL code, and runs it on the target DB instance.
1. The Python script creates AWS DMS replication tasks for the source and target DB instances.
1. The user deploys Python scripts to start the AWS DMS tasks and then stops the tasks after the data migration is complete.
**Automation and scale**
You can automate this migration by adding parameters and security-related changes to your Python script, to provide additional functionality.
## Tools
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command line shell.
+ [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions. This pattern converts the `.csv` input file to a `.json` input file by using a Python script. The `.json` file is used in AWS CLI commands to create an CloudFormation stack that creates multiple AWS DMS replication tasks with Amazon Resource Names (ARNs), migration types, task settings, and table mappings.
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores to the AWS Cloud or between combinations of cloud and on-premises setups. This pattern uses AWS DMS to create, start, and stop tasks with a Python script that runs on the command line, and to create the CloudFormation template.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database. This patterns requires the `AWSSchemaConversionToolBatch.jar` file from the installed AWS SCT directory.
**Code**
The `cli-sct-dms-cft.zip` file (attached) contains the complete source code for this pattern.
## Epics
### Configure AWS SCT and create database objects in the AWS CLI
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS SCT to run from the AWS CLI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.html) | DBA |
| Run the `run_aws_sct.py` Python script. | Run the `run_aws_sct.py` Python script by using the following command:`$ python run_aws_sct.py database_migration.txt`The Python script converts the database objects from Oracle to PostgreSQL and creates SQL files in PostgreSQL format. The script also creates the PDF file `Database migration assessment report`, which provides you with detailed recommendations and conversion statistics for database objects. | DBA |
| Create objects in Amazon RDS for PostgreSQL. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.html) | DBA |
### Configure and create AWS DMS tasks by using the AWS CLI and CloudFormation
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS DMS replication instance. | Sign in to the AWS Management Console, open the [AWS DMS console](https://console.aws.amazon.com/dms/v2/), and create a replication instance that is configured according to your requirements.For more information, see [Creating a replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Creating.html) in the AWS DMS documentation and [How do I create an AWS DMS replication instance](https://aws.amazon.com/premiumsupport/knowledge-center/create-aws-dms-replication-instance/) in the AWS Support documentation. | DBA |
| Create the source endpoint. | On the AWS DMS console, choose **Endpoints** and then create a source endpoint for the Oracle database according to your requirements. The extra connection attribute must be `numberDataTypeScale` with a `-2` value.For more information, see [Creating source and target endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html) in the AWS DMS documentation. | DBA |
| Create the target endpoint. | On the AWS DMS console, choose **Endpoints** and then create a target endpoint for the PostgreSQL database according to your requirements. For more information, see [Creating source and target endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html) in the AWS DMS documentation. | DevOps engineer |
| Configure the AWS DMS replication details to run from the AWS CLI. | Configure the AWS DMS source and target endpoints and replication details in the `dms-arn-list.txt` file with the source endpoint ARN, target endpoint ARN, and the replication instance ARN by using the following format:
| DBA |
| Run the `dms-create-task.py` Python script to create the AWS DMS tasks. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.html) | DBA |
| Verify that AWS DMS tasks are ready. | On the AWS DMS console, check that your AWS DMS tasks are in `Ready` status in the **Status **section. | DBA |
### Start and stop the AWS DMS tasks by using the AWS CLI
| Task | Description | Skills required |
| --- | --- | --- |
| Start the AWS DMS tasks. | Run the `dms-start-task.py` Python script by using the following command:
$ python dms-start-task.py start ''
The start date and time must be in the `'DD-MON-YYYY'` or `'YYYY-MM-DDTHH:MI:SS'` format (for example, `'01-Dec-2019'` or `'2018-03-08T12:12:12'`).You can review the AWS DMS task status in the **Table statistics** tab on the **Tasks **page of the AWS DMS console. | DBA |
| Validate the data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.html)For more information, see [AWS DMS data validation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html) in the AWS DMS documentation. | DBA |
| Stop the AWS DMS tasks. | Run the Python script by using the following command:
$ python dms-start-task.py stop
AWS DMS tasks might stop with a `failed`status, depending on the validation status. For more information, see the next section. | DBA |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| AWS SCT source and target test connections fail. | Configure the JDBC driver versions and VPC security group inbound rules to accept the incoming traffic. |
| Source or target endpoint test run fails. | Check if the endpoint settings and replication instance are in `Available` status. Check if the endpoint connection status is `Successful`. For more information, see [How do I troubleshoot AWS DMS endpoint connectivity failures](https://aws.amazon.com/premiumsupport/knowledge-center/dms-endpoint-connectivity-failures/) in the AWS Support documentation. |
| Full-load run fails. | Check if the source and target databases have matching data types and sizes. For more information, see [Troubleshooting migration tasks in AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting.html) in the AWS DMS documentation. |
| You encounter validation run errors. | Check if the table has a primary key because non-primary key tables are not validated.If the table has a primary key and errors, check that the extra connection attribute in the source endpoint has `numberDataTypeScale=-2`.For more information, see [Endpoint settings when using Oracle as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.ConnectionAttrib), [OracleSettings](https://docs.aws.amazon.com/dms/latest/APIReference/API_OracleSettings.html), and [Troubleshooting](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html#CHAP_Validating.Troubleshooting) in the AWS DMS documentation. |
## Related resources
+ [Installing and configuring AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html#CHAP_Installing.Procedure)
+ [Introduction to AWS DMS](https://www.youtube.com/watch?v=ouia1Sc5QGo) (video)
+ [Examples of CloudFormation stack operation commands for the AWS CLI and PowerShell](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cli.html)
+ [Navigating the user interface of the AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html)
+ [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Connecting to Oracle databases with AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html)
+ [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
+ [Sources for data migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html)
+ [Targets for data migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html)
+ [cloudformation](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/index.html) (AWS CLI documentation)
+ [create-stack](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/create-stack.html) (AWS CLI documentation)
+ [dms](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/dms/index.html) (AWS CLI documentation)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/5e041494-2e64-4f09-b6ec-0e0cba3a4972/attachments/attachment.zip)
# Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL in SSL mode by using AWS DMS
*Pinesh Singal, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an Amazon Relational Database Service (Amazon RDS) for Oracle database instance to an Amazon RDS for PostgreSQL database on the Amazon Web Services (AWS) Cloud. To encrypt connections between the databases, the pattern uses certificate authority (CA) and SSL mode in Amazon RDS and AWS Database Migration Service (AWS DMS).
The pattern describes an online migration strategy with little or no downtime for a multi-terabyte Oracle source database with a high number of transactions. For data security, the pattern uses SSL when transferring the data.
This pattern uses AWS Schema Conversion Tool (AWS SCT) to convert the Amazon RDS for Oracle database schema to an Amazon RDS for PostgreSQL schema. Then the pattern uses AWS DMS to migrate data from the Amazon RDS for Oracle database to the Amazon RDS for PostgreSQL database.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ Amazon RDS database certificate authority (CA) configured with ***rds-ca-rsa2048-g1*** only
+ The ***rds-ca-2019*** certificate expired in August 2024.
+ The ***rds-ca-2015*** certificate expired on March 5, 2020.
+ AWS SCT
+ AWS DMS
+ pgAdmin
+ SQL tools (for example, SQL Developer or SQL\$1Plus)
**Limitations **
+ Amazon RDS for Oracle database – The minimum requirement is for Oracle versions 19c for the Enterprise and Standard Two editions.
+ Amazon RDS for PostgreSQL database – The minimum requirement is for PostgreSQL version 12 and later (for versions 9.x and later).
**Product versions**
+ Amazon RDS for Oracle database version 12.1.0.2 instance
+ Amazon RDS for PostgreSQL database version 11.5 instance
## Architecture
**Source technology stack **
+ An Amazon RDS for Oracle database instance with version 12.1.0.2.v18.
**Target technology stack **
+ AWS DMS
+ An Amazon RDS for PostgreSQL database instance with version 11.5.
**Target architecture**
The following diagram shows the architecture for data migration architecture between Oracle (source) and PostgreSQL (target) databases. The architecture includes the following:
+ A virtual private cloud (VPC)
+ An Availability Zone
+ A private subnet
+ An Amazon RDS for Oracle database
+ An AWS DMS replication instance
+ An RDS for PostgreSQL database
To encrypt connections for source and target databases, CA and SSL mode must be enabled in Amazon RDS and AWS DMS.
![\[Data moving between RDS for Oracle and AWS DMS, and between AWS DMS and RDS for PostgreSQL.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7098e2a3-b456-4e14-8881-c97145aef483/images/55b50ff7-1e6a-4ff0-9bcd-2fd419d5316a.png)
## Tools
**AWS services**
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Relational Database Service (Amazon RDS) for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) helps you set up, operate, and scale an Oracle relational database in the AWS Cloud.
+ [Amazon Relational Database Service (Amazon RDS) for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
**Other services**
+ [pgAdmin](https://www.pgadmin.org/) is an open source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
## Best practices
Amazon RDS provides new CA certificates as an AWS security best practice. For information about the new certificates and the supported AWS Regions, see [Using SSL/TLS to encrypt a connection to a DB instance or cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html).
If your RDS instance is currently on CA certificate `rds-ca-2019`, and you want to upgrade to `rds-ca-rsa2048-g1`, follow the instructions in [Updating your CA certificate by modifying your DB instance or cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html#UsingWithRDS.SSL-certificate-rotation-updating) or [Updating your CA certificate by applying maintenance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html#UsingWithRDS.SSL-certificate-rotation-maintenance-update).
## Epics
### Configure the Amazon RDS for Oracle instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Oracle database instance. | Sign in to your AWS account, open the AWS Management Console, and navigate to the Amazon RDS console. On the console, choose **Create database**, and then choose **Oracle**. | General AWS, DBA |
| Configure security groups. | Configure inbound and outbound security groups. | General AWS |
| Create an option group. | Create an option group in the same VPC and security group as the Amazon RDS for Oracle database. For **Option**, choose **SSL**. For **Port**, choose **2484** (for SSL connections). | General AWS |
| Configure the option settings. | Use the following settings:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Modify the RDS for Oracle DB instance. | Set the CA certificate as **rds-ca-rsa2048-g1**. Under **Option group**, attach the previously created option group. | DBA, General AWS |
| Confirm that the RDS for Oracle DB instance is available. | Make sure that the Amazon RDS for Oracle database instance is up and running and that the database schema is accessible.To connect to the RDS for Oracle DB, use the `sqlplus` command from the command line.
$ sqlplus orcl/****@myoracledb.cokmvis0v46q.us-east-1.rds.amazonaws.com:1521/ORCL SQL*Plus: Release 12.1.0.2.0 Production on Tue Oct 15 18:11:07 2019 Copyright (c) 1982, 2016, Oracle. All rights reserved. Last Successful login time: Mon Dec 16 2019 23:17:31 +05:30 Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options SQL>
| DBA |
| Create objects and data in the RDS for Oracle database. | Create objects and insert data in the schema. | DBA |
### Configure the Amazon RDS for PostgreSQL instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create the RDS for PostgreSQL database. | On the Amazon RDS console **Create database** page, choose **PostgreSQL** to create an Amazon RDS for PostgreSQL database instance. | DBA, General AWS |
| Configure security groups. | Configure inbound and outbound security groups. | General AWS |
| Create a parameter group. | If you are using PostgreSQL version 11.x, create a parameter group to set SSL parameters. In PostgreSQL version 12, the SSL parameter group is enabled by default. | General AWS |
| Edit parameters. | Change the `rds.force_ssl` parameter to `1` (on).By default, the `ssl` parameter is `1` (on). By setting the `rds.force_ssl` parameter to `1`, you force all connections to connect through SSL mode only. | General AWS |
| Modify the RDS for PostgreSQL DB instance. | Set the CA certificate as **rds-ca-rsa2048-g1**. Attach the default parameter group or the previously created parameter group, depending on your PostgreSQL version. | DBA, General AWS |
| Confirm that the RDS for PostgreSQL DB instance is available. | Make sure that the Amazon RDS for PostgreSQL database is up and running.The `psql` command establishes an SSL connection with `sslmode` set from the command line.One option is to set `sslmode=1` in the parameter group and use a `psql` connection without including the `sslmode` parameter in the command.The following output shows that the SSL connection is established.
$ psql -h mypgdbinstance.cokmvis0v46q.us-east-1.rds.amazonaws.com -p 5432 "dbname=pgdb user=pguser" Password for user pguser: psql (11.3, server 11.5) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type "help" for help. pgdb=>
A second option is to set `sslmode=1` in the parameter group and to include the `sslmode` parameter in the `psql` command.The following output shows that the SSL connection is established.
$ psql -h mypgdbinstance.cokmvis0v46q.us-east-1.rds.amazonaws.com -p 5432 "dbname=pgdb user=pguser sslmode=require" Password for user pguser: psql (11.3, server 11.5) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type "help" for help. pgdb=>
| DBA |
### Configure and run AWS SCT
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS SCT. | Install the latest version of the AWS SCT application. | General AWS |
| Configure AWS SCT with JDBC drivers. | Download the Java Database Connectivity (JDBC) drivers for Oracle ([ojdbc8.jar](https://download.oracle.com/otn-pub/otn_software/jdbc/233/ojdbc8.jar)) and PostgreSQL ([postgresql-42.2.5.jar](https://jdbc.postgresql.org/download/postgresql-42.2.19.jar)).To configure the drivers in AWS SCT, choose **Settings**, **Global settings**, **Drivers**. | General AWS |
| Create the AWS SCT project. | Create the AWS SCT project and report, using Oracle as the source DB engine and Amazon RDS for PostgreSQL as the target DB engine:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Validate database objects. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | DBA, General AWS |
### Configure and run AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Import the certificate. | Download the [certificate bundle (PEM)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.CertificatesAllRegions) for your AWS Region.The bundle contains both the `rds-ca-2019` intermediate and root certificates. The bundle also contains the `rds-ca-rsa2048-g1`, `rds-ca-rsa4096-g1`, and `rds-ca-ecc384-g1` root CA certificates. Your application trust store needs to register only the root CA certificate. | General AWS |
| Create the source endpoint. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html)For more information, see [Using an Oracle database as a source for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). | General AWS |
| Create the target endpoint. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html)For more information, see [Using a PostgreSQL database as a target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html). | General AWS |
| Test the endpoints. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Create migration tasks. | To create a migration task for full load and change data capture (CDC) or for data validation, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Plan the production run. | Confirm downtime with stakeholders such as application owners to run AWS DMS in production systems. | Migration lead |
| Run the migration task. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Validate the data. | Review migration task results and data in the source Oracle and target PostgreSQL databases:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | DBA |
| Stop the migration task. | After you successfully complete the data validation, stop the migration task. | General AWS |
### Clean up the resources
| Task | Description | Skills required |
| --- | --- | --- |
| Delete the AWS DMS tasks. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Delete the AWS DMS endpoints. | Select the source and target endpoints that you created, choose **Actions**, and choose **Delete**. | General AWS |
| Delete the AWS DMS replication instance. | Choose the replication instance, choose **Actions**, and then choose **Delete**. | General AWS |
| Delete the PostgreSQL database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) | General AWS |
| Delete the Oracle database. | On the Amazon RDS console, select the Oracle database instance, choose **Actions**, and then choose **Delete**. | General AWS |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| AWS SCT source and target test connections are failing. | Configure JDBC driver versions and VPC security group inbound rules to accept the incoming traffic. |
| The Oracle source endpoint test run fails. | Check the endpoint settings and whether the replication instance is available. |
| The AWS DMS task full-load run fails. | Check whether the source and target databases have matching data types and sizes. |
| The AWS DMS validation migration task returns errors. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.html) |
## Related resources
**Databases**
+ [Amazon RDS for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html)
+ [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html)
**SSL DB connection**
+ [Using SSL/TLS to encrypt a connection to a DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)
+ [Using SSL with an RDS for Oracle DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Concepts.SSL.html)
+ [Securing connections to RDS for PostgreSQL with SSL/TLS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.Security.html)
+ [Download certificate bundles for specific AWS Regions](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.CertificatesAllRegions)
+ [Download CA-2019 root certificate](https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem) (expired in August 2024)
+ [Working with option groups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithOptionGroups.html)
+ [Adding options to Oracle DB instances](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.Options.html)
+ [Oracle Secure Sockets Layer](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.Options.SSL.html)
+ [Working with parameter groups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html)
+ [PostgreSQL sslmode connection parameter](https://www.postgresql.org/docs/11/libpq-connect.html#LIBPQ-CONNECT-SSLMODE)
+ [Using SSL from JDBC](https://jdbc.postgresql.org/documentation/ssl/)
+ [Rotating your SSL/TLS certificate](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html)
+ [Updating your CA certificate by modifying your DB instance or cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html#UsingWithRDS.SSL-certificate-rotation-updating)
+ [Updating your CA certificate by applying maintenance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html#UsingWithRDS.SSL-certificate-rotation-maintenance-update)
**AWS SCT**
+ [AWS Schema Conversion Tool](https://aws.amazon.com/dms/schema-conversion-tool/)
+ [AWS Schema Conversion Tool User Guide](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Using the AWS SCT user interface](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html)
+ [Using Oracle Database as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html)
**AWS DMS**
+ [AWS Database Migration Service](https://aws.amazon.com/dms/)
+ [AWS Database Migration Service User Guide](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
+ [Using SSL with AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.SSL.html)
+ [Migrating Applications Running Relational Databases to AWS](https://d1.awsstatic.com/whitepapers/Migration/migrating-applications-to-aws.pdf)
## Additional information
Amazon RDS Certificate Authority certificates `rds-ca-2019` expired in August 2024. If you use or plan to use SSL or TLS with certificate verification to connect to your RDS DB instances or Multi-AZ DB clusters, consider using one of the new CA certificates: `rds-ca-rsa2048-g1`, `rds-ca-rsa4096-g1`, or `rds-ca-ecc384-g1`.
# Migrate Oracle SERIALLY\$1REUSABLE pragma packages into PostgreSQL
*Vinay Paladi, Amazon Web Services*
## Summary
This pattern provides a step-by-step approach for migrating Oracle packages that are defined as SERIALLY\$1REUSABLE pragma to PostgreSQL on Amazon Web Services (AWS). This approach maintains the functionality of the SERIALLY\$1REUSABLE pragma.
PostgreSQL doesn’t support the concept of packages and the SERIALLY\$1REUSABLE pragma. To get similar functionality in PostgreSQL, you can create schemas for packages and deploy all the related objects (such as functions, procedures, and types) inside the schemas. To achieve the functionality of the SERIALLY\$1REUSABLE pragma, the example wrapper function script that’s provided in this pattern uses an [AWS Schema Conversion Tool (AWS SCT) extension pack](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_ExtensionPack.html).
For more information, see [SERIALLY\$1REUSABLE Pragma](https://docs.oracle.com/cd/B13789_01/appdev.101/b10807/13_elems046.htm) in the Oracle documentation.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ The latest version of AWS SCT and the required drivers
+ An Amazon Aurora PostgreSQL-Compatible Edition database or an Amazon Relational Database Service (Amazon RDS) for PostgreSQL database
**Product versions**
+ Oracle Database version 10g and later
## Architecture
**Source technology stack**
+ Oracle Database on premises
**Target technology stack**
+ [Aurora PostgreSQL-Compatible](https://aws.amazon.com/rds/aurora/details/postgresql-details/) or Amazon RDS for PostgreSQL
+ AWS SCT
**Migration architecture**
![\[On-premises Oracle DB data going to AWS using AWS SCT, .sql files, manual conversion, to PostgreSQL.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fe3c45d2-6ea4-43b5-adb1-18f068f126b9/images/2dc90708-e300-4251-9d12-de97b6588b72.png)
## Tools
**AWS services**
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [Amazon Relational Database Service (Amazon RDS) for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
**Other tools**
+ [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
## Epics
### Migrate the Oracle package by using AWS SCT
| Task | Description | Skills required |
| --- | --- | --- |
| Set up AWS SCT. | Configure AWS SCT connectivity to the source database. For more information, see [Using Oracle Database as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html). | DBA, Developer |
| Convert the script. | Use AWS SCT to convert the Oracle package by selecting the target database as Aurora PostgreSQL-Compatible. | DBA, Developer |
| Save the .sql files. | Before you save the .sql file, modify the **Project Settings** option in AWS SCT to **Single file per stage**. AWS SCT will separate the .sql file into multiple .sql files based on object type. | DBA, Developer |
| Change the code. | Open the `init` function generated by AWS SCT, and change it as shown in the example in the *Additional information* section. It will add a variable to achieve the functionality `pg_serialize = 0`. | DBA, Developer |
| Test the conversion. | Deploy the `init` function to the Aurora PostgreSQL-Compatible database, and test the results. | DBA, Developer |
## Related resources
+ [AWS Schema Conversion Tool](https://aws.amazon.com/dms/schema-conversion-tool/)
+ [Amazon RDS](https://aws.amazon.com/rds/)
+ [Amazon Aurora features](https://aws.amazon.com/rds/aurora/postgresql-features/)
+ [SERIALLY\$1REUSABLE Pragma](https://docs.oracle.com/cd/B28359_01/appdev.111/b28370/seriallyreusable_pragma.htm#LNPLS01346)
## Additional information
```
Source Oracle Code:
CREATE OR REPLACE PACKAGE test_pkg_var
IS
PRAGMA SERIALLY_REUSABLE;
PROCEDURE function_1
(test_id number);
PROCEDURE function_2
(test_id number
);
END;
CREATE OR REPLACE PACKAGE BODY test_pkg_var
IS
PRAGMA SERIALLY_REUSABLE;
v_char VARCHAR2(20) := 'shared.airline';
v_num number := 123;
PROCEDURE function_1(test_id number)
IS
begin
dbms_output.put_line( 'v_char-'|| v_char);
dbms_output.put_line( 'v_num-'||v_num);
v_char:='test1';
function_2(0);
END;
PROCEDURE function_2(test_id number)
is
begin
dbms_output.put_line( 'v_char-'|| v_char);
dbms_output.put_line( 'v_num-'||v_num);
END;
END test_pkg_var;
Calling the above functions
set serveroutput on
EXEC test_pkg_var.function_1(1);
EXEC test_pkg_var.function_2(1);
Target Postgresql Code:
CREATE SCHEMA test_pkg_var;
CREATE OR REPLACE FUNCTION test_pkg_var.init(pg_serialize IN INTEGER DEFAULT 0)
RETURNS void
AS
$BODY$
DECLARE
BEGIN
if aws_oracle_ext.is_package_initialized( 'test_pkg_var' ) AND pg_serialize = 0
then
return;
end if;
PERFORM aws_oracle_ext.set_package_initialized( 'test_pkg_var' );
PERFORM aws_oracle_ext.set_package_variable( 'test_pkg_var', 'v_char', 'shared.airline.basecurrency'::CHARACTER
VARYING(100));
PERFORM aws_oracle_ext.set_package_variable('test_pkg_var', 'v_num', 123::integer);
END;
$BODY$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION test_pkg_var.function_1(pg_serialize int default 1)
RETURNS void
AS
$BODY$
DECLARE
BEGIN
PERFORM test_pkg_var.init(pg_serialize);
raise notice 'v_char%',aws_oracle_ext.get_package_variable( 'test_pkg_var', 'v_char');
raise notice 'v_num%',aws_oracle_ext.get_package_variable( 'test_pkg_var', 'v_num');
PERFORM aws_oracle_ext.set_package_variable( 'test_pkg_var', 'v_char', 'test1'::varchar);
PERFORM test_pkg_var.function_2(0);
END;
$BODY$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION test_pkg_var.function_2(IN pg_serialize integer default 1)
RETURNS void
AS
$BODY$
DECLARE
BEGIN
PERFORM test_pkg_var.init(pg_serialize);
raise notice 'v_char%',aws_oracle_ext.get_package_variable( 'test_pkg_var', 'v_char');
raise notice 'v_num%',aws_oracle_ext.get_package_variable( 'test_pkg_var', 'v_num');
END;
$BODY$
LANGUAGE plpgsql;
Calling the above functions
select test_pkg_var.function_1()
select test_pkg_var.function_2()
```
# Migrate Oracle external tables to Amazon Aurora PostgreSQL-Compatible
*anuradha chintha and Rakesh Raghav, Amazon Web Services*
## Summary
External tables give Oracle the ability to query data that is stored outside the database in flat files. You can use the ORACLE\$1LOADER driver to access any data stored in any format that can be loaded by the SQL\$1Loader utility. You can't use Data Manipulation Language (DML) on external tables, but you can use external tables for query, join, and sort operations.
Amazon Aurora PostgreSQL-Compatible Edition doesn't provide functionality similar to external tables in Oracle. Instead, you must use modernization to develop a scalable solution that meets functional requirements and is frugal.
This pattern provides steps for migrating different types of Oracle external tables to Aurora PostgreSQL-Compatible Edition on the Amazon Web Services (AWS) Cloud by using the `aws_s3` extension.
We recommend thoroughly testing this solution before implementing it in a production environment.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ AWS Command Line Interface (AWS CLI)
+ An available Aurora PostgreSQL-Compatible database instance.
+ An on-premises Oracle database with an external table
+ pg.Client API
+ Data files
**Limitations **
+ This pattern doesn't provide the functionality to act as a replacement for Oracle external tables. However, the steps and sample code can be enhanced further to achieve your database modernization goals.
+ Files should not contain the character that is passing as a delimiter in `aws_s3` export and import functions.
**Product versions**
+ To import from Amazon S3 into RDS for PostgreSQL the database must be running PostgreSQL version 10.7 or later.
## Architecture
**Source technology stack **
+ Oracle
**Source architecture **
![\[Diagram of data files going to a directory and table in the on-premises Oracle database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/555e69af-36fc-4ff5-b66c-af22b4cf262a/images/3fbc507d-b0fa-4e05-b999-043dc7327ed7.png)
**Target technology stack **
+ Amazon Aurora PostgreSQL-Compatible
+ Amazon CloudWatch
+ AWS Lambda
+ AWS Secrets Manager
+ Amazon Simple Notification Service (Amazon SNS)
+ Amazon Simple Storage Service (Amazon S3)
**Target architecture **
The following diagram shows a high-level representation of the solution.
![\[The description is after the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/555e69af-36fc-4ff5-b66c-af22b4cf262a/images/5421540e-d2e3-4361-89cc-d8415fcb21fd.png)
1. Files are uploaded to the S3 bucket.
1. The Lambda function is initiated.
1. The Lambda function initiates the DB function call.
1. Secrets Manager provides the credentials for database access.
1. Depending on the DB function, an SNS alarm is created.
**Automation and scale**
Any additions or changes to the external tables can be handled with metadata maintenance.
## Tools
+ [Amazon Aurora PostgreSQL-Compatible](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) – Amazon Aurora PostgreSQL-Compatible Edition is a fully managed, PostgreSQL-compatible, and ACID-compliant relational database engine that combines the speed and reliability of high-end commercial databases with the cost-effectiveness of open-source databases.
+ [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) – AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services. With only one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) – Amazon CloudWatch monitors Amazon S3 resources and utilization.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) – AWS Lambda is a serverless compute service that supports running code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. In this pattern, Lambda runs the database function whenever a file is uploaded to Amazon S3.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) – AWS Secrets Manager is a service for credential storage and retrieval. Using Secrets Manager, you can replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
+ [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) – Amazon Simple Storage Service (Amazon S3) provides a storage layer to receive and store files for consumption and transmission to and from the Aurora PostgreSQL-Compatible cluster.
+ [aws\$1s3](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Procedural.Importing.html#aws_s3.table_import_from_s3) – The `aws_s3` extension integrates Amazon S3 and Aurora PostgreSQL-Compatible.
+ [Amazon SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) – Amazon Simple Notification Service (Amazon SNS) coordinates and manages the delivery or sending of messages between publishers and clients. In this pattern, Amazon SNS is used to send notifications.
**Code **
Whenever a file is placed in the S3 bucket, a DB function must be created and called from the processing application or the Lambda function. For details, see the code (attached).
## Epics
### Create an external file
| Task | Description | Skills required |
| --- | --- | --- |
| Add an external file to the source database. | Create an external file, and move it to the `oracle` directory. | DBA |
### Configure the target (Aurora PostgreSQL-Compatible)
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Aurora PostgreSQL database. | Create a DB instance in your Amazon Aurora PostgreSQL-Compatible cluster. | DBA |
| Create a schema, aws\$1s3 extension, and tables. | Use the code under `ext_tbl_scripts` in the *Additional information* section. The tables include actual tables, staging tables, error and log tables, and a metatable. | DBA, Developer |
| Create the DB function. | To create the DB function, use the code under `load_external_table_latest` function in the *Additional information* section. | DBA, Developer |
### Create and configure the Lambda function
| Task | Description | Skills required |
| --- | --- | --- |
| Create a role. | Create a role with permissions to access Amazon S3 and Amazon Relational Database Service (Amazon RDS). This role will be assigned to Lambda for running the pattern. | DBA |
| Create the Lambda function. | Create a Lambda function that reads the file name from Amazon S3 (for example, `file_key = info.get('object', {}).get('key')`) and calls the DB function (for example, `curs.callproc("load_external_tables", [file_key])`) with the file name as the input parameter.Depending on the function call result, an SNS notification will be initiated (for example, `client.publish(TopicArn='arn:',Message='fileloadsuccess',Subject='fileloadsuccess')`).Based on your business needs, you can create a Lambda function with extra code if required. For more information, see the [Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html). | DBA |
| Configure an S3 bucket event trigger. | Configure a mechanism to call the Lambda function for all object creation events in the S3 bucket. | DBA |
| Create a secret. | Create a secret name for the database credentials by using Secrets Manager. Pass the secret in the Lambda function. | DBA |
| Upload the Lambda supporting files. | Upload a .zip file that contains the Lambda support packages and the attached Python script for connecting to Aurora PostgreSQL-Compatible. The Python code calls the function that you created in the database. | DBA |
| Create an SNS topic. | Create an SNS topic to send mail for the success or failure of the data load. | DBA |
### Add integration with Amazon S3
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | On the Amazon S3 console, create an S3 bucket with a unique name that does not contain leading slashes. An S3 bucket name is globally unique, and the namespace is shared by all AWS accounts. | DBA |
| Create IAM policies. | To create the AWS Identity and Access Management (IAM) policies, use the code under `s3bucketpolicy_for_import` in the *Additional information* section. | DBA |
| Create roles. | Create two roles for Aurora PostgreSQL-Compatible, one role for Import and one role for Export. Assign the corresponding policies to the roles. | DBA |
| Attach the roles to the Aurora PostgreSQL-Compatible cluster. | Under **Manage roles**, attach the Import and Export roles to the Aurora PostgreSQL cluster. | DBA |
| Create supporting objects for Aurora PostgreSQL-Compatible. | For the table scripts, use the code under `ext_tbl_scripts` in the *Additional information* section.For the custom function, use the code under `load_external_Table_latest` in the *Additional information* section. | DBA |
### Process a test file
| Task | Description | Skills required |
| --- | --- | --- |
| Upload a file into the S3 bucket. | To upload a test file into the S3 bucket, use the console or the following command in AWS CLI.
As soon as the file is uploaded, a bucket event initiates the Lambda function, which runs the Aurora PostgreSQL-Compatible function. | DBA |
| Check the data and the log and error files. | The Aurora PostgreSQL-Compatible function loads the files into the main table, and it creates `.log` and `.bad` files in the S3 bucket. | DBA |
| Monitor the solution. | In the Amazon CloudWatch console, monitor the Lambda function. | DBA |
## Related resources
+ [Amazon S3 integration](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-s3-integration.html)
+ [Amazon S3](https://aws.amazon.com/s3/)
+ [Working with Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html)
+ [AWS Lambda](https://aws.amazon.com/lambda/)
+ [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/)
+ [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/)
+ [Setting up Amazon SNS notifications](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html)
## Additional information
**ext\$1table\$1scripts**
```
CREATE EXTENSION aws_s3 CASCADE;
CREATE TABLE IF NOT EXISTS meta_EXTERNAL_TABLE
(
table_name_stg character varying(100) ,
table_name character varying(100) ,
col_list character varying(1000) ,
data_type character varying(100) ,
col_order numeric,
start_pos numeric,
end_pos numeric,
no_position character varying(100) ,
date_mask character varying(100) ,
delimeter character(1) ,
directory character varying(100) ,
file_name character varying(100) ,
header_exist character varying(5)
);
CREATE TABLE IF NOT EXISTS ext_tbl_stg
(
col1 text
);
CREATE TABLE IF NOT EXISTS error_table
(
error_details text,
file_name character varying(100),
processed_time timestamp without time zone
);
CREATE TABLE IF NOT EXISTS log_table
(
file_name character varying(50) COLLATE pg_catalog."default",
processed_date timestamp without time zone,
tot_rec_count numeric,
proc_rec_count numeric,
error_rec_count numeric
);
sample insert scripts of meta data:
INSERT INTO meta_EXTERNAL_TABLE (table_name_stg, table_name, col_list, data_type, col_order, start_pos, end_pos, no_position, date_mask, delimeter, directory, file_name, header_exist) VALUES ('F_EX_APS_TRANSACTIONS_STG', 'F_EX_APS_TRANSACTIONS', 'source_filename', 'character varying', 2, 8, 27, NULL, NULL, NULL, 'databasedev', 'externalinterface/loaddir/APS', 'NO');
INSERT INTO meta_EXTERNAL_TABLE (table_name_stg, table_name, col_list, data_type, col_order, start_pos, end_pos, no_position, date_mask, delimeter, directory, file_name, header_exist) VALUES ('F_EX_APS_TRANSACTIONS_STG', 'F_EX_APS_TRANSACTIONS', 'record_type_identifier', 'character varying', 3, 28, 30, NULL, NULL, NULL, 'databasedev', 'externalinterface/loaddir/APS', 'NO');
INSERT INTO meta_EXTERNAL_TABLE (table_name_stg, table_name, col_list, data_type, col_order, start_pos, end_pos, no_position, date_mask, delimeter, directory, file_name, header_exist) VALUES ('F_EX_APS_TRANSACTIONS_STG', 'F_EX_APS_TRANSACTIONS', 'fad_code', 'numeric', 4, 31, 36, NULL, NULL, NULL, 'databasedev', 'externalinterface/loaddir/APS', 'NO');
INSERT INTO meta_EXTERNAL_TABLE (table_name_stg, table_name, col_list, data_type, col_order, start_pos, end_pos, no_position, date_mask, delimeter, directory, file_name, header_exist) VALUES ('F_EX_APS_TRANSACTIONS_STG', 'F_EX_APS_TRANSACTIONS', 'session_sequence_number', 'numeric', 5, 37, 42, NULL, NULL, NULL, 'databasedev', 'externalinterface/loaddir/APS', 'NO');
INSERT INTO meta_EXTERNAL_TABLE (table_name_stg, table_name, col_list, data_type, col_order, start_pos, end_pos, no_position, date_mask, delimeter, directory, file_name, header_exist) VALUES ('F_EX_APS_TRANSACTIONS_STG', 'F_EX_APS_TRANSACTIONS', 'transaction_sequence_number', 'numeric', 6, 43, 48, NULL, NULL, NULL, 'databasedev', 'externalinterface/loaddir/APS', 'NO');
```
**s3bucketpolicy\$1for import**
```
---Import role policy
--Create an IAM policy to allow, Get, and list actions on S3 bucket
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "s3import",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::s3importtest",
"arn:aws:s3:::s3importtest/*"
]
}
]
}
--Export Role policy
--Create an IAM policy to allow, put, and list actions on S3 bucket
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "s3export",
"Action": [
"S3:PutObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::s3importtest/*"
]
}
]
}
```
**Sample DB function load\$1external\$1tables\$1latest**
```
CREATE OR REPLACE FUNCTION public.load_external_tables(pi_filename text)
RETURNS character varying
LANGUAGE plpgsql
AS $function$
/* Loading data from S3 bucket into a APG table */
DECLARE
v_final_sql TEXT;
pi_ext_table TEXT;
r refCURSOR;
v_sqlerrm text;
v_chunk numeric;
i integer;
v_col_list TEXT;
v_postion_list CHARACTER VARYING(1000);
v_len integer;
v_delim varchar;
v_file_name CHARACTER VARYING(1000);
v_directory CHARACTER VARYING(1000);
v_table_name_stg CHARACTER VARYING(1000);
v_sql_col TEXT;
v_sql TEXT;
v_sql1 TEXT;
v_sql2 TEXT;
v_sql3 TEXT;
v_cnt integer;
v_sql_dynamic TEXT;
v_sql_ins TEXT;
proc_rec_COUNT integer;
error_rec_COUNT integer;
tot_rec_COUNT integer;
v_rec_val integer;
rec record;
v_col_cnt integer;
kv record;
v_val text;
v_header text;
j integer;
ERCODE VARCHAR(5);
v_region text;
cr CURSOR FOR
SELECT distinct DELIMETER,
FILE_NAME,
DIRECTORY
FROM meta_EXTERNAL_TABLE
WHERE table_name = pi_ext_table
AND DELIMETER IS NOT NULL;
cr1 CURSOR FOR
SELECT col_list,
data_type,
start_pos,
END_pos,
concat_ws('',' ',TABLE_NAME_STG) as TABLE_NAME_STG,
no_position,date_mask
FROM meta_EXTERNAL_TABLE
WHERE table_name = pi_ext_table
order by col_order asc;
cr2 cursor FOR
SELECT distinct table_name,table_name_stg
FROM meta_EXTERNAL_TABLE
WHERE upper(file_name) = upper(pi_filename);
BEGIN
-- PERFORM utl_file_utility.init();
v_region := 'us-east-1';
/* find tab details from file name */
--DELETE FROM ERROR_TABLE WHERE file_name= pi_filename;
-- DELETE FROM log_table WHERE file_name= pi_filename;
BEGIN
SELECT distinct table_name,table_name_stg INTO strict pi_ext_table,v_table_name_stg
FROM meta_EXTERNAL_TABLE
WHERE upper(file_name) = upper(pi_filename);
EXCEPTION
WHEN NO_DATA_FOUND THEN
raise notice 'error 1,%',sqlerrm;
pi_ext_table := null;
v_table_name_stg := null;
RAISE USING errcode = 'NTFIP' ;
when others then
raise notice 'error others,%',sqlerrm;
END;
j :=1 ;
for rec in cr2
LOOP
pi_ext_table := rec.table_name;
v_table_name_stg := rec.table_name_stg;
v_col_list := null;
IF pi_ext_table IS NOT NULL
THEN
--EXECUTE concat_ws('','truncate table ' ,pi_ext_table) ;
EXECUTE concat_ws('','truncate table ' ,v_table_name_stg) ;
SELECT distinct DELIMETER INTO STRICT v_delim
FROM meta_EXTERNAL_TABLE
WHERE table_name = pi_ext_table;
IF v_delim IS NOT NULL THEN
SELECT distinct DELIMETER,
FILE_NAME,
DIRECTORY ,
concat_ws('',' ',table_name_stg),
case header_exist when 'YES' then 'CSV HEADER' else 'CSV' end as header_exist
INTO STRICT v_delim,v_file_name,v_directory,v_table_name_stg,v_header
FROM meta_EXTERNAL_TABLE
WHERE table_name = pi_ext_table
AND DELIMETER IS NOT NULL;
IF upper(v_delim) = 'CSV'
THEN
v_sql := concat_ws('','SELECT aws_s3.table_import_FROM_s3 ( ''',
v_table_name_stg,''','''',
''DELIMITER '''','''' CSV HEADER QUOTE ''''"'''''', aws_commons.create_s3_uri ( ''',
v_directory,''',''',v_file_name,''', ''',v_region,'''))');
ELSE
v_sql := concat_ws('','SELECT aws_s3.table_import_FROM_s3(''',
v_table_name_stg, ''','''', ''DELIMITER AS ''''^''''',''',','
aws_commons.create_s3_uri
( ''',v_directory, ''',''',
v_file_name, ''',',
'''',v_region,''')
)');
raise notice 'v_sql , %',v_sql;
begin
EXECUTE v_sql;
EXCEPTION
WHEN OTHERS THEN
raise notice 'error 1';
RAISE USING errcode = 'S3IMP' ;
END;
select count(col_list) INTO v_col_cnt
from meta_EXTERNAL_TABLE where table_name = pi_ext_table;
-- raise notice 'v_sql 2, %',concat_ws('','update ',v_table_name_stg, ' set col1 = col1||''',v_delim,'''');
execute concat_ws('','update ',v_table_name_stg, ' set col1 = col1||''',v_delim,'''');
i :=1;
FOR rec in cr1
loop
v_sql1 := concat_ws('',v_sql1,'split_part(col1,''',v_delim,''',', i,')',' as ',rec.col_list,',');
v_sql2 := concat_ws('',v_sql2,rec.col_list,',');
-- v_sql3 := concat_ws('',v_sql3,'rec.',rec.col_list,'::',rec.data_type,',');
case
WHEN upper(rec.data_type) = 'NUMERIC'
THEN v_sql3 := concat_ws('',v_sql3,' case WHEN length(trim(split_part(col1,''',v_delim,''',', i,'))) =0
THEN null
ELSE
coalesce((trim(split_part(col1,''',v_delim,''',', i,')))::NUMERIC,0)::',rec.data_type,' END as ',rec.col_list,',') ;
WHEN UPPER(rec.data_type) = 'TIMESTAMP WITHOUT TIME ZONE' AND rec.date_mask = 'YYYYMMDD'
THEN v_sql3 := concat_ws('',v_sql3,' case WHEN length(trim(split_part(col1,''',v_delim,''',', i,'))) =0
THEN null
ELSE
to_date(coalesce((trim(split_part(col1,''',v_delim,''',', i,'))),''99990101''),''YYYYMMDD'')::',rec.data_type,' END as ',rec.col_list,',');
WHEN UPPER(rec.data_type) = 'TIMESTAMP WITHOUT TIME ZONE' AND rec.date_mask = 'MM/DD/YYYY hh24:mi:ss'
THEN v_sql3 := concat_ws('',v_sql3,' case WHEN length(trim(split_part(col1,''',v_delim,''',', i,'))) =0
THEN null
ELSE
to_date(coalesce((trim(split_part(col1,''',v_delim,''',', i,'))),''01/01/9999 0024:00:00''),''MM/DD/YYYY hh24:mi:ss'')::',rec.data_type,' END as ',rec.col_list,',');
ELSE
v_sql3 := concat_ws('',v_sql3,' case WHEN length(trim(split_part(col1,''',v_delim,''',', i,'))) =0
THEN null
ELSE
coalesce((trim(split_part(col1,''',v_delim,''',', i,'))),'''')::',rec.data_type,' END as ',rec.col_list,',') ;
END case;
i :=i+1;
end loop;
-- raise notice 'v_sql 3, %',v_sql3;
SELECT trim(trailing ' ' FROM v_sql1) INTO v_sql1;
SELECT trim(trailing ',' FROM v_sql1) INTO v_sql1;
SELECT trim(trailing ' ' FROM v_sql2) INTO v_sql2;
SELECT trim(trailing ',' FROM v_sql2) INTO v_sql2;
SELECT trim(trailing ' ' FROM v_sql3) INTO v_sql3;
SELECT trim(trailing ',' FROM v_sql3) INTO v_sql3;
END IF;
raise notice 'v_delim , %',v_delim;
EXECUTE concat_ws('','SELECT COUNT(*) FROM ',v_table_name_stg) INTO v_cnt;
raise notice 'stg cnt , %',v_cnt;
/* if upper(v_delim) = 'CSV' then
v_sql_ins := concat_ws('', ' SELECT * from ' ,v_table_name_stg );
else
-- v_sql_ins := concat_ws('',' SELECT ',v_sql1,' from (select col1 from ' ,v_table_name_stg , ')sub ');
v_sql_ins := concat_ws('',' SELECT ',v_sql3,' from (select col1 from ' ,v_table_name_stg , ')sub ');
END IF;*/
v_chunk := v_cnt/100;
for i in 1..101
loop
BEGIN
-- raise notice 'v_sql , %',v_sql;
-- raise notice 'Chunk number , %',i;
v_sql_ins := concat_ws('',' SELECT ',v_sql3,' from (select col1 from ' ,v_table_name_stg , ' offset ',v_chunk*(i-1), ' limit ',v_chunk,') sub ');
v_sql := concat_ws('','insert into ', pi_ext_table ,' ', v_sql_ins);
-- raise notice 'select statement , %',v_sql_ins;
-- v_sql := null;
-- EXECUTE concat_ws('','insert into ', pi_ext_table ,' ', v_sql_ins, 'offset ',v_chunk*(i-1), ' limit ',v_chunk );
--v_sql := concat_ws('','insert into ', pi_ext_table ,' ', v_sql_ins );
-- raise notice 'insert statement , %',v_sql;
raise NOTICE 'CHUNK START %',v_chunk*(i-1);
raise NOTICE 'CHUNK END %',v_chunk;
EXECUTE v_sql;
EXCEPTION
WHEN OTHERS THEN
-- v_sql_ins := concat_ws('',' SELECT ',v_sql1, ' from (select col1 from ' ,v_table_name_stg , ' )sub ');
-- raise notice 'Chunk number for cursor , %',i;
raise NOTICE 'Cursor - CHUNK START %',v_chunk*(i-1);
raise NOTICE 'Cursor - CHUNK END %',v_chunk;
v_sql_ins := concat_ws('',' SELECT ',v_sql3, ' from (select col1 from ' ,v_table_name_stg , ' )sub ');
v_final_sql := REPLACE (v_sql_ins, ''''::text, ''''''::text);
-- raise notice 'v_final_sql %',v_final_sql;
v_sql :=concat_ws('','do $a$ declare r refcursor;v_sql text; i numeric;v_conname text; v_typ ',pi_ext_table,'[]; v_rec ','record',';
begin
open r for execute ''select col1 from ',v_table_name_stg ,' offset ',v_chunk*(i-1), ' limit ',v_chunk,''';
loop
begin
fetch r into v_rec;
EXIT WHEN NOT FOUND;
v_sql := concat_ws('''',''insert into ',pi_ext_table,' SELECT ',REPLACE (v_sql3, ''''::text, ''''''::text) , ' from ( select '''''',v_rec.col1,'''''' as col1) v'');
execute v_sql;
exception
when others then
v_sql := ''INSERT INTO ERROR_TABLE VALUES (concat_ws('''''''',''''Error Name: '''',$$''||SQLERRM||''$$,''''Error State: '''',''''''||SQLSTATE||'''''',''''record : '''',$$''||v_rec.col1||''$$),'''''||pi_filename||''''',now())'';
execute v_sql;
continue;
end ;
end loop;
close r;
exception
when others then
raise;
end ; $a$');
-- raise notice ' inside excp v_sql %',v_sql;
execute v_sql;
-- raise notice 'v_sql %',v_sql;
END;
END LOOP;
ELSE
SELECT distinct DELIMETER,FILE_NAME,DIRECTORY ,concat_ws('',' ',table_name_stg),
case header_exist when 'YES' then 'CSV HEADER' else 'CSV' end as header_exist
INTO STRICT v_delim,v_file_name,v_directory,v_table_name_stg,v_header
FROM meta_EXTERNAL_TABLE
WHERE table_name = pi_ext_table ;
v_sql := concat_ws('','SELECT aws_s3.table_import_FROM_s3(''',
v_table_name_stg, ''','''', ''DELIMITER AS ''''#'''' ',v_header,' '',','
aws_commons.create_s3_uri
( ''',v_directory, ''',''',
v_file_name, ''',',
'''',v_region,''')
)');
EXECUTE v_sql;
FOR rec in cr1
LOOP
IF rec.start_pos IS NULL AND rec.END_pos IS NULL AND rec.no_position = 'recnum'
THEN
v_rec_val := 1;
ELSE
case
WHEN upper(rec.data_type) = 'NUMERIC'
THEN v_sql1 := concat_ws('',' case WHEN length(trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))) =0
THEN null
ELSE
coalesce((trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1)))::NUMERIC,0)::',rec.data_type,' END as ',rec.col_list,',') ;
WHEN UPPER(rec.data_type) = 'TIMESTAMP WITHOUT TIME ZONE' AND rec.date_mask = 'YYYYMMDD'
THEN v_sql1 := concat_ws('','case WHEN length(trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))) =0
THEN null
ELSE
to_date(coalesce((trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))),''99990101''),''YYYYMMDD'')::',rec.data_type,' END as ',rec.col_list,',');
WHEN UPPER(rec.data_type) = 'TIMESTAMP WITHOUT TIME ZONE' AND rec.date_mask = 'YYYYMMDDHH24MISS'
THEN v_sql1 := concat_ws('','case WHEN length(trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))) =0
THEN null
ELSE
to_date(coalesce((trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))),''9999010100240000''),''YYYYMMDDHH24MISS'')::',rec.data_type,' END as ',rec.col_list,',');
ELSE
v_sql1 := concat_ws('',' case WHEN length(trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))) =0
THEN null
ELSE
coalesce((trim(substring(COL1, ',rec.start_pos ,',', rec.END_pos,'-',rec.start_pos ,'+1))),'''')::',rec.data_type,' END as ',rec.col_list,',') ;
END case;
END IF;
v_col_list := concat_ws('',v_col_list ,v_sql1);
END LOOP;
SELECT trim(trailing ' ' FROM v_col_list) INTO v_col_list;
SELECT trim(trailing ',' FROM v_col_list) INTO v_col_list;
v_sql_col := concat_ws('',trim(trailing ',' FROM v_col_list) , ' FROM ',v_table_name_stg,' WHERE col1 IS NOT NULL AND length(col1)>0 ');
v_sql_dynamic := v_sql_col;
EXECUTE concat_ws('','SELECT COUNT(*) FROM ',v_table_name_stg) INTO v_cnt;
IF v_rec_val = 1 THEN
v_sql_ins := concat_ws('',' select row_number() over(order by ctid) as line_number ,' ,v_sql_dynamic) ;
ELSE
v_sql_ins := concat_ws('',' SELECT' ,v_sql_dynamic) ;
END IF;
BEGIN
EXECUTE concat_ws('','insert into ', pi_ext_table ,' ', v_sql_ins);
EXCEPTION
WHEN OTHERS THEN
IF v_rec_val = 1 THEN
v_final_sql := ' select row_number() over(order by ctid) as line_number ,col1 from ';
ELSE
v_final_sql := ' SELECT col1 from';
END IF;
v_sql :=concat_ws('','do $a$ declare r refcursor;v_rec_val numeric := ',coalesce(v_rec_val,0),';line_number numeric; col1 text; v_typ ',pi_ext_table,'[]; v_rec ',pi_ext_table,';
begin
open r for execute ''',v_final_sql, ' ',v_table_name_stg,' WHERE col1 IS NOT NULL AND length(col1)>0 '' ;
loop
begin
if v_rec_val = 1 then
fetch r into line_number,col1;
else
fetch r into col1;
end if;
EXIT WHEN NOT FOUND;
if v_rec_val = 1 then
select line_number,',trim(trailing ',' FROM v_col_list) ,' into v_rec;
else
select ',trim(trailing ',' FROM v_col_list) ,' into v_rec;
end if;
insert into ',pi_ext_table,' select v_rec.*;
exception
when others then
INSERT INTO ERROR_TABLE VALUES (concat_ws('''',''Error Name: '',SQLERRM,''Error State: '',SQLSTATE,''record : '',v_rec),''',pi_filename,''',now());
continue;
end ;
end loop;
close r;
exception
when others then
raise;
end ; $a$');
execute v_sql;
END;
END IF;
EXECUTE concat_ws('','SELECT COUNT(*) FROM ' ,pi_ext_table) INTO proc_rec_COUNT;
EXECUTE concat_ws('','SELECT COUNT(*) FROM error_table WHERE file_name =''',pi_filename,''' and processed_time::date = clock_timestamp()::date') INTO error_rec_COUNT;
EXECUTE concat_ws('','SELECT COUNT(*) FROM ',v_table_name_stg) INTO tot_rec_COUNT;
INSERT INTO log_table values(pi_filename,now(),tot_rec_COUNT,proc_rec_COUNT, error_rec_COUNT);
raise notice 'v_directory, %',v_directory;
raise notice 'pi_filename, %',pi_filename;
raise notice 'v_region, %',v_region;
perform aws_s3.query_export_to_s3('SELECT replace(trim(substring(error_details,position(''('' in error_details)+1),'')''),'','','';''),file_name,processed_time FROM error_table WHERE file_name = '''||pi_filename||'''',
aws_commons.create_s3_uri(v_directory, pi_filename||'.bad', v_region),
options :='FORmat csv, header, delimiter $$,$$'
);
raise notice 'v_directory, %',v_directory;
raise notice 'pi_filename, %',pi_filename;
raise notice 'v_region, %',v_region;
perform aws_s3.query_export_to_s3('SELECT * FROM log_table WHERE file_name = '''||pi_filename||'''',
aws_commons.create_s3_uri(v_directory, pi_filename||'.log', v_region),
options :='FORmat csv, header, delimiter $$,$$'
);
END IF;
j := j+1;
END LOOP;
RETURN 'OK';
EXCEPTION
WHEN OTHERS THEN
raise notice 'error %',sqlerrm;
ERCODE=SQLSTATE;
IF ERCODE = 'NTFIP' THEN
v_sqlerrm := concat_Ws('',sqlerrm,'No data for the filename');
ELSIF ERCODE = 'S3IMP' THEN
v_sqlerrm := concat_Ws('',sqlerrm,'Error While exporting the file from S3');
ELSE
v_sqlerrm := sqlerrm;
END IF;
select distinct directory into v_directory from meta_EXTERNAL_TABLE;
raise notice 'exc v_directory, %',v_directory;
raise notice 'exc pi_filename, %',pi_filename;
raise notice 'exc v_region, %',v_region;
perform aws_s3.query_export_to_s3('SELECT * FROM error_table WHERE file_name = '''||pi_filename||'''',
aws_commons.create_s3_uri(v_directory, pi_filename||'.bad', v_region),
options :='FORmat csv, header, delimiter $$,$$'
);
RETURN null;
END;
$function$
```
# Migrate function-based indexes from Oracle to PostgreSQL
*Veeranjaneyulu Grandhi and Navakanth Talluri, Amazon Web Services*
## Summary
Indexes are a common way to enhance database performance. An index allows the database server to find and retrieve specific rows much faster than it could without an index. But indexes also add overhead to the database system as a whole, so they should be used sensibly. Function-based indexes, which are based on a function or expression, can involve multiple columns and mathematical expressions. A function-based index improves the performance of queries that use the index expression.
Natively, PostgreSQL doesn't support creating function-based indexes using functions that have volatility defined as stable. However, you can create similar functions with volatility as `IMMUTABLE` and use them in index creation.
An `IMMUTABLE` function cannot modify the database, and it’s guaranteed to return the same results given the same arguments forever. This category allows the optimizer to pre-evaluate the function when a query calls it with constant arguments.
This pattern helps in migrating the Oracle function-based indexes when used with functions such as `to_char`, `to_date`, and `to_number` to the PostgreSQL equivalent.
## Prerequisites and limitations
**Prerequisites **
+ An active Amazon Web Services (AWS) account
+ A source Oracle database instance with the listener service set up and running
+ Familiarity with PostgreSQL databases
**Limitations**
+ Database size limit is 64 TB.
+ Functions used in index creation must be IMMUTABLE.
**Product versions**
+ All Oracle database editions for versions 11g (versions 11.2.0.3.v1 and later) and up to 12.2, and 18c
+ PostgreSQL versions 9.6 and later
## Architecture
**Source technology stack**
+ An Oracle database on premises or on an Amazon Elastic Compute Cloud (Amazon EC2) instance, or an Amazon RDS for Oracle DB instance
**Target technology stack**
+ Any PostgreSQL engine
## Tools
+ **pgAdmin 4** is an open source management tool for Postgres. The pgAdmin 4 tool provides a graphical interface for creating, maintaining, and using database objects.
+ **Oracle SQL Developer** is an integrated development environment (IDE) for developing and managing Oracle Database in both traditional and cloud deployments.
## Epics
### Create a function-based index using a default function
| Task | Description | Skills required |
| --- | --- | --- |
| Create a function-based index on a column using the to\$1char function. | Use the following code to create the function-based index.
postgres=# create table funcindex( col1 timestamp without time zone); CREATE TABLE postgres=# insert into funcindex values (now()); INSERT 0 1 postgres=# select * from funcindex; col1 ---------------------------- 2022-08-09 16:00:57.77414 (1 rows)
postgres=# create index funcindex_idx on funcindex(to_char(col1,'DD-MM-YYYY HH24:MI:SS')); ERROR: functions in index expression must be marked IMMUTABLE
PostgreSQL doesn’t allow creating a function-based index without the `IMMUTABLE` clause. | DBA, App developer |
| Check the volatility of the function. | To check the function volatility, use the code in the *Additional information* section. | DBA |
### Create function-based indexes using a wrapper function
| Task | Description | Skills required |
| --- | --- | --- |
| Create a wrapper function. | To create a wrapper function, use the code in the *Additional information section*. | PostgreSQL developer |
| Create an index by using the wrapper function. | Use the code in the *Additional information* section to create a user-defined function with the keyword `IMMUTABLE` in the same schema as the application, and refer to it in the index-creation script.If a user-defined function is created in a common schema (from the previous example), update the `search_path` as shown.
ALTER ROLE set search_path=$user, COMMON;
| DBA, PostgreSQL developer |
### Validate index creation
| Task | Description | Skills required |
| --- | --- | --- |
| Validate index creation. | Validate that the index needs to be created, based on query access patterns. | DBA |
| Validate that the index can be used. | To check whether the function-based index is picked up by the PostgreSQL Optimizer, run an SQL statement using explain or explain analyze. Use the code in the *Additional information* section. If possible, gather the table statistics as well.If you notice the explain plan, PostgreSQL optimizer has chosen a function-based index because of the predicate condition. | DBA |
## Related resources
+ [Function-based indexes](https://docs.oracle.com/cd/E11882_01/appdev.112/e41502/adfns_indexes.htm#ADFNS00505) (Oracle documentation)
+ [Indexes on Expressions](https://www.postgresql.org/docs/9.4/indexes-expressional.html) (PostgreSQL documentation)
+ [PostgreSQL volatility](https://www.postgresql.org/docs/current/xfunc-volatility.html) (PostgreSQL documentation)
+ [PostgreSQL search\$1path](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) (PostgreSQL documentation)
+ [Oracle Database 19c to Amazon Aurora PostgreSQL Migration Playbook](https://docs.aws.amazon.com/dms/latest/oracle-to-aurora-postgresql-migration-playbook/chap-oracle-aurora-pg.html)
## Additional information
**Create a wrapper function**
```
CREATE OR REPLACE FUNCTION myschema.to_char(var1 timestamp without time zone, var2 varchar) RETURNS varchar AS $BODY$ select to_char(var1, 'YYYYMMDD'); $BODY$ LANGUAGE sql IMMUTABLE;
```
**Create an index by using the wrapper function**
```
postgres=# create function common.to_char(var1 timestamp without time zone, var2 varchar) RETURNS varchar AS $BODY$ select to_char(var1, 'YYYYMMDD'); $BODY$ LANGUAGE sql IMMUTABLE;
CREATE FUNCTION
postgres=# create index funcindex_idx on funcindex(common.to_char(col1,'DD-MM-YYYY HH24:MI:SS'));
CREATE INDEX
```
**Check the volatility of the function**
```
SELECT DISTINCT p.proname as "Name",p.provolatile as "volatility" FROM pg_catalog.pg_proc p
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace
LEFT JOIN pg_catalog.pg_language l ON l.oid = p.prolang
WHERE n.nspname OPERATOR(pg_catalog.~) '^(pg_catalog)$' COLLATE pg_catalog.default AND p.proname='to_char'GROUP BY p.proname,p.provolatile
ORDER BY 1;
```
**Validate that the index can be used**
```
explain analyze
postgres=# explain select col1 from funcindex where common.to_char(col1,'DD-MM-YYYY HH24:MI:SS') = '09-08-2022 16:00:57';
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Index Scan using funcindex_idx on funcindex (cost=0.42..8.44 rows=1 width=8)
Index Cond: ((common.to_char(col1, 'DD-MM-YYYY HH24:MI:SS'::character varying))::text = '09-08-2022 16:00:57'::text)
(2 rows)
```
# Migrate Oracle native functions to PostgreSQL using extensions
*Pinesh Singal, Amazon Web Services*
## Summary
This migration pattern provides step-by-step guidance for migrating an Amazon Relational Database Service (Amazon RDS) for Oracle database instance to an Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition database by modifying the `aws_oracle_ext` and `orafce` extensions to PostgreSQL (`psql`) native built-in code. This will save processing time.
The pattern describes an offline manual migration strategy with no downtime for a multi-terabyte Oracle source database with a high number of transactions.
The migration process uses AWS Schema Conversion Tool (AWS SCT) with the `aws_oracle_ext` and `orafce` extensions to convert an Amazon RDS for Oracle database schema to an Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database schema. Then the code is manually changed to PostgreSQL supported native `psql` built-in code. This is because the extension calls impact code processing on the PostgreSQL database server, and not all the extension code is fully complaint or compatible with PostgreSQL code.
This pattern primarily focuses on manually migrating SQL codes using AWS SCT and the extensions `aws_oracle_ext` and `orafce`. You convert the already used extensions into native PostgreSQL (`psql`) built-ins. Then you remove all references to the extensions and convert the codes accordingly.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ Operating system (Windows or Mac) or Amazon EC2 instance (up and running)
+ Orafce
**Limitations **
Not all Oracle functions using `aws_oracle_ext` or `orafce` extensions can be converted to native PostgreSQL functions. It might need manual rework so as to compile it with PostgreSQL libraries.
One drawback of using AWS SCT extensions is its slow performance in running and fetching the results. Its cost can be understood from simple [PostgreSQL EXPLAIN plan](https://www.postgresql.org/docs/current/sql-explain.html) (execution plan of a statement) on the Oracle `SYSDATE` function migration to the PostgreSQL `NOW()` function between all three codes (`aws_oracle_ext`, `orafce`, and `psql` default), as explained in the *Performance comparison check* section in the attached document.
**Product versions**
+ **Source: **Amazon RDS for Oracle database 10.2 and later (for 10.x), 11g (11.2.0.3.v1 and later) and up to 12.2, 18c, and 19c (and later) for Enterprise Edition, Standard Edition, Standard Edition 1, and Standard Edition 2
+ **Target**: Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database 9.4 and later (for 9.x), 10.x, 11.x, 12.x, 13.x, and 14.x (and later)
+ **AWS SCT**: Latest version (this pattern was tested with 1.0.632)
+ **Orafce**: Latest version (this pattern was tested with 3.9.0)
## Architecture
**Source technology stack **
+ An Amazon RDS for Oracle database instance with version 12.1.0.2.v18
**Target technology stack **
+ An Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database instance with version 11.5
**Database migration architecture**
The following diagram represents the database migration architecture between the source Oracle and target PostgreSQL databases. The architecture involves AWS Cloud, a virtual private cloud (VPC), Availability Zones, a private subnet, an Amazon RDS for Oracle database, AWS SCT, an Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database, extensions for Oracle (`aws_oracle_ext` and `orafce`), and structured query language (SQL) files.
![\[The process is explained in the following list.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/158847bb-27ef-4915-a9ca-7d87073792c1/images/234b824a-bfe5-4ef0-9fa7-8401370b92a5.png)
1. Launch Amazon RDS for Oracle DB instance (source DB).
1. Use AWS SCT with the `aws_oracle_ext` and `orafce` extension packs to convert the source code from Oracle to PostreSQL.
1. The conversion produces PostgreSQL-supported migrated .sql files.
1. Manually convert the non-converted Oracle extension codes to PostgreSQL (`psql`) codes.
1. The manual conversion produces PostgreSQL-supported converted .sql files.
1. Run these .sql files on your Amazon RDS for PostgreSQL DB instance (target DB).
## Tools
**Tools**
*AWS services*
+ [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) - AWS Schema Conversion Tool (AWS SCT) converts your existing database schema from one database engine to another. You can convert relational Online Transactional Processing (OLTP) schema, or data warehouse schema. Your converted schema is suitable for an Amazon RDS for MySQL DB instance, an Amazon Aurora DB cluster, an Amazon RDS for PostgreSQL DB instance, or an Amazon Redshift cluster. The converted schema can also be used with a database on an Amazon EC2 instance or stored as data in an Amazon S3 bucket.
AWS SCT provides a project-based user interface to automatically convert the database schema of your source database into a format compatible with your target Amazon RDS instance.
You can use AWS SCT to do migration from an Oracle source database to any of the targets listed preceding. Using AWS SCT, you can export the source database object definitions such as schema, views, stored procedures, and functions.
You can use AWS SCT to convert data from Oracle to Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition.
In this pattern, you use AWS SCT to convert and migrate Oracle code into PostgreSQL using the extensions `aws_oracle_ext` and `orafce`, and manually migrating the extension codes into `psql` default or native built-in code.
+ The [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_ExtensionPack.html) extension pack is an add-on module that emulates functions present in the source database that are required when converting objects to the target database. Before you can install the AWS SCT extension pack, you need to convert your database schema.
When you convert your database or data warehouse schema, AWS SCT adds an additional schema to your target database. This schema implements SQL system functions of the source database that are required when writing your converted schema to your target database. This additional schema is called the extension pack schema.
The extension pack schema for OLTP databases is named according to the source database. For Oracle databases, the extension pack schema is `AWS_ORACLE_EXT`.
*Other tools*
+ [Orafce](https://github.com/orafce/orafce) – Orafce is a module that implements Oracle compatible functions, data types, and packages. It’s an open-source tool with a Berkeley Source Distribution (BSD) license so that anyone can use it. The `orafce` module is useful for migrating from Oracle to PostgreSQL because it has many Oracle functions implemented in PostgreSQL.
**Code**
For a list of all commonly used and migrated code from Oracle to PostgreSQL to avoid AWS SCT extension code usage, see the attached document.
## Epics
### Configure the Amazon RDS for Oracle source database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Oracle database instance. | Create an Amazon RDS for Oracle or Aurora PostgreSQL-Compatible database instance from the Amazon RDS console. | General AWS, DBA |
| Configure the security groups. | Configure inbound and outbound security groups. | General AWS |
| Create the database. | Create the Oracle database with needed users and schemas. | General AWS, DBA |
| Create the objects. | Create objects and insert data in schema. | DBA |
### Configure the Amazon RDS for PostgreSQL target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the PostgreSQL database instance. | Create an Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL database instance from the Amazon RDS console. | General AWS, DBA |
| Configure the security groups. | Configure inbound and outbound security groups. | General AWS |
| Create the database. | Create the PostgreSQL database with needed users and schemas. | General AWS, DBA |
| Validate the extensions. | Make sure that `aws_oracle_ext` and `orafce` are installed and configured correctly in the PostgreSQL database. | DBA |
| Verify that the PostgreSQL database is available. | Make sure that the PostgreSQL database is up and running. | DBA |
### Migrate the Oracle schema into PostgreSQL using AWS SCT and the extensions
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS SCT. | Install the latest version of AWS SCT. | DBA |
| Configure AWS SCT. | Configure AWS SCT with Java Database Connectivity (JDBC) drivers for Oracle (`ojdbc8.jar`) and PostgreSQL (`postgresql-42.2.5.jar`). | DBA |
| Enable the AWS SCT extension pack or template. | Under AWS SCT **Project Settings**, enable built-in function implementation with the `aws_oracle_ext` and `orafce` extensions for the Oracle database schema. | DBA |
| Convert the schema. | In AWS SCT, choose **Convert Schema** to convert the schema from Oracle to PostgreSQL and generate the .sql files. | DBA |
### Convert AWS SCT extension code to psql code
| Task | Description | Skills required |
| --- | --- | --- |
| Manually convert the code. | Manually convert each line of extension-supported code into `psql` default built-in code, as detailed in the attached document. For example, change `AWS_ORACLE_EXT.SYSDATE()` or `ORACLE.SYSDATE()` to `NOW()`. | DBA |
| Validate the code | (Optional) Validate each line of code by temporary running it in the PostgreSQL database. | DBA |
| Create objects in the PostgreSQL database. | To create objects in the PostgreSQL database, run the .sql files that were generated by AWS SCT and modified in the previous two steps. | DBA |
## Related resources
+ Database
+ [Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html)
+ [PostgreSQL on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html)
+ [Working with Amazon Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html)
+ [PostgreSQL EXPLAIN plan](https://www.postgresql.org/docs/current/sql-explain.html)
+ AWS SCT
+ [AWS Schema Conversion Tool Overview](https://aws.amazon.com/dms/schema-conversion-tool/)
+ [AWS SCT User Guide](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Using the AWS SCT user interface](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html)
+ [Using Oracle Database as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html)
+ Extensions for AWS SCT
+ [Using the AWS SCT extension pack](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_ExtensionPack.html)
+ [Oracle functionality (en)](https://postgres.cz/wiki/Oracle_functionality_(en))
+ [PGXN orafce](https://pgxn.org/dist/orafce/)
+ [GitHub orafce](https://github.com/orafce/orafce)
## Additional information
For more information, follow the detailed commands, with syntax and examples, for manually converting code in the attached document.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/158847bb-27ef-4915-a9ca-7d87073792c1/attachments/attachment.zip)
# Migrate a Db2 database from Amazon EC2 to Aurora MySQL-Compatible by using AWS DMS
*Pinesh Singal, Amazon Web Services*
## Summary
After you migrate your [IBM Db2 for LUW database](https://www.ibm.com/docs/en/db2/11.5?topic=federation) to [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/), consider re-architecting the database by moving to an Amazon Web Services (AWS) cloud-native database. This pattern covers migrating an IBM [Db2](https://www.ibm.com/docs/en/db2/11.5) for LUW database running on an [Amazon](https://docs.aws.amazon.com/ec2/) EC2 instance to an [Amazon Aurora MySQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMySQL.html) database on AWS.
The pattern describes an online migration strategy with minimal downtime for a multi-terabyte Db2 source database with a high number of transactions.
This pattern uses [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) to convert the Db2 database schema to an Aurora MySQL-Compatible schema. Then the pattern uses [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) to migrate data from the Db2 database to the Aurora MySQL-Compatible database. Manual conversions will be required for the code that isn’t converted by AWS SCT.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account with a virtual private cloud (VPC)
+ AWS SCT
+ AWS DMS
**Product versions**
+ AWS SCT latest version
+ Db2 for Linux version 11.1.4.4 and later
## Architecture
**Source technology stack**
+ DB2/Linux x86-64 bit mounted on an EC2 instance
** Target technology stack**
+ An Amazon Aurora MySQL-Compatible Edition database instance
**Source and target architecture**
The following diagram shows the data migration architecture between the source Db2 and target Aurora MySQL-Compatible databases. The architecture on the AWS Cloud includes a virtual private cloud (VPC) (Virtual Private Cloud), an Availability Zone, a public subnet for the Db2 instance and the AWS DMS replication instance, and a private subnet for the Aurora MySQL-Compatible database.
![\[Architecture of data migration between source Db2 and target Aurora MySQL-Compatible databases.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5abfccc4-148c-4794-8d80-e3c122679125/images/f30664f8-2d6a-4448-8d5c-cff3988a52c7.png)
## Tools
**AWS services**
+ [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL.
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database. AWS SCT supports as a source IBM Db2 for LUW versions 9.1, 9.5, 9.7, 10.1, 10.5, 11.1, and 11.5.
## Best practices
For best practices, see [Best practices for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html).
## Epics
### Configure the source IBM Db2 database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the IBM Db2 database on Amazon EC2. | You can create an IBM Db2 database on an EC2 instance by using an Amazon Machine Image (AMI) from AWS Marketplace or by installing Db2 software on an EC2 instance.Launch an EC2 instance by selecting an AMI for IBM Db2 (for example, [IBM Db2 v11.5.7 RHEL 7.9](https://aws.amazon.com/marketplace/pp/prodview-aclrjj4hq2ols?sr=0-1&ref_=beagle&applicationId=AWS-EC2-Console)), which is similar to an on-premises database. | DBA, General AWS |
| Configure security groups. | Configure the VPC security group inbound rules for SSH (Secure Shell) and TCP with port 22 and 50000, respectively. | General AWS |
| Create the database instance. | Create a new instance (user) and database (schema), or use the default `db2inst1` instance and sample database.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | DBA |
| Confirm that the Db2 DB instance is available. | To confirm that the Db2 database instance is up and running, use the `Db2pd -` command. | DBA |
### Configure the target Aurora MySQL-Compatible database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Aurora MySQL-Compatible database. | Create an Amazon Aurora with MySQL compatibility Database from AWS RDS service[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | DBA, General AWS |
| Configure security groups. | Configure the VPC security group inbound rules for SSH and TCP connections. | General AWS |
| Confirm that the Aurora database is available. | To make sure that the Aurora MySQL-Compatible database is up and running, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | DBA |
### Configure and run AWS SCT
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS SCT. | Download and install the latest version of [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html) (the current latest version 1.0.628). | General AWS |
| Configure AWS SCT. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | General AWS |
| Create an AWS SCT project. | Create an AWS SCT project and report that uses Db2 for LUW as the source DB engine and Aurora MySQL-Compatible for the target DB engine.To identify the privileges needed to connect to a Db2 for LUW database, see [Using Db2 LUW as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.DB2LUW.html). | General AWS |
| Validate the objects. | Choose **Load schema**, validate the objects. Update any incorrect objects on the target database:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | DBA, General AWS |
### Configure and run AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance. | Sign in to the AWS Management Console, navigate to the AWS DMS service, and create a replication instance with valid settings for the VPC security group that you configured for the source and target databases. | General AWS |
| Create endpoints. | Create the source endpoint for the Db2 database, and create the target endpoint for the Aurora MySQL-Compatible database:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | General AWS |
| Create migration tasks. | Create a single migration task or multiple migration tasks for full load and CDC or Data validation:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | General AWS |
| Plan the production run. | Confirm downtime with stakeholders such as application owners to run AWS DMS in production systems. | Migration lead |
| Run the migration tasks. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | General AWS |
| Validate the data. | Review migration task results and data in the source Db2 and target MySQL databases:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) | DBA |
| Stop migration tasks. | After data validation is successfully completed, stop the validation migration tasks. | General AWS |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| AWS SCT source and target test connections are failing. | Configure JDBC driver versions and VPC security group inbound rules to accept the incoming traffic. |
| The Db2 source endpoint test run fails. | Configure the extra connection setting `CurrentLSN=;`. |
| The AWSDMS task fails to connect to the Db2 source, and the following error is returned.`database is recoverable if either or both of the database configuration parameters LOGARCHMETH1 and LOGARCHMETH2 are set to ON` | To avoid the error, run the following commands:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.html) |
## Related resources
**Amazon EC2**
+ [Amazon EC2](https://aws.amazon.com/ec2/)
+ [Amazon EC2 User Guides](https://docs.aws.amazon.com/ec2/)
**Databases**
+ [IBM Db2 Database](https://www.ibm.com/products/db2-database)
+ [Amazon Aurora](https://aws.amazon.com/rds/aurora/)
+ [Working with Amazon Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMySQL.html)
**AWS SCT**
+ [AWS DMS Schema Conversion](https://aws.amazon.com/dms/schema-conversion-tool/)
+ [AWS Schema Conversion Tool User Guide](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Using the AWS SCT user interface](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html)
+ [Using IBM Db2 LUW as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.DB2LUW.html)
**AWS DMS**
+ [AWS Database Migration Service](https://aws.amazon.com/dms/)
+ [AWS Database Migration Service User Guide](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [Sources for data migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html)
+ [Targets for data migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html)
+ [AWS Database Migration Service and AWS Schema Conversion Tool now support IBM Db2 LUW as a source](https://aws.amazon.com/blogs/database/aws-database-migration-service-and-aws-schema-conversion-tool-now-support-ibm-db2-as-a-source/) (blog post)
+ [Migrating Applications Running Relational Databases to AWS](https://d1.awsstatic.com/whitepapers/Migration/migrating-applications-to-aws.pdf)
# Migrate a Microsoft SQL Server database from Amazon EC2 to Amazon DocumentDB by using AWS DMS
*Umamaheswara Nooka, Amazon Web Services*
## Summary
This pattern describes how to use AWS Database Migration Service (AWS DMS) to migrate a Microsoft SQL Server database hosted on an Amazon Elastic Compute Cloud (Amazon EC2) instance to an Amazon DocumentDB (with MongoDB compatibility) database.
The AWS DMS replication task reads the table structure of the SQL Server database, creates the corresponding collection in Amazon DocumentDB, and performs a full-load migration.
You can also use this pattern to migrate an on-premises SQL Server or an Amazon Relational Database Service (Amazon RDS) for SQL Server DB instance to Amazon DocumentDB. For more information, see the guide [Migrating Microsoft SQL Server databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-sql-server/welcome.html) on the AWS Prescriptive Guidance website.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ An existing SQL Server database on an EC2 instance.
+ Fixed database (**db\$1owner**) role assigned to AWS DMS in the SQL Server database. For more information, see [Database-level roles](https://docs.microsoft.com/en-us/sql/relational-databases/security/authentication-access/database-level-roles?view=sql-server-ver15) in the SQL Server documentation.
+ Familiarity with using the `mongodump`, `mongorestore`, `mongoexport`, and `mongoimport` utilities to [move data in and out of an Amazon DocumentDB cluster](https://docs.aws.amazon.com/documentdb/latest/developerguide/backup_restore-dump_restore_import_export_data.html).
+ [Microsoft SQL Server Management Studio](https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver15), installed and configured.
**Limitations **
+ The cluster size limit in Amazon DocumentDB is 64 TB. For more information, see [Cluster limits](https://docs.aws.amazon.com/documentdb/latest/developerguide/limits.html#limits-cluster) in the Amazon DocumentDB documentation.
+ AWS DMS doesn't support the merging of multiple source tables into a single Amazon DocumentDB collection.
+ If AWS DMS processes any changes from a source table without a primary key, it will ignore large object (LOB) columns in the source table.
## Architecture
**Source technology stack **
+ Amazon EC2
**Target technology stack **
+ Amazon DocumentDB
**Target architecture**
![\[AWS Cloud architecture showing VPC with private DB subnet and components for SQL Server and DocumentDB.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f186220b-5a94-48b2-840d-f04aedf51651/images/00962b85-8b71-49df-b84a-3adcbc9ad3a3.png)
## Tools
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html) – AWS Database Migration Service (AWS DMS) helps you migrate databases easily and securely.
+ [Amazon DocumentDB](https://docs.aws.amazon.com/documentdb/latest/developerguide/get-started-guide.html) – Amazon DocumentDB (with MongoDB compatibility) is a fast, reliable, and fully managed database service.
+ [Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) – Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the AWS Cloud.
+ [Microsoft SQL Server](https://docs.microsoft.com/en-us/sql/sql-server/?view=sql-server-ver15) – SQL Server is a relational database management system.
+ [SQL Server Management Studio (SSMS)](https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-ssms?view=sql-server-ver15) – SSMS is a tool for managing SQL Server, including accessing, configuring, and administering SQL Server components.
## Epics
### Create and configure a VPC
| Task | Description | Skills required |
| --- | --- | --- |
| Create a VPC. | Sign in to the AWS Management Console and open the Amazon VPC console. Create a virtual private cloud (VPC) with an IPv4 CIDR block range. | System administrator |
| Create security groups and network ACLs. | On the Amazon VPC console, create security groups and network access control lists (network ACLs) for your VPC, according to your requirements. You can also use the default settings for these configurations. For more information about this and other stories, see the "Related resources" section. | System administrator |
### Create and configure the Amazon DocumentDB cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon DocumentDB cluster. | Open the Amazon DocumentDB console and choose "Clusters." Choose "Create," and create an Amazon DocumentDB cluster with one instance. Important: Make sure you configure this cluster with your VPC’s security groups. | System administrator |
| Install the mongo shell. | The mongo shell is a command-line utility that you use to connect to and query your Amazon DocumentDB cluster. To install it, run the "/etc/yum.repos.d/mongodb-org-3.6.repo" command to create the repository file. Run the "sudo yum install -y mongodb-org-shell" command to install the mongo shell. To encrypt data in transit, download the public key for Amazon DocumentDB, and then connect to your Amazon DocumentDB instance. For more information about these steps, see the "Related resources" section. | System administrator |
| Create a database in the Amazon DocumentDB cluster. | Run the "use" command with the name of your database to create a database in your Amazon DocumentDB cluster. | System administrator |
### Create and configure the AWS DMS replication instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create the AWS DMS replication instance. | Open the AWS DMS console and choose "Create replication instance." Enter a name and description for your replication task. Choose the instance class, engine version, storage, VPC, Multi-AZ, and make it publicly accessible. Choose the "Advanced" tab to set the network and encryption settings. Specify the maintenance settings, and then choose "Create replication instance." | System administrator |
| Configure the SQL Server database. | Log in to Microsoft SQL Server and add an inbound rule for communication between the source endpoint and the AWS DMS replication instance. Use the replication instance’s private IP address as the source. Important: The replication instance and target endpoint should be on the same VPC. Use an alternative source in the security group if the VPCs are different for the source and replication instances. | System administrator |
### Create and test the source and target endpoints in AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create the source and target database endpoints. | Open the AWS DMS console and choose "Connect source and target database endpoints." Specify the connection information for the source and target databases. If required, choose the "Advanced" tab to set values for "Extra connection attributes." Download and use the certificate bundle in your endpoint configuration. | System administrator |
| Test the endpoint connection. | Choose "Run test" to test the connection. Troubleshoot any error messages by verifying the security group settings and the connections to the AWS DMS replication instance from both the source and target database instances. | System administrator |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Create the AWS DMS migration task. | On the AWS DMS console, choose "Tasks," "Create task." Specify the task options, including the source and destination endpoint names, and replication instance names. Under "Migration type" choose "Migrate existing data," and "Replicate data changes only." Choose "Start task." | System administrator |
| Run the AWS DMS migration task. | Under "Task settings," specify the settings for the table preparation mode, such as "Do nothing," "Drop tables on target," "Truncate," and "Include LOB columns in replication." Set a maximum LOB size that AWS DMS will accept and choose "Enable logging." Leave the "Advanced settings" at their default values and choose "Create task." | System administrator |
| Monitor the migration. | On the AWS DMS console, choose "Tasks" and choose your migration task. Choose "Task monitoring" to monitor your task. The task stops when the full-load migration is complete and cached changes are applied. | System administrator |
### Test and verify the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Connect to the Amazon DocumentDB cluster by using the mongo shell. | Open the Amazon DocumentDB console, choose your cluster under "Clusters." In the "Connectivity and Security" tab, choose "Connect to this cluster with the mongo shell." | System administrator |
| Verify the results of your migration. | Run the "use" command with the name of your database and then run the "show collections" command. Run the "db. .count();" command with the name of your database. If the results match your source database, then your migration was successful. | System administrator |
## Related resources
**Create and configure a VPC **
+ [Create a security group for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#CreatingSecurityGroups)
+ [Create a network ACL](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html)
** **
**Create and configure the Amazon DocumentDB cluster**
+ [Create an Amazon DocumentDB cluster](https://docs.aws.amazon.com/documentdb/latest/developerguide/get-started-guide.html#cloud9-cluster)
+ [Install the mongo shell for Amazon DocumentDB ](https://docs.aws.amazon.com/documentdb/latest/developerguide/get-started-guide.html#cloud9-mongoshell)
+ [Connect to your Amazon DocumentDB cluster](https://docs.aws.amazon.com/documentdb/latest/developerguide/get-started-guide.html#cloud9-connectcluster)
** **
**Create and configure the AWS DMS replication instance **
+ [Use public and private replication instances](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.html#CHAP_ReplicationInstance.PublicPrivate)
** **
**Create and test the source and target endpoints in AWS DMS **
+ [Use Amazon DocumentDB as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/target.docdb.html)
+ [Use a SQL Server database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html)
+ [Use AWS DMS endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.html)
** **
**Migrate data **
+ [Migrate to Amazon DocumentDB](https://docs.aws.amazon.com/documentdb/latest/developerguide/docdb-migration.html)
** **
**Other resources**
+ [Limitations on using SQL Server as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html#CHAP_Source.SQLServer.Limitations)
+ [How to use Amazon DocumentDB to build and manage applications at scale](https://aws.amazon.com/blogs/database/how-to-use-amazon-documentdb-with-mongodb-compatibility-to-build-and-manage-applications-at-scale/)
# Migrate an on-premises ThoughtSpot Falcon database to Amazon Redshift
*Battulga Purevragchaa and Antony Prasad Thevaraj, Amazon Web Services*
## Summary
On-premises data warehouses require significant administration time and resources, particularly for large datasets. The financial cost of building, maintaining, and growing these warehouses is also very high. To help manage costs, keep extract, transform, and load (ETL) complexity low, and deliver performance as your data grows, you must constantly choose which data to load and which data to archive.
By migrating your on-premises [ThoughtSpot Falcon databases](https://docs.thoughtspot.com/software/latest/data-caching) to the Amazon Web Services (AWS) Cloud, you can access cloud-based data lakes and data warehouses that increase your business agility, security, and application reliability, in addition to reducing your overall infrastructure costs. Amazon Redshift helps to significantly lower the cost and operational overhead of a data warehouse. You can also use Amazon Redshift Spectrum to analyze large amounts of data in its native format without data loading.
This pattern describes the steps and process for migrating a ThoughtSpot Falcon database from an on-premises data center to an Amazon Redshift database on the AWS Cloud.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A ThoughtSpot Falcon database hosted in an on-premises data center
**Product versions**
+ ThoughtSpot version 7.0.1
## Architecture
![\[Migrating a ThoughtSpot Falcon database from an on-premises data center to Amazon Redshift.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b0ca29f4-b269-4b57-b386-738693a6b334/images/2b483990-1f30-439c-ba13-dc0cb0650360.png)
The diagram shows the following workflow:
1. Data is hosted in an on-premises relational database.
1. AWS Schema Conversion Tool (AWS SCT) converts the data definition language (DDL) that is compatible with Amazon Redshift.
1. After the tables are created, you can migrate the data by using AWS Database Migration Service (AWS DMS).
1. The data is loaded into Amazon Redshift.
1. The data is stored in Amazon Simple Storage Service (Amazon S3) if you use Redshift Spectrum or already host the data in Amazon S3.
## Tools
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) – AWS Data Migration Service (AWS DMS) helps you quickly and securely migrate databases to AWS.
+ [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html) – Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools.
+ [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) – AWS Schema Conversion Tool (AWS SCT) converts your existing database schema from one database engine to another.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Identify the appropriate Amazon Redshift configuration. | Identify the appropriate Amazon Redshift cluster configuration based on your requirements and data volume. For more information, see [Amazon Redshift clusters](https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-clusters.html) in the Amazon Redshift documentation. | DBA |
| Research Amazon Redshift to evaluate if it meets your requirements. | Use the [Amazon Redshift FAQs](https://aws.amazon.com/redshift/faqs/) to understand and evaluate whether Amazon Redshift meets your requirements. | DBA |
### Prepare the target Amazon Redshift cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon Redshift cluster. | Sign in to the AWS Management Console, open the Amazon Redshift console, and then create an Amazon Redshift cluster in a virtual private cloud (VPC). For more information, see [Creating a cluster in a VPC](https://docs.aws.amazon.com/redshift/latest/mgmt/getting-started-cluster-in-vpc.html) in the Amazon Redshift documentation. | DBA |
| Conduct a PoC for your Amazon Redshift database design. | Follow Amazon Redshift best practices by conducting a proof of concept (PoC) for your database design. For more information, see [Conducting a proof of concept for Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/proof-of-concept-playbook.html) in the Amazon Redshift documentation. | DBA |
| Create database users. | Create the users in your Amazon Redshift database and grant the appropriate roles for access to the schema and tables. For more information, see [Grant access privileges for a user or user group](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) in the Amazon Redshift documentation. | DBA |
| Apply configuration settings to the target database. | Apply configuration settings to the Amazon Redshift database according to your requirements. For more information about enabling database, session, and server-level parameters, see the [Configuration reference](https://docs.aws.amazon.com/redshift/latest/dg/cm_chap_ConfigurationRef.html) in the Amazon Redshift documentation. | DBA |
### Create objects in the Amazon Redshift cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Manually create tables with DDL in Amazon Redshift. | (Optional) If you use AWS SCT, the tables are automatically created. However, if there are failures when replicating DDLs, you have to manually create the tables | DBA |
| Create external tables for Redshift Spectrum. | Create an external table with an external schema for Amazon Redshift Spectrum. To create external tables, you must be the owner of the external schema or a [database superuser](https://docs.aws.amazon.com/redshift/latest/dg/r_superusers.html). For more information, see [Creating external tables for Amazon Redshift Spectrum](https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-tables.html) in the Amazon Redshift documentation. | DBA |
### Migrate data using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Use AWS DMS to migrate the data. | After you create the DDL of the tables in the Amazon Redshift database, migrate your data to Amazon Redshift by using AWS DMS.For detailed steps and instructions, see [Using an Amazon Redshift database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Redshift.html) in the AWS DMS documentation. | DBA |
| Use the COPY command to load the data. | Use the Amazon Redshift `COPY` command to load the data from Amazon S3 to Amazon Redshift.For more information, see [Using the COPY command to load from Amazon S3](https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html) in the Amazon Redshift documentation. | DBA |
### Validate the Amazon Redshift cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target records. | Validate the table count for the source and target records that were loaded from your source system. | DBA |
| Implement Amazon Redshift best practices for performance tuning. | Implement Amazon Redshift best practices for table and database design. For more information, see the blog post [Top 10 performance tuning techniques for Amazon Redshift](https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-techniques-for-amazon-redshift/). | DBA |
| Optimize query performance. | Amazon Redshift uses SQL-based queries to interact with data and objects in the system. Data manipulation language (DML) is the subset of SQL that you can use to view, add, change, and delete data. DDL is the subset of SQL that you use to add, change, and delete database objects such as tables and views.For more information, see [Tuning query performance](https://docs.aws.amazon.com/redshift/latest/dg/c-optimizing-query-performance.html) in the Amazon Redshift documentation. | DBA |
| Implement WLM. | You can use workload management (WLM) to define multiple query queues and route queries to appropriate queues at runtime.For more information, see [Implementing workload management](https://docs.aws.amazon.com/redshift/latest/dg/cm-c-implementing-workload-management.html) in the Amazon Redshift documentation. | DBA |
| Work with concurrency scaling. | By using the Concurrency Scaling feature, you can support virtually unlimited concurrent users and concurrent queries, with consistently fast query performance.For more information, see [Working with concurrency scaling](https://docs.aws.amazon.com/redshift/latest/dg/concurrency-scaling.html) in the Amazon Redshift documentation. | DBA |
| Use Amazon Redshift best practices for table design. | When you plan your database, certain important table design decisions can strongly influence overall query performance.For more information about choosing the most appropriate table design option, see [Amazon Redshift best practices for designing tables](https://docs.aws.amazon.com/redshift/latest/dg/c_designing-tables-best-practices.html) in the Amazon Redshift documentation. | DBA |
| Create materialized views in Amazon Redshift. | A materialized view contains a precomputed results set based on an SQL query over one or more base tables. You can issue `SELECT` statements to query a materialized view in the same way that you query other tables or views in the database.For more information, see [Creating materialized views in Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) in the Amazon Redshift documentation. | DBA |
| Define joins between the tables. | To search more than one table at the same time in ThoughtSpot, you must define joins between the tables by specifying columns that contain matching data across two tables. These columns represent the `primary key` and `foreign key` of the join.You can define them by using the `ALTER TABLE` command in Amazon Redshift or ThoughtSpot. For more information, see [ALTER TABLE](https://docs.aws.amazon.com/redshift/latest/dg/r_ALTER_TABLE.html) in the Amazon Redshift documentation. | DBA |
### Set up ThoughtSpot connection to Amazon Redshift
| Task | Description | Skills required |
| --- | --- | --- |
| Add an Amazon Redshift connection. | Add an Amazon Redshift connection to your on-premises ThoughtSpot Falcon database.For more information, see [Add an Amazon Redshift connection](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-add-connection.html) in the ThoughtSpot documentation. | DBA |
| Edit the Amazon Redshift connection. | You can edit the Amazon Redshift connection to add tables and columns.For more information, see [Edit an Amazon Redshift connection](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-edit-connection.html) in the ThoughtSpot documentation. | DBA |
| Remap the Amazon Redshift connection. | Modify the connection parameters by editing the source mapping .yaml file that was created when you added the Amazon Redshift connection. For example, you can remap the existing table or column to a different table or column in an existing database connection. ThoughtSpot recommends that you check the dependencies before and after you remap a table or column in a connection to ensure that they display as required.For more information, see [Remap an Amazon Redshift connection](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-remap-connection.html) in the ThoughtSpot documentation. | DBA |
| Delete a table from the Amazon Redshift connection. | (Optional) If you attempt to remove a table in an Amazon Redshift connection, ThoughtSpot checks for dependencies and shows a list of dependent objects. You can choose the listed objects to delete them or remove the dependency. You can then remove the table.For more information, see [Delete a table from an Amazon Redshift connection](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-delete-table.html) in the ThoughtSpot documentation. | DBA |
| Delete a table with dependent objects from an Amazon Redshift connection. | (Optional) If you try to delete a table with dependent objects, the operation is blocked. A `Cannot delete` window appears, with a list of links to dependent objects. When all the dependencies are removed, you can then delete the tableFor more information, see [Delete a table with dependent objects from an Amazon Redshift connection](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-delete-table-dependencies.html) in the ThoughtSpot documentation. | DBA |
| Delete an Amazon Redshift connection. | (Optional) Because a connection can be used in multiple data sources or visualizations, you must delete all of the sources and tasks that use that connection before you can delete the Amazon Redshift connection.For more information, see [Delete an Amazon Redshift connection](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-delete-connection.html) in the ThoughtSpot documentation. | DBA |
| Check connection reference for Amazon Redshift. | Make sure that you provide the required information for your Amazon Redshift connection by using the [Connection reference](https://cloud-docs.thoughtspot.com/admin/ts-cloud/ts-cloud-embrace-redshift-connection-reference.html) in the ThoughtSpot documentation. | DBA |
## Additional information
+ [AI-driven analytics at any scale with ThoughtSpot and Amazon Redshift](https://aws.amazon.com/blogs/apn/ai-driven-analytics-at-any-scale-with-thoughtspot-and-amazon-redshift/)
+ [Amazon Redshift pricing](https://aws.amazon.com/redshift/pricing/)
+ [Getting started with AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_GettingStarted.html)
+ [Getting started with Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html)
+ [Using data extraction agents](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/agents.html)
+ [Chick-fil-A improves speed to insight with ThoughtSpot and AWS](https://www.thoughtspot.com/sites/default/files/pdf/ThoughtSpot-Chick-fil-A-AWS-Case-Study.pdf)
# Migrate from Oracle Database to Amazon RDS for PostgreSQL by using Oracle GoldenGate
*Dhairya Jindani, Sindhusha Paturu, and Rajeshkumar Sabankar, Amazon Web Services*
## Summary
This pattern shows how to migrate an Oracle database to Amazon Relational Database Service (Amazon RDS) for PostgreSQL by using Oracle Cloud Infrastructure (OCI) GoldenGate.
By using Oracle GoldenGate, you can replicate data between your source database and one or more destination databases with minimal downtime.
**Note**
The source Oracle database can be either on-premises or on an Amazon Elastic Compute Cloud (Amazon EC2) instance. You can use a similar procedure when using on-premises replication tools.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An Oracle GoldenGate license
+ Java Database Connectivity (JDBC) driver to connect to the PostgreSQL database
+ Schema and tables created with the [AWS Schema Conversion Tool (AWS SCT)](https://aws.amazon.com/dms/schema-conversion-tool/) on the target Amazon RDS for PostgreSQL database
**Limitations**
+ Oracle GoldenGate can replicate existing table data (initial load) and ongoing changes (change data capture) only
**Product versions**
+ Oracle Database Enterprise Edition 10g or newer versions
+ Oracle GoldenGate 12.2.0.1.1 for Oracle or newer versions
+ Oracle GoldenGate 12.2.0.1.1 for PostgreSQL or newer versions
## Architecture
The following diagram shows an example workflow for migrating an Oracle database to Amazon RDS for PostgreSQL by using Oracle GoldenGate:
![\[Migration workflow from on-premises Oracle database to Amazon RDS for PostgreSQL.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/384f0eaf-8582-474a-a7f4-ec1048a4feb3/images/de541887-0d5f-4a9a-b136-ce2599355cb8.png)
The diagram shows the following workflow:
1. The Oracle GoldenGate [Extract process](https://docs.oracle.com/goldengate/c1230/gg-winux/GGCON/processes-and-terminology.htm#GUID-6419F3A9-71EC-4D14-9C41-3BAA1E3CA19C) runs against the source database to extract data.
1. The Oracle GoldenGate [Replicat process](https://docs.oracle.com/goldengate/c1230/gg-winux/GGCON/processes-and-terminology.htm#GUID-5EF0326C-9058-4C40-8925-98A223388C95) delivers the extracted data to the target Amazon RDS for PostgreSQL database.
## Tools
+ [Oracle GoldenGate](https://www.oracle.com/integration/goldengate/#:~:text=OCI%20GoldenGate%20is%20a%20real,in%20the%20Oracle%20Cloud%20Infrastructure.) helps you design, run, orchestrate, and monitor your data replication and stream data processing solutions in the Oracle Cloud Infrastructure.
+ [Amazon Relational Database Service (Amazon RDS) for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
## Epics
### Download and install Oracle GoldenGate
| Task | Description | Skills required |
| --- | --- | --- |
| Download Oracle GoldenGate. | Download the following versions of Oracle GoldenGate:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.html)To download the software, see [Oracle GoldenGate Downloads](https://www.oracle.com/middleware/technologies/goldengate-downloads.html) on the Oracle website. | DBA |
| Install Oracle GoldenGate for Oracle on the source Oracle Database server. | For instructions, see the [Oracle GoldenGate documentation](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/toc.htm). | DBA |
| Install Oracle GoldenGate for PostgreSQL database on the Amazon EC2 instance. | For instructions, see the [Oracle GoldenGate documentation](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/toc.htm). | DBA |
### Configure Oracle GoldenGate on the source and target databases
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Oracle GoldenGate for Oracle Database on the source database. | For instructions, see the [Oracle GoldenGate documentation](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/toc.htm).Make sure that you configure the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.html) | DBA |
| Set up Oracle GoldenGate for PostgreSQL on the target database. | For instructions, see [Part VI Using Oracle GoldenGate for PostgreSQL](https://docs.oracle.com/en/middleware/goldengate/core/19.1/gghdb/using-oracle-goldengate-postgresql.html) on the Oracle website.Make sure that you configure the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.html) | DBA |
### Configure the data capture
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the Extract process in the source database. | In the source Oracle Database, create an extract file to extract data.For instructions, see [ADD EXTRACT](https://docs.oracle.com/goldengate/1212/gg-winux/GWURF/ggsci_commands006.htm#GWURF122) in the Oracle documentation.The extract file includes the creation of the extract parameter file and trail file directory. | DBA |
| Set up a data pump to transfer the trail file from the source to the target database. | Create an EXTRACT parameter file and trail file directory by following the instructions in [PARFILE](https://docs.oracle.com/database/121/SUTIL/GUID-7A045C82-5993-44EB-AFAD-B7D39C34BCCD.htm#SUTIL859) in *Database Utilities* on the Oracle website.For more information, see [What is a Trail?](https://docs.oracle.com/goldengate/c1230/gg-winux/GGCON/processes-and-terminology.htm#GUID-88674F53-1E07-4C00-9868-598F82D7113C) in *Fusion Middleware Understanding Oracle GoldenGate* on the Oracle website. | DBA |
| Set up replication on the Amazon EC2 instance. | Create a replication parameter file and trail file directory.For more information about creating replication parameter files, see section [3.5 Validating a parameter file](https://docs.oracle.com/en/middleware/goldengate/core/21.3/admin/using-oracle-goldengate-parameter-files.html#GUID-1E32A9AD-25DB-4243-93CD-E643E7116215) in the Oracle Database documentation.For more information about creating a trail file directory, see [Creating a trail](https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/creating-trail.html) in the Oracle Cloud documentation.Make sure that you add a checkpoint table entry in the GLOBALS file at the target.For more information, see [What is a Replicat?](https://docs.oracle.com/goldengate/c1230/gg-winux/GGCON/processes-and-terminology.htm#GGCON-GUID-5EF0326C-9058-4C40-8925-98A223388C95) in *Fusion Middleware Understanding Oracle GoldenGate* on the Oracle website. | DBA |
### Configure the data replication
| Task | Description | Skills required |
| --- | --- | --- |
| In the source database, create a parameter file to extract data for the initial load. | Follow the instructions in [Creating a parameter file in GGSCI](https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/using-oracle-goldengate-parameter-files.html#GUID-5C49C522-8B28-4E4B-908D-66A33717CE6C) in the Oracle Cloud documentation.Make sure that the Manager is running on the target. | DBA |
| In the target database, create a parameter file to replicate data for the initial load. | Follow the instructions in [Creating a parameter file in GGSCI](https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/using-oracle-goldengate-parameter-files.html#GUID-5C49C522-8B28-4E4B-908D-66A33717CE6C) in the Oracle Cloud documentation.Make sure that you add and start the Replicat process. | DBA |
### Cut over to the Amazon RDS for PostgreSQL database
| Task | Description | Skills required |
| --- | --- | --- |
| Stop the Replicat process and make sure that the source and target databases are in sync. | Compare row counts between the source and target databases to make sure that the data replication was successful. | DBA |
| Configure data definition language (DDL) support. | Run the DDL script for creating triggers, sequence, synonyms, and referential keys on PostgreSQL.You can use any standard SQL client application to connect to a database in your DB cluster. For example, you can use [pgAdmin](https://www.pgadmin.org/) to connect to your DB instance. | DBA |
## Related resources
+ [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) (*Amazon RDS User Guide*)
+ [Amazon EC2 documentation](https://docs.aws.amazon.com/ec2/)
+ [Oracle GoldenGate supported processing methods and databases](https://docs.oracle.com/goldengate/1212/gg-winux/GWUAD/wu_about_gg.htm#GWUAD112) (Oracle documentation)
# Migrate an Oracle partitioned table to PostgreSQL by using AWS DMS
*Saurav Mishra and Eduardo Valentim, Amazon Web Services*
## Summary
This pattern describes how to speed up loading a partitioned table from Oracle to PostgreSQL by using AWS Database Migration Service (AWS DMS), which doesn't support native partitioning. The target PostgreSQL database can be installed on Amazon Elastic Compute Cloud (Amazon EC2), or it can be an Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition DB instance.
Uploading a partitioned table includes the following steps:
1. Create a parent table similar to the Oracle partition table, but don't include any partition.
1. Create child tables that will inherit from the parent table that you created in step 1.
1. Create a procedure function and trigger to handle the inserts in the parent table.
However, because the trigger is fired for every insert, the initial load using AWS DMS can be very slow.
To speed up initial loads from Oracle to PostgreSQL 9.0, this pattern creates a separate AWS DMS task for each partition and loads the corresponding child tables. You then create a trigger during cutover.
PostgreSQL version 10 supports native partitioning. However, you might decide to use inherited partitioning in some cases. For more information, see the [Additional information](#migrate-an-oracle-partitioned-table-to-postgresql-by-using-aws-dms-additional) section.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Oracle database with a partitioned table
+ A PostgreSQL database on AWS
**Product versions**
+ PostgreSQL 9.0
## Architecture
**Source technology stack**
+ A partitioned table in Oracle
**Target technology stack**
+ A partitioned table in PostgreSQL (on Amazon EC2, Amazon RDS for PostgreSQL, or Aurora PostgreSQL)
**Target architecture**
![\[Partitioned table data in Oracle moving to an AWS DMS task for each partition, then into PostgreSQL.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7fa2898e-3308-436a-aec8-ab6f680d7bac/images/1b9742ea-a13d-434c-83a7-56686cf76ea0.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
## Epics
### Set up AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create the tables in PostgreSQL. | Create the parent and corresponding child tables in PostgreSQL with the required check conditions for partitions. | DBA |
| Create the AWS DMS task for each partition. | Include the filter condition of the partition in the AWS DMS task. Map the partitions to the corresponding PostgreSQL child tables. | DBA |
| Run the AWS DMS tasks using full load and change data capture (CDC). | Make sure that the `StopTaskCachedChangesApplied` parameter is set to `true` and the `StopTaskCachedChangesNotApplied` parameter is set to `false`. | DBA |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Stop the replication tasks. | Before you stop the tasks, confirm that the source and destination are in sync. | DBA |
| Create a trigger on the parent table. | Because the parent table will receive all insert and update commands, create a trigger that will route these commands to the respective child tables based on the partitioning condition. | DBA |
## Related resources
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [Table Partitioning (PostgreSQL documentation)](https://www.postgresql.org/docs/10/ddl-partitioning.html)
## Additional information
Although PostgreSQL version 10 supports native partitioning, you might decide to use inherited partitioning for the following use cases:
+ Partitioning enforces a rule that all partitions must have the same set of columns as the parent, but table inheritance supports children having extra columns.
+ Table inheritance supports multiple inheritances.
+ Declarative partitioning supports only list and range partitioning. With table inheritance, you can divide the data as you want. However, if the constraint exclusion can't prune partitions effectively, query performance will suffer.
+ Some operations need a stronger lock when using declarative partitioning than when using table inheritance. For example, adding or removing a partition to or from a partitioned table requires an `ACCESS EXCLUSIVE` lock on the parent table, whereas a `SHARE UPDATE EXCLUSIVE` lock is enough for regular inheritance.
When you use separate job partitions, you can also reload partitions if there are any AWS DMS validation issues. For better performance and replication control, run tasks on separate replication instances.
# Migrate from Amazon RDS for Oracle to Amazon RDS for MySQL
*Jitender Kumar, Srini Ramaswamy, and Neha Sharma, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an Amazon Relational Database Service (Amazon RDS) for Oracle DB instance to an Amazon RDS for MySQL DB instance on Amazon Web Services (AWS). The pattern uses AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT).
The pattern provides best practices for handling the migration of stored procedures. It also covers and code changes to support the application layer.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An Amazon RDS for Oracle source database.
+ An Amazon RDS for MySQL target database. Source and target databases should be in the same virtual private cloud (VPC). If you’re using multiple VPCs, or you must have the required access permissions.
+ Security groups that allow connectivity between the source and target databases, AWS SCT, the application server, and AWS DMS.
+ A user account with the required privilege to run AWS SCT on the source database.
+ Supplemental logging enabled for running AWS DMS on the source database.
**Limitations**
+ The source and target Amazon RDS database size limit is 64 TB. For Amazon RDS size information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html).
+ Oracle is case-insensitive for database objects, but MySQL is not. AWS SCT can handle this issue while creating an object. However, some manual work is required to support full case insensitivity.
+ This migration doesn't use MySQL extensions to enable Oracle-native functions. AWS SCT handles most of the conversion, but some work is required to change code manually.
+ Java Database Connectivity (JDBC) driver changes are required in the application.
**Product versions**
+ Amazon RDS for Oracle 12.2.0.1 and later. For currently supported RDS for Oracle versions, see the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Concepts.database-versions.html).
+ Amazon RDS for MySQL 8.0.15 and later. For currently supported RDS for MySQL versions, see the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html).
+ AWS DMS version 3.3.0 and later. See the AWS documentation for more information about AWS DMS supported [source endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html) and [target endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html).
+ AWS SCT version 1.0.628 and later. See the [AWS SCT source and target endpoint support matrix](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) in the AWS documentation.
## Architecture
**Source technology stack**
+ Amazon RDS for Oracle. For more information, see [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html).
**Target technology stack**
+ Amazon RDS for MySQL. For more information, see [Using a MySQL-Compatible database as a target for AWS DMS](http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html).
**Migration architecture**
In the following diagram, AWS SCT copies and converts schema objects from the Amazon RDS for Oracle source database and sends the objects to the Amazon RDS for MySQL target database. AWS DMS replicates data from the source database and sends it to the Amazon RDS for MySQL instance.
![\[AWS SCT, AWS DMS, and Amazon RDS deployed in a private subnet.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e1efa7c2-47c1-4677-80bc-6b19250fc0d6/images/b54a8442-9ab9-4074-b8f6-a08f87fa2f52.jpeg)
## Tools
+ [AWS Data Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. This pattern uses [Amazon RDS for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) and [Amazon RDS for MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html).
+ [AWS Schema Conversion Tool (AWS SCT)](http://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
## Epics
### Prepare for migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions and engines. | | DBA |
| Identify hardware requirements for the target server instance. | | DBA, SysAdmin |
| Identify storage requirements (storage type and capacity). | | DBA, SysAdmin |
| Choose the proper instance type (capacity, storage features, network features). | | DBA, SysAdmin |
| Identify network-access security requirements for the source and target databases. | | DBA, SysAdmin |
| Choose an application migration strategy. | Consider whether you want full downtime or partial downtime for cutover activities. | DBA, SysAdmin, App owner |
### Configure infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a VPC and subnets. | | SysAdmin |
| Create security groups and network access control lists (ACLs). | | SysAdmin |
| Configure and start the Amazon RDS for Oracle instance. | | DBA, SysAdmin |
| Configure and start the Amazon RDS for MySQL instance. | | DBA, SysAdmin |
| Prepare a test case for validation of code conversion. | This will help in unit-testing for the converted code. | DBA, Developer |
| Configure the AWS DMS instance. | | |
| Configure source and target endpoints in AWS DMS. | | |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Generate the target database script using AWS SCT. | Check the accuracy of the code that was converted by AWS SCT. Some manual work will be required. | DBA, Developer |
| In AWS SCT, choose the "Case Insensitive" setting. | In AWS SCT, choose Project Settings, Target Case Sensitivity, Case Insensitive. | DBA, Developer |
| In AWS SCT, choose not to use the Oracle native function. | In Project Settings, check the functions TO\$1CHAR/TO\$1NUMBER/TO\$1DATE. | DBA, Developer |
| Make changes for "sql%notfound" code. | You might have to convert the code manually. | |
| Query on tables and objects in stored procedures (use lowercase queries). | | DBA, Developer |
| Create the primary script after all changes are made, and then deploy the primary script on the target database. | | DBA, Developer |
| Unit-test stored procedures and application calls using sample data. | | |
| Clean up data that was created during unit testing. | | DBA, Developer |
| Drop foreign key constraints on the target database. | This step is required to load initial data. If you don't want to drop the foreign key constraints, you must create a migration task for data specific to the primary and secondary tables. | DBA, Developer |
| Drop primary keys and unique keys on the target database. | This step results in better performance for the initial load. | DBA, Developer |
| Enable supplemental logging on the source database. | | DBA |
| Create a migration task for the initial load in AWS DMS, and then run it. | Choose the option to migrate existing data. | DBA |
| Add the primary keys and foreign keys to the target database. | Constraints need to be added after the initial load. | DBA, Developer |
| Create a migration task for ongoing replication. | Ongoing replication keeps the target database synchronized with the source database. | DBA |
### Migrate applications
| Task | Description | Skills required |
| --- | --- | --- |
| Replace Oracle native functions with MySQL native functions. | | App owner |
| Make sure that only lowercase names are used for database objects in SQL queries. | | DBA, SysAdmin, App owner |
### Cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the application server. | | App owner |
| Validate that the source and target databases are in sync. | | DBA, App owner |
| Stop the Amazon RDS for Oracle DB instance. | | DBA |
| Stop the migration task. | This will stop automatically after you complete the previous step. | DBA |
| Change the JDBC connection from Oracle to MySQL. | | App owner, DBA |
| Start the application. | | DBA, SysAdmin, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Review and validate the project documents. | | DBA, SysAdmin |
| Gather metrics about time to migrate, percentage of manual versus tool tasks, cost savings, etc. | | DBA, SysAdmin |
| Stop and delete AWS DMS instances. | | DBA |
| Remove the source and target endpoints. | | DBA |
| Remove migration tasks. | | DBA |
| Take a snapshot of the Amazon RDS for Oracle DB instance. | | DBA |
| Delete the Amazon RDS for Oracle DB instance. | | DBA |
| Shut down and delete any other temporary AWS resources you used. | | DBA, SysAdmin |
| Close the project and provide any feedback. | | DBA |
## Related resources
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/Welcome.html)
+ [Amazon RDS Pricing](https://aws.amazon.com/rds/pricing/)
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
# Migrate from IBM Db2 on Amazon EC2 to Aurora PostgreSQL-Compatible using AWS DMS and AWS SCT
*Sirsendu Halder and Abhimanyu Chhabra, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an IBM Db2 database on an Amazon Elastic Compute Cloud (Amazon EC2) instance to an Amazon Aurora PostgreSQL-Compatible Edition DB instance. This pattern uses AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT) for data migration and schema conversion.
The pattern targets an online migration strategy with little or no downtime for a multi-terabyte IBM Db2 database that has a high number of transactions. We recommend that you convert the columns in primary keys (PKs) and foreign keys (FKs) with the data type `NUMERIC` to `INT` or `BIGINT` in PostgreSQL for better performance.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source IBM Db2 database on an EC2 instance
**Product versions**
+ DB2/LINUXX8664 version 11.1.4.4 and later
## Architecture
**Source technology stack**** **
+ A Db2 database on an EC2 instance
**Target technology stack**
+ An Aurora PostgreSQL-Compatible version 10.18 or later DB instance
**Database migration architecture**** **
![\[Using AWS DMS to migrate from IMB Db2 on Amazon EC2 to Aurora PostgreSQL-Compatible.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5e737fab-3e04-4887-9fb0-d1c88503b57d/images/789fabcc-8052-40d5-a746-986d799576e9.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate databases into the AWS Cloud or between combinations of cloud and on-premises setups. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. You can use AWS DMS to migrate your data to and from the most widely used commercial and open-source databases. AWS DMS supports heterogeneous migrations between different database platforms, such as IBM Db2 to Aurora PostgreSQL-Compatible version 10.18 or higher. For details, see [Sources for Data Migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html) and [Targets for Data Migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html) in the AWS DMS documentation.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format that's compatible with the target database. Any objects that are not automatically converted are clearly marked so that they can be manually converted to complete the migration. AWS SCT can also scan the application source code for embedded SQL statements and convert them.
## Epics
### Set up the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Aurora PostgreSQL-Compatible DB instance. | To create the DB instance, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html). For engine type, choose **Amazon Aurora**. For edition, choose **Amazon Aurora PostgreSQL-Compatible Edition**.The Aurora PostgreSQL-Compatible version 10.18 or later DB instance should be in the same virtual private cloud (VPC) as the source IBM Db2 database. | Amazon RDS |
### Convert your database schema
| Task | Description | Skills required |
| --- | --- | --- |
| Install and verify AWS SCT. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.html) | AWS administrator, DBA, Migration engineer |
| Start AWS SCT and create a project. | To start the AWS SCT tool and create a new project to run a database migration assessment report, follow the instructions in the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.Launching). | Migration engineer |
| Add database servers and create a mapping rule. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.html) | Migration engineer |
| Create a database migration assessment report. | Create the database migration assessment report by following the steps in the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.AssessmentReport). | Migration engineer |
| View the assessment report. | Use the **Summary** tab of the database migration assessment report to view the report and analyze the data. This analysis will help you determine the complexity of the migration. For more information, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_AssessmentReport.View.html). | Migration engineer |
| Convert the schema. | To convert your source database schemas:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.html)For more information, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.Converting). | Migration engineer |
| Apply the converted database schema to the target DB instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.html)For more information, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.ApplyingConversion). | Migration engineer |
### Migrate your data
| Task | Description | Skills required |
| --- | --- | --- |
| Set up a VPC and DB parameter groups. | Set up a VPC and DB parameter groups, and configure the inbound rules and parameters required for migration. For instructions, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Prerequisites.html).For the VPC security group, select both the EC2 instance for Db2 and the Aurora PostgreSQL-Compatible DB instance. This replication instance must be in the same VPC as the source and target DB instances. | Migration engineer |
| Prepare source and target DB instances. | Prepare the source and target DB instances for migration. In a production environment, the source database will already exist.For the source database, the server name must be the public Domain Name System (DNS) of the EC2 instance where Db2 is running. For user name, you can use `db2inst1` followed by the port, which will be 5000 for IBM Db2. | Migration engineer |
| Create an Amazon EC2 client and endpoints. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.html) | Migration engineer |
| Create a replication instance. | Create a replication instance by using the AWS DMS console and specify the source and target endpoints. The replication instance performs the data migration between the endpoints. For more information, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Replication.html). | Migration engineer |
| Create an AWS DMS task to migrate the data. | Create a task to load the source IBM Db2 tables to the target PostgreSQL DB instance by following the steps in the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Replication.html#CHAP_GettingStarted.Replication.Tasks).[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.html) | Migration engineer |
## Related resources
**References**
+ [Amazon Aurora documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html)
+ [PostgreSQL foreign data wrapper (FDW) documentation](https://www.postgresql.org/docs/10/postgres-fdw.html)
+ [PostgreSQL IMPORT FOREIGN SCHEMA documentation ](https://www.postgresql.org/docs/10/sql-importforeignschema.html)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/index.html)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
**Tutorials and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/) (walkthrough)
+ [Introduction to Amazon EC2 - Elastic Cloud Server & Hosting with AWS](https://www.youtube.com/watch?v=TsRBftzZsQo) (video)
# Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using SharePlex and AWS DMS
*Kumar Babu P G, Amazon Web Services*
## Summary
This pattern describes how to migrate an on-premises Oracle 8i or 9i database to Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL. AWS Database Migration Service (AWS DMS) doesn't support Oracle 8i or 9i as a source, so Quest SharePlex replicates data from an on-premises 8i or 9i database to an intermediate Oracle database (Oracle 10g or 11g), which is compatible with AWS DMS.
From the intermediate Oracle instance, the schema and data are migrated to the PostgreSQL database on AWS by using AWS Schema Conversion Tool (AWS SCT) and AWS DMS. This method helps achieve continuous streaming of data from the source Oracle database to the target PostgreSQL DB instance with minimum replication lag. In this implementation, the downtime is limited to the length of time it takes to create or validate all the foreign keys, triggers, and sequences on the target PostgreSQL database.
The migration uses an Amazon Elastic Compute Cloud (Amazon EC2) instance with Oracle 10g or 11g installed to host the changes from the source Oracle database. AWS DMS uses this intermediate Oracle instance as the source to stream the data to Amazon RDS for PostgreSQL or Aurora PostgreSQL. Data replication can be paused and resumed from the on-premises Oracle database to the intermediate Oracle instance. It can also be paused and resumed from the intermediate Oracle instance to the target PostgreSQL database so you can validate the data by using either AWS DMS data validation or a custom data validation tool.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A source Oracle 8i or 9i database in an on-premises data center
+ AWS Direct Connect configured between the on-premises data center and AWS
+ Java Database Connectivity (JDBC) drivers for AWS SCT connectors installed either on a local machine or on the EC2 instance where AWS SCT is installed
+ Familiarity with [using an Oracle database as an AWS DMS source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [using a PostgreSQL database as an AWS DMS target](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
+ Familiarity with Quest SharePlex data replication
**Limitations**
+ The database size limit is 64 TB
+ The on-premises Oracle database must be Enterprise Edition
**Product versions**
+ Oracle 8i or 9i for the source database
+ Oracle 10g or 11g for the intermediate database
+ PostgreSQL 9.6 or later
## Architecture
**Source technology stack **
+ Oracle 8i or 9i database
+ Quest SharePlex
**Target technology stack **
+ Amazon RDS for PostgreSQL or Aurora PostgreSQL
** **
**Source and target architecture**
![\[Architecture diagram showing migration from on-premises Oracle database to AWS cloud using various services.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b6c30668-fc2e-4293-a59a-e01fd151f4bb/images/25082670-0bf3-4b20-8c80-99c6633b046f.png)
## Tools
+ **AWS DMS** – [AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html) (AWS DMS) helps you migrate databases quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. AWS DMS can migrate your data to and from the most widely used commercial and open-source databases.
+ **AWS SCT** – [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) (AWS SCT) makes heterogeneous database migrations predictable by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database. Objects that cannot be automatically converted are clearly marked so that they can be manually converted to complete the migration. AWS SCT can also scan your application source code for embedded SQL statements and convert them as part of a database schema conversion project. During this process, AWS SCT performs cloud-native code optimization by converting legacy Oracle and SQL Server functions to their AWS equivalents, to help you modernize your applications while migrating your databases. When schema conversion is complete, AWS SCT can help migrate data from a range of data warehouses to Amazon Redshift by using built-in data migration agents.
+ **Quest SharePlex** – [Quest SharePlex](https://www.quest.com/register/120420/?gclid=Cj0KCQiA6IHwBRCJARIsALNjViVSt9fHqAsf9XbWkoCwKKyQqollR_5kSxNhBagh9s3spQT4IQCaVy0aAmCnEALw_wcB) is an Oracle-to-Oracle data replication tool for moving data with minimal downtime and no data loss.
## Epics
### Create the EC2 instance and install Oracle
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the network for Amazon EC2. | Create the virtual private cloud (VPC), subnets, internet gateway, route tables, and security groups. | AWS SysAdmin |
| Create the new EC2 instance. | Select the Amazon Machine Image (AMI) for the EC2 instance. Choose the instance size and configure instance details: the number of instances (1), the VPC and subnet from the previous step, auto-assign public IP, and other options. Add storage, configure security groups, and launch the instance. When prompted, create and save a key pair for the next step. | AWS SysAdmin |
| Install Oracle on the EC2 instance. | Acquire the licenses and the required Oracle binaries, and install Oracle 10g or 11g on the EC2 instance. | DBA |
### Set up SharePlex on an EC2 instance and configure data replication
| Task | Description | Skills required |
| --- | --- | --- |
| Set up SharePlex. | Create an Amazon EC2 instance and install the SharePlex binaries that are compatible with Oracle 8i or 9i. | AWS SysAdmin, DBA |
| Configure data replication. | Follow SharePlex best practices to configure data replication from an on-premises Oracle 8i/9i database to an Oracle 10g/11g instance. | DBA |
### Convert the Oracle database schema to PostgreSQL
| Task | Description | Skills required |
| --- | --- | --- |
| Set up AWS SCT. | Create a new report, and then connect to Oracle as the source and PostgreSQL as the target. In project settings, open the SQL Scripting tab and change the target SQL script to Multiple Files. | DBA |
| Convert the Oracle database schema. | In the Action tab, choose Generate Report, Convert Schema, and then Save as SQL. | DBA |
| Modify the SQL scripts generated by AWS SCT. | | DBA |
### Create and configure the Amazon RDS DB instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Amazon RDS DB instance. | In the Amazon RDS console, create a new PostgreSQL DB instance. | AWS SysAdmin, DBA |
| Configure the DB instance. | Specify the DB engine version, DB instance class, Multi-AZ deployment, storage type, and allocated storage. Enter the DB instance identifier, a master user name, and a master password. | AWS SysAdmin, DBA |
| Configure network and security. | Specify the VPC, subnet group, public accessibility, Availability Zone preference, and security groups. | AWS SysAdmin, DBA |
| Configure database options. | Specify the database name, port, parameter group, encryption, and master key. | AWS SysAdmin, DBA |
| Configure backups. | Specify the backup retention period, backup window, start time, duration, and whether to copy tags to snapshots. | AWS SysAdmin, DBA |
| Configure monitoring options. | Enable or disable enhanced monitoring and performance insights. | AWS SysAdmin, DBA |
| Configure maintenance options. | Specify auto minor version upgrade, maintenance window, and the start day, time, and duration. | AWS SysAdmin, DBA |
| Run the pre-migration scripts from AWS SCT. | On the Amazon RDS instance, run these scripts: create\$1database.sql, create\$1sequence.sql, create\$1table.sql, create\$1view.sql, and create\$1function.sql. | AWS SysAdmin, DBA |
### Migrate data by using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance in AWS DMS. | Complete the fields for the name, instance class, VPC (same as for the EC2 instance), Multi-AZ, and public accessibility. In the advanced configuration section, specify allocated storage, subnet group, Availability Zone, VPC security groups, and AWS Key Management Service (AWS KMS) root key. | AWS SysAdmin, DBA |
| Create the source database endpoint. | Specify the endpoint name, type, source engine (Oracle), server name (Amazon EC2 private DNS name), port, SSL mode, user name, password, SID, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose Run Test, and then create the endpoint. You can also configure the following advanced settings: maxFileSize and numberDataTypeScale. | AWS SysAdmin, DBA |
| Create the AWS DMS replication task. | Specify the task name, replication instance, source and target endpoints, and replication instance. For migration type, choose "Migrate existing data and replicate ongoing changes." Clear the "Start task on create" check box. | AWS SysAdmin, DBA |
| Configure the AWS DMS replication task settings. | For target table preparation mode, choose "Do nothing." Stop the task after the full load completes to create primary keys. Specify limited or full LOB mode, and enable control tables. Optionally, you can configure the CommitRate advanced setting. | DBA |
| Configure the table mappings. | In the table mappings section, create an Include rule for all tables in all schemas included in the migration, and then create an Exclude rule. Add three transformation rules to convert the schema, table, and column names to lowercase, and add any other rules needed for this specific migration. | DBA |
| Start the task. | Start the replication task. Make sure that the full load is running. Run ALTER SYSTEM SWITCH LOGFILE on the primary Oracle database to kick-start the task. | DBA |
| Run the mid-migration scripts from AWS SCT. | In Amazon RDS for PostgreSQL, run these scripts: create\$1index.sql and create\$1constraint.sql. | DBA |
| Restart the task to continue change data capture (CDC). | In the Amazon RDS for PostgreSQL DB instance, run VACUUM, and restart the AWS DMS task to apply the cached CDC changes. | DBA |
### Cut over to the PostgreSQL database
| Task | Description | Skills required |
| --- | --- | --- |
| Check the AWS DMS logs and metadata tables. | Validate any errors and fix if required. | DBA |
| Stop all Oracle dependencies. | Shut down listeners on the Oracle database and run ALTER SYSTEM SWITCH LOGFILE. Stop the AWS DMS task when it shows no activity. | DBA |
| Run the post-migration scripts from AWS SCT. | In Amazon RDS for PostgreSQL, run these scripts: create\$1foreign\$1key\$1constraint.sql and create\$1triggers.sql. | DBA |
| Complete any additional Amazon RDS for PostgreSQL steps. | Increment sequences to match Oracle if needed, run VACUUM and ANALYZE, and take a snapshot for compliance. | DBA |
| Open the connections to Amazon RDS for PostgreSQL. | Remove the AWS DMS security groups from Amazon RDS for PostgreSQL, add production security groups, and point your applications to the new database. | DBA |
| Clean up AWS DMS resources. | Remove the endpoints, replication tasks, replication instances, and the EC2 instance. | SysAdmin, DBA |
## Related resources
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon RDS for PostgreSQL pricing](https://aws.amazon.com/rds/postgresql/pricing/)
+ [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
+ [Quest SharePlex documentation](https://support.quest.com/shareplex/9.0.2/technical-documents)
# Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using materialized views and AWS DMS
*Kumar Babu P G and Pragnesh Patel, Amazon Web Services*
## Summary
This pattern describes how to migrate an on-premises legacy Oracle 8i or 9i database to Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition.
AWS Database Migration Service (AWS DMS) doesn't support Oracle 8i or 9i as a source, so this pattern uses an intermediate Oracle database instance that's compatible with AWS DMS, such as Oracle 10g or 11g. It also uses the materialized views feature to migrate data from the source Oracle 8i/9i instance to the intermediate Oracle 10g/11g instance.
AWS Schema Conversion Tool (AWS SCT) converts the database schema, and AWS DMS migrates the data to the target PostgreSQL database.
This pattern helps users who want to migrate from legacy Oracle databases with minimum database downtime. In this implementation, the downtime would be limited to the length of time it takes to create or validate all the foreign keys, triggers, and sequences on the target database.
The pattern uses Amazon Elastic Compute Cloud (Amazon EC2) instances with an Oracle 10g/11g database installed to help AWS DMS stream the data. You can temporarily pause streaming replication from the on-premises Oracle database to the intermediate Oracle instance to enable AWS DMS to catch up on data validation or to use another data validation tool. The PostgreSQL DB instance and intermediate Oracle database will have the same data when AWS DMS has finished migrating current changes.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A source Oracle 8i or 9i database in an on-premises data center
+ AWS Direct Connect configured between the on-premises data center and AWS
+ Java Database Connectivity (JDBC) drivers for AWS SCT connectors installed either on a local machine or on the EC2 instance where AWS SCT is installed
+ Familiarity with [using an Oracle database as an AWS DMS source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [using a PostgreSQL database as an AWS DMS target](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
**Limitations **
+ The database size limit is 64 TB
**Product versions**
+ Oracle 8i or 9i for the source database
+ Oracle 10g or 11g for the intermediate database
+ PostgreSQL 10.17 or later
## Architecture
**Source technology stack **
+ Oracle 8i or 9i database
**Target technology stack **
+ Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible
**Target architecture**
![\[Architecture for migrating from a legacy Oracle database to Amazon RDS or Aurora\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8add9b21-1b62-46a2-bb8e-0350f36a924a/images/f34f9b0f-f1da-4c27-a385-71b12d16c375.png)
## Tools
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html) helps migrate databases quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. AWS DMS can migrate your data to and from the most widely used commercial and open-source databases.
+ [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format compatible with the target database. Objects that cannot be automatically converted are clearly marked so that they can be manually converted to complete the migration. AWS SCT can also scan your application source code for embedded SQL statements and convert them as part of a database schema conversion project. During this process, AWS SCT performs cloud-native code optimization by converting legacy Oracle and SQL Server functions to their AWS equivalents, to help you modernize your applications while migrating your databases. When schema conversion is complete, AWS SCT can help migrate data from a range of data warehouses to Amazon Redshift by using built-in data migration agents.
## Best practices
For best practices for refreshing materialized views, see the following Oracle documentation:
+ [Refreshing materialized views](https://docs.oracle.com/database/121/DWHSG/refresh.htm#DWHSG-GUID-64068234-BDB0-4C12-AE70-75571046A586)
+ [Fast refresh for materialized views](https://docs.oracle.com/database/121/DWHSG/refresh.htm#DWHSG8361)
## Epics
### Install Oracle on an EC2 instance and create materialized views
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the network for the EC2 instance. | Create the virtual private cloud (VPC), subnets, internet gateway, route tables, and security groups. | AWS SysAdmin |
| Create the EC2 instance. | Select the Amazon Machine Image (AMI) for the EC2 instance. Choose the instance size and configure instance details: the number of instances (1), the VPC and subnet from the previous step, auto-assign public IP, and other options. Add storage, configure security groups, and launch the instance. When prompted, create and save a key pair for the next step. | AWS SysAdmin |
| Install Oracle on the EC2 instance. | Acquire the licenses and the required Oracle binaries, and install Oracle 10g or 11g on the EC2 instance. | DBA |
| Configure Oracle networking. | Modify or add entries in `listener.ora` to connect to the on-premises source Oracle 8i/9i database, and then create the database links. | DBA |
| Create materialized views. | Identify the database objects to replicate in the source Oracle 8i/9i database, and then create materialized views for all the objects by using the database link. | DBA |
| Deploy scripts to refresh materialized views at required intervals. | Develop and deploy scripts to refresh materialized views at required intervals on the Amazon EC2 Oracle 10g/11g instance. Use the incremental refresh option to refresh materialized views. | DBA |
### Convert the Oracle database schema to PostgreSQL
| Task | Description | Skills required |
| --- | --- | --- |
| Set up AWS SCT. | Create a new report, and then connect to Oracle as the source and PostgreSQL as the target. In project settings, open the **SQL Scripting** tab. Change the target SQL script to **Multiple Files**. (AWS SCT doesn't support Oracle 8i/9i databases, so you have to restore the schema-only dump on the intermediate Oracle 10g/11g instance and use it as a source for AWS SCT.) | DBA |
| Convert the Oracle database schema. | On the **Action** tab, choose **Generate Report**, **Convert Schema**, and then **Save as SQL**. | DBA |
| Modify the SQL scripts. | Make modifications based on best practices. For example, switch to suitable data types and develop PostgreSQL equivalents for Oracle-specific functions. | DBA, DevDBA |
### Create and configure the Amazon RDS DB instance to host the converted database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Amazon RDS DB instance. | In the Amazon RDS console, create a new PostgreSQL DB instance. | AWS SysAdmin, DBA |
| Configure the DB instance. | Specify the DB engine version, DB instance class, Multi-AZ deployment, storage type, and allocated storage. Enter the DB instance identifier, a master user name, and a master password. | AWS SysAdmin, DBA |
| Configure network and security. | Specify the VPC, subnet group, public accessibility, Availability Zone preference, and security groups. | DBA, SysAdmin |
| Configure database options. | Specify the database name, port, parameter group, encryption, and master key. | DBA, AWS SysAdmin |
| Configure backups. | Specify the backup retention period, backup window, start time, duration, and whether to copy tags to snapshots. | AWS SysAdmin, DBA |
| Configure monitoring options. | Enable or disable enhanced monitoring and performance insights. | AWS SysAdmin, DBA |
| Configure maintenance options. | Specify auto minor version upgrade, maintenance window, and the start day, time, and duration. | AWS SysAdmin, DBA |
| Run the pre-migration scripts from AWS SCT. | On the target Amazon RDS for PostgreSQL instance, create the database schema by using the SQL scripts from AWS SCT with other modifications. These might include running multiple scripts and including user creation, database creation, schema creation, tables, views, functions, and other code objects. | AWS SysAdmin, DBA |
### Migrate data by using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance in AWS DMS. | Complete the fields for the name, instance class, VPC (same as for the EC2 instance), Multi-AZ, and public accessibility. In the advanced configuration section, specify allocated storage, subnet group, Availability Zone, VPC security groups, and AWS Key Management Service (AWS KMS) key. | AWS SysAdmin, DBA |
| Create the source database endpoint. | Specify the endpoint name, type, source engine (Oracle), server name (the EC2 instance's private DNS name), port, SSL mode, user name, password, SID, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose **Run Test**, and then create the endpoint. You can also configure the following advanced settings: **maxFileSize** and **numberDataTypeScale**. | AWS SysAdmin, DBA |
| Connect AWS DMS to Amazon RDS for PostgreSQL. | Create a migration security group for connections across VPCs, if your PostgreSQL database is in another VPC. | AWS SysAdmin, DBA |
| Create the target database endpoint. | Specify the endpoint name, type, source engine (PostgreSQL), server name (Amazon RDS endpoint), port, SSL mode, user name, password, database name, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose **Run Test**, and then create the endpoint. You can also configure the following advanced settings: **maxFileSize** and **numberDataTypeScale**. | AWS SysAdmin, DBA |
| Create the AWS DMS replication task. | Specify the task name, replication instance, source and target endpoints, and replication instance. For migration type, choose **Migrate existing data and replicate ongoing changes**. Clear the **Start task on create** check box. | AWS SysAdmin, DBA |
| Configure the AWS DMS replication task settings. | For target table preparation mode, choose **Do nothing**. Stop the task after full load completes (to create primary keys). Specify limited or full LOB mode, and enable control tables. Optionally, you can configure the **CommitRate** advanced setting. | DBA |
| Configure the table mappings. | In the **Table mappings** section, create an Include rule for all tables in all schemas included in the migration, and then create an Exclude rule. Add three transformation rules to convert the schema, table, and column names to lowercase, and add any other rules you need for this specific migration. | DBA |
| Start the task. | Start the replication task. Make sure that the full load is running. Run `ALTER SYSTEM SWITCH LOGFILE` on the primary Oracle database to start the task. | DBA |
| Run the mid-migration scripts from AWS SCT. | In Amazon RDS for PostgreSQL, run the following scripts: `create_index.sql` and `create_constraint.sql` (if the complete schema wasn't initially created). | DBA |
| Resume the task to continue change data capture (CDC). | Run `VACUUM` on the Amazon RDS for PostgreSQL DB instance, and restart the AWS DMS task to apply cached CDC changes. | DBA |
### Cut over to the PostgreSQL database
| Task | Description | Skills required |
| --- | --- | --- |
| Check the AWS DMS logs and validation tables. | Check and fix any replication or validation errors. | DBA |
| Stop using the on-premises Oracle database and its dependencies. | Stop all Oracle dependencies, shut down listeners on the Oracle database, and run `ALTER SYSTEM SWITCH LOGFILE`. Stop the AWS DMS task when it shows no activity. | DBA |
| Run the post-migration scripts from AWS SCT. | In Amazon RDS for PostgreSQL, run these scripts: `create_foreign_key_constraint.sql and create_triggers.sql`. Make sure that the sequences are up to date. | DBA |
| Complete additional Amazon RDS for PostgreSQL steps. | Increment sequences to match Oracle if needed, run `VACUUM` and `ANALYZE`, and take a snapshot for compliance. | DBA |
| Open the connections to Amazon RDS for PostgreSQL. | Remove the AWS DMS security groups from Amazon RDS for PostgreSQL, add production security groups, and point your applications to the new database. | DBA |
| Clean up the AWS DMS objects. | Remove the endpoints, replication tasks, replication instances, and the EC2 instance. | SysAdmin, DBA |
## Related resources
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon RDS for PostgreSQL pricing](https://aws.amazon.com/rds/postgresql/pricing/)
+ [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
# Migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL using AWS DMS and AWS SCT
*Anil Kunapareddy, Amazon Web Services*
*Harshad Gohil, None*
## Summary
Managing Oracle databases on Amazon Elastic Compute Cloud (Amazon EC2) instances requires resources and can be costly. Moving these databases to an Amazon Relational Database Service (Amazon RDS) for MySQL DB instance will ease your job by optimizing the overall IT budget. Amazon RDS for MySQL also provides features like Multi-AZ, scalability, and automatic backups.
This pattern walks you through the migration of a source Oracle database on Amazon EC2 to a target Amazon RDS for MySQL DB instance. It uses AWS Database Migration Service (AWS DMS) to migrate the data, and AWS Schema Conversion Tool (AWS SCT) to convert the source database schema and objects to a format that's compatible with Amazon RDS for MySQL.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source database with instance and listener services running, in ARCHIVELOG mode
+ A target Amazon RDS for MySQL database, with sufficient storage for data migration
**Limitations**
+ AWS DMS does not create a schema on the target database; you have to do that. The schema name must already exist for the target. Tables from the source schema are imported to user/schema, which AWS DMS uses to connect to the target instance. You must create multiple replication tasks if you have to migrate multiple schemas.
**Product versions**
+ All Oracle database editions for versions 10.2 and later, 11g and up to 12.2, and 18c. For the latest list of supported versions, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and [Using a MySQL-Compatible Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support. For information about Oracle database versions supported by AWS SCT, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html).
+ AWS DMS supports versions 5.5, 5.6, and 5.7 of MySQL.
## Architecture
**Source technology stack**
+ An Oracle database on an EC2 instance
**Target technology stack**
+ Amazon RDS for MySQL DB instance
**Data migration architecture**
![\[Using AWS DMS to migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8a8e346e-7944-4999-bc11-208efead3792/images/c00f908c-f348-41dd-a31c-3931b990777a.png)
**Source and target architecture **
![\[Using AWS DMS and AWS SCT to migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8a8e346e-7944-4999-bc11-208efead3792/images/e7ba7ac0-3094-4142-b355-fb192e242432.png)
## Tools
+ **AWS DMS** - [AWS Database Migration Service](https://docs.aws.amazon.com/dms/) (AWS DMS) is a web service you can use to migrate data from your database that is on-premises, on an Amazon RDS DB instance, or in a database on an EC2 instance, to a database on an AWS service such as Amazon RDS for MySQL or an EC2 instance. You can also migrate a database from an AWS service to an on-premises database. You can migrate data between heterogeneous or homogeneous database engines.
+ **AWS SCT** - [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) (AWS SCT) makes heterogeneous database migrations predictable by automatically converting the source database schema and a majority of the database code objects, including views, stored procedures, and functions, to a format that's compatible with the target database. After converting your database schema and code objects using AWS SCT, you can use AWS DMS to migrate data from the source database to the target database to complete your migration projects.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Identify the source and target database versions and engines. | | DBA/Developer |
| Identify the DMS replication instance. | | DBA/Developer |
| Identify storage requirements such as storage type and capacity. | | DBA/Developer |
| Identify network requirements such as latency and bandwidth. | | DBA/Developer |
| Identify hardware requirements for the source and target server instances (based on Oracle compatibility list and capacity requirements). | | DBA/Developer |
| Identify network access security requirements for source and target databases. | | DBA/Developer |
| Install AWS SCT and Oracle drivers. | | DBA/Developer |
| Determine a backup strategy. | | DBA/Developer |
| Determine availability requirements. | | DBA/Developer |
| Identify application migration and switch-over strategy. | | DBA/Developer |
| Select the proper DB instance type based on capacity, storage, and network features. | | DBA/Developer |
### Configure the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). The source, target, and replication instance should be in the same VPC. It is also good to have these in the same Availability Zone. | | Developer |
| Create the necessary security groups for database access. | | Developer |
| Generate and configure a key pair. | | Developer |
| Configure subnets, Availability Zones, and CIDR blocks. | | Developer |
### Configure the source: Oracle database on EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Install Oracle Database on Amazon EC2 with required users and roles. | | DBA |
| Perform the three steps in the next column to access Oracle from outside the EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-on-amazon-ec2-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.html) | DBA |
| When Amazon EC2 is restarted, the public DNS changes. Make sure to update Amazon EC2 public DNS in 'tnsnames' and 'listener' or use an Elastic IP address. | | DBA/Developer |
| Configure the EC2 instance security group so that the replication instance and required clients can access the source database. | | DBA/Developer |
### Configure the target: Amazon RDS for MySQL
| Task | Description | Skills required |
| --- | --- | --- |
| Configure and start the Amazon RDS for MySQL DB instance. | | Developer |
| Create the necessary tablespace in the Amazon RDS for MySQL DB instance. | | DBA |
| Configure the security group so that the replication instance and required clients can access the target database. | | Developer |
### Configure AWS SCT and create a schema in the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS SCT and Oracle drivers. | | Developer |
| Enter the appropriate parameters and connect to the source and target. | | Developer |
| Generate a schema conversion report. | | Developer |
| Correct the code and schema as necessary, especially tablespaces and quotes, and run on the target database. | | Developer |
| Validate the schema on source vs. target before migrating data. | | Developer |
### Migrate data using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| For full-load and change data capture (CDC) or just CDC, you must set an extra connection attribute. | | Developer |
| The user specified in the AWS DMS source Oracle database definitions must be granted all the required privileges. For a complete list, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP\$1Source.Oracle.html\$1CHAP\$1Source.Oracle.Self-Managed. | | DBA/Developer |
| Enable supplemental logging in the source database. | | DBA/Developer |
| For full-load and change data capture (CDC) or just CDC, enable ARCHIVELOG mode in the source database. | | DBA |
| Create source and target endpoints, and test the connections. | | Developer |
| When the endpoints are connected successfully, create a replication task. | | Developer |
| Select CDC only (or) full load plus CDC in the task to capture changes for continuous replication only (or) full load plus ongoing changes, respectively. | | Developer |
| Run the replication task and monitor Amazon CloudWatch logs. | | Developer |
| Validate the data in the source and target databases. | | Developer |
### Migrate your application and cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the steps for your application migration strategy. | | DBA, Developer, App owner |
| Follow the steps for your application cutover/switch-over strategy. | | DBA, Developer, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the schema and data in source vs. target databases. | | DBA/Developer |
| Gather metrics around time to migrate, percent of manual vs. tool, cost savings, etc. | | DBA/Developer/AppOwner |
| Review the project documents and artifacts. | | DBA/Developer/AppOwner |
| Shut down temporary AWS resources. | | DBA/Developer |
| Close out the project and provide feedback. | | DBA/Developer/AppOwner |
## Related resources
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [AWS DMS blog posts](https://aws.amazon.com/blogs/database/tag/dms/)
+ [Strategies for Migrating Oracle Database to AWS](https://d1.awsstatic.com/whitepapers/strategies-for-migrating-oracle-database-to-aws.pdf)
+ [Amazon RDS for Oracle FAQs](https://aws.amazon.com/rds/oracle/faqs/)
+ [Oracle FAQ](https://aws.amazon.com/oracle/faq/)
+ [Amazon EC2](https://aws.amazon.com/ec2/)
+ [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/)
+ [Licensing Oracle Software in the Cloud Computing Environment](http://www.oracle.com/us/corporate/pricing/cloud-licensing-070579.pdf)
# Migrate an Oracle database from Amazon EC2 to Amazon RDS for MariaDB using AWS DMS and AWS SCT
*Veeranjaneyulu Grandhi and vinod kumar, Amazon Web Services*
## Summary
This pattern walks you through the steps for migrating an Oracle database on an Amazon Elastic Compute Cloud (Amazon EC2) instance to an Amazon Relational Database Service (Amazon RDS) for MariaDB DB instance. The pattern uses AWS Data Migration Service (AWS DMS) for data migration and AWS Schema Conversion Tool (AWS SCT) for schema conversion.
Managing Oracle databases on EC2 instances requires more resources and is more costly than using a database on Amazon RDS. Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud. Amazon RDS provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching, and backups.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ A source Oracle database with instance and listener services up and running. This database should be in ARCHIVELOG mode.
+ Familiarity with [Using an Oracle Database as a Source for AWS DMS.](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [Using Oracle as a Source for AWS SCT.](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html)
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ All Oracle database editions for versions 10.2 and later, 11g and up to 12.2, and 18c. For the latest list of supported versions, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and the [AWS SCT version table](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) in the AWS documentation.
+ Amazon RDS supports MariaDB Server Community Server versions 10.3, 10.4, 10.5, and 10.6. For the latest list of supported versions, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html).
## Architecture
**Source technology stack**
+ An Oracle database on an EC2 instance
**Target technology stack**
+ Amazon RDS for MariaDB
**Data migration architecture**
![\[Using AWS DMS for the migration.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0b4269c6-8ea3-4672-ad14-1ffac1dc14f3/images/ed191145-e5c2-4d61-8827-31f081450c03.png)
**Target architecture**
![\[Using AWS SCT for the migration.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0b4269c6-8ea3-4672-ad14-1ffac1dc14f3/images/0171f548-37dd-4110-851c-7e74dfff3732.png)
## Tools
+ [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) (AWS SCT) makes heterogeneous database migrations predictable by automatically converting the source database schema and a majority of the database code objects—including views, stored procedures, and functions—to a format compatible with the target database. After converting your database schema and code objects using AWS SCT, you can use AWS DMS to migrate data from the source database to the target database to complete your migration projects. For more information, see [Using Oracle as a Source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html) in the AWS SCT documentation.
+ [AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) (AWS DMS) helps you migrate databases to AWS quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. AWS DMS can migrate your data to and from the most widely used commercial and open-source databases. AWS DMS supports homogeneous migrations such as Oracle to Oracle, as well as heterogeneous migrations between different database platforms, such as Oracle or Microsoft SQL Server to Amazon Aurora. To learn more about migrating Oracle databases, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) in the AWS DMS documentation.
## Epics
### Plan for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Identify versions and database engines. | Identify the source and target database versions and engines. | DBA, Developer |
| Identify the replication instance. | Identify the AWS DMS replication instance. | DBA, Developer |
| Identify storage requirements. | Identify storage type and capacity. | DBA, Developer |
| Identify network requirements. | Identify network latency and bandwidth. | DBA, Developer |
| Identify hardware requirements. | Identify hardware requirements for the source and target server instances (based on the Oracle compatibility list and capacity requirements). | DBA, Developer |
| Identify security requirements. | Identify network-access security requirements for the source and target databases. | DBA, Developer |
| Install drivers. | Install the latest AWS SCT and Oracle drivers. | DBA, Developer |
| Determine a backup strategy. | | DBA, Developer |
| Determine availability requirements. | | DBA, Developer |
| Choose an application migration/switchover strategy. | | DBA, Developer |
| Select the instance type. | Select the proper instance type based on capacity, storage, and network features. | DBA, Developer |
### Configure the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | The source, target, and replication instances should be in the same VPC and in the same Availability Zone (recommended). | Developer |
| Create security groups. | Create the necessary security groups for database access. | Developer |
| Generate a key pair. | Generate and configure a key pair. | Developer |
| Configure other resources. | Configure subnets, Availability Zones, and CIDR blocks. | Developer |
### Configure the source
| Task | Description | Skills required |
| --- | --- | --- |
| Launch the EC2 instance. | For instructions, see the [Amazon EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html). | Developer |
| Install the Oracle database. | Install the Oracle database on the EC2 instance, with required users and roles. | DBA |
| Follow the steps in the task description to access Oracle from outside of the EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-mariadb-using-aws-dms-and-aws-sct.html) | DBA |
| Update the Amazon EC2 public DNS. | After the EC2 instance restarts, the public DNS changes. Make sure to update the Amazon EC2 public DNS in `tnsnames` and `listener`, or use an Elastic IP address. | DBA, Developer |
| Configure the EC2 instance security group. | Configure the EC2 instance security group so the replication instance and required clients can access the source database. | DBA, Developer |
### Configure the target Amazon RDS for MariaDB environment
| Task | Description | Skills required |
| --- | --- | --- |
| Start the RDS DB instance. | Configure and start the Amazon RDS for MariaDB DB instance. | Developer |
| Create tablespaces. | Create any necessary tablespaces in the Amazon RDS MariaDB database. | DBA |
| Configure a security group. | Configure a security group so the replication instance and required clients can access the target database. | Developer |
### Configure AWS SCT
| Task | Description | Skills required |
| --- | --- | --- |
| Install drivers. | Install the latest AWS SCT and Oracle drivers. | Developer |
| Connect. | Enter appropriate parameters and then connect to the source and target. | Developer |
| Generate a schema conversion report. | Generate an AWS SCT schema conversion report. | Developer |
| Correct the code and schema as necessary. | Make any necessary corrections to the code and schema (especially tablespaces and quotation marks). | DBA, Developer |
| Validate the schema. | Validate the schema on the source versus the target before loading data. | Developer |
### Migrate data using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Set a connection attribute. | For full-load and change data capture (CDC) or just for CDC, set an extra connection attribute. For more information, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html). | Developer |
| Enable supplemental logging. | Enable supplemental logging on the source database. | DBA, Developer |
| Enable archive log mode. | For full-load and CDC (or just for CDC), enable archive log mode on the source database. | DBA |
| Create and test endpoints. | Create source and target endpoints and test the connections. For more information, see the [Amazon DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html). | Developer |
| Create a replication task. | When the endpoints are connected successfully, create a replication task. For more information, see the [Amazon DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html). | Developer |
| Choose replication type. | Choose **CDC only** or **Full load plus CDC** in the task to capture changes for continuous replication only, or for full load and ongoing changes, respectively. | Developer |
| Start and monitor the task. | Start the replication task and monitor Amazon CloudWatch logs. For more information, see the [Amazon DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html). | Developer |
| Validate the data. | Validate the data in the source and target databases. | Developer |
### Migrate applications and cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the chosen application migration strategy. | | DBA, App owner, Developer |
| Follow the chosen application cutover/switchover strategy. | | DBA, App owner, Developer |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the schema and data. | Ensure that the schema and data are validated successfully in the source versus the target before project closure. | DBA, Developer |
| Gather metrics. | Gather metrics for time to migrate, percentage of manual versus tool tasks, cost savings, and similar criteria. | DBA, App owner, Developer |
| Review documentation. | Review the project documents and artifacts. | DBA, App owner, Developer |
| Shut down resources. | Shut down temporary AWS resources. | DBA, Developer |
| Close the project. | Close the migration project and provide any feedback. | DBA, App owner, Developer |
## Related resources
+ [MariaDB Amazon RDS overview](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html)
+ [Amazon RDS for MariaDB product details](https://aws.amazon.com/rds/mariadb/features)
+ [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Strategies for Migrating Oracle Databases to AWS](https://docs.aws.amazon.com/whitepapers/latest/strategies-migrating-oracle-db-to-aws/strategies-migrating-oracle-db-to-aws.html)
+ [Licensing Oracle Software in the Cloud Computing Environment](http://www.oracle.com/us/corporate/pricing/cloud-licensing-070579.pdf)
+ [Amazon RDS for Oracle FAQs](https://aws.amazon.com/rds/oracle/faqs/)
+ [AWS DMS overview](https://aws.amazon.com/dms/)
+ [AWS DMS blog posts](https://aws.amazon.com/blogs/database/tag/dms/)
+ [Amazon EC2 overview](https://aws.amazon.com/ec2/)
+ [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
# Migrate an on-premises Oracle database to Amazon RDS for MySQL using AWS DMS and AWS SCT
*Sergey Dmitriev and Naresh Damera, Amazon Web Services*
## Summary
This pattern walks you through the migration of an on-premises Oracle database to an Amazon Relational Database Service (Amazon RDS) for MySQL DB instance. It uses AWS Database Migration Service (AWS DMS) to migrate the data, and AWS Schema Conversion Tool (AWS SCT) to convert the source database schema and objects to a format that's compatible with Amazon RDS for MySQL.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Oracle database in an on-premises data center
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ All Oracle database editions for versions 11g (versions 11.2.0.3.v1 and later) and up to 12.2, and 18c. For the latest list of supported versions, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support. For information about Oracle database versions supported by AWS SCT, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html).
+ AWS DMS currently supports MySQL versions 5.5, 5.6, and 5.7. For the latest list of supported versions, see [Using a MySQL-Compatible Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html) in the AWS documentation.
## Architecture
**Source technology stack**
+ On-premises Oracle database
**Target technology stack**
+ Amazon RDS for MySQL DB instance
**Data migration architecture**
![\[AWS Cloud architecture showing data migration from on-premises to RDS via VPC, Internet Gateway, and AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0385e5ad-a1ca-4c29-945b-592321d95f9d/images/c872e033-b13a-4436-b503-0632b5d437ae.png)
## Tools
+ **AWS DMS** - [AWS Database Migration Services](https://docs.aws.amazon.com/dms/latest/userguide/) (AWS DMS) helps you migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud, between on-premises instances (through an AWS Cloud setup), or between combinations of cloud and on-premises setups.
+ **AWS SCT** - [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) (AWS SCT) is used to convert your database schema from one database engine to another. The custom code that the tool converts includes views, stored procedures, and functions. Any code that the tool cannot convert automatically is clearly marked so that you can convert it yourself.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database version and engine. | | DBA |
| Identify the hardware requirements for the target server instance. | | DBA, SysAdmin |
| Identify the storage requirements (storage type and capacity). | | DBA, SysAdmin |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, SysAdmin |
| Identify the network access security requirements for the source and target databases. | | DBA, SysAdmin |
| Identify the application migration strategy. | | DBA, SysAdmin, App owner |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC) and subnets. | | SysAdmin |
| Create the security groups and network access control lists (ACLs). | | SysAdmin |
| Configure and start an Amazon RDS DB instance. | | DBA, SysAdmin |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the database schema by using AWS SCT. | | DBA |
| Migrate data by using AWS DMS. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Use AWS SCT to analyze and convert the SQL code inside the application code. | For more information, see https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP\$1Converting.App.html. | App owner |
| Follow the application migration strategy. | | DBA, SysAdmin, App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | | DBA, SysAdmin, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary AWS resources. | | DBA, SysAdmin |
| Review and validate the project documents. | | DBA, SysAdmin |
| Gather metrics around time to migrate, % of manual vs. tool, cost savings, etc. | | DBA, SysAdmin |
| Close out the project and provide feedback. | | |
## Related resources
**References**
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon RDS Pricing](https://aws.amazon.com/rds/pricing/)
**Tutorial and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [AWS DMS (video)](https://www.youtube.com/watch?v=zb4GcjEdl8U)
+ [Amazon RDS (video)](https://www.youtube.com/watch?v=igRfulrrYCo)
# Migrate an on-premises Oracle database to Amazon RDS for PostgreSQL by using an Oracle bystander and AWS DMS
*Cady Motyka, Amazon Web Services*
## Summary
This pattern describes how you can migrate an on-premises Oracle database to either of the following PostgreSQL-compatible AWS database services with minimal downtime:
+ Amazon Relational Database Service (Amazon RDS) for PostgreSQL
+ Amazon Aurora PostgreSQL-Compatible Edition
The solution uses AWS Database Migration Service (AWS DMS) to migrate the data, AWS Schema Conversion Tool (AWS SCT) to convert the database schema, and an Oracle bystander database to help manage the migration. In this implementation, the downtime is restricted to however long it takes to create or validate all of the foreign keys on the database.
The solution also uses Amazon Elastic Compute Cloud (Amazon EC2) instances with an Oracle bystander database to help control the stream of data through AWS DMS. You can temporarily pause streaming replication from the on-premises Oracle database to the Oracle bystander to activate AWS DMS to catch up on data validation, or to use another data validation tool. The Amazon RDS for PostgreSQL DB instance or Aurora PostgreSQL-Compatible DB instance and the bystander database will have the same data when AWS DMS finishes migrating current changes.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Oracle database in an on-premises data center with an Active Data Guard standby database configured
+ AWS Direct Connect configured between the on-premises data center and AWS Secrets Manager for storing the database secrets
+ Java Database Connectivity (JDBC) drivers for AWS SCT connectors, installed either on a local machine or on the EC2 instance where AWS SCT is installed
+ Familiarity with [using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ AWS DMS supports all Oracle database editions for versions 10.2 and later (for versions 10.x), 11g and up to 12.2, 18c, and 19c. For the latest list of supported versions, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support. For information about Oracle database versions supported by AWS SCT, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html).
+ AWS DMS supports PostgreSQL versions 9.4 and later (for versions 9.x), 10.x, 11.x, 12.x, and 13.x. For the latest information, see [Using a PostgreSQL Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html) in the AWS documentation.
## Architecture
**Source technology stack**
+ An on-premises Oracle database
+ An EC2 instance that holds a bystander for the Oracle database
**Target technology stack**
+ Amazon RDS for PostgreSQL or Aurora PostgreSQL instance, PostgreSQL 9.3 and later
**Target architecture**
The following diagram shows an example workflow for migrating an Oracle database to a PostgreSQL-compatible AWS database by using AWS DMS and an Oracle bystander:
![\[Migrating an on-premises Oracle database to PostgreSQL on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6f5d5500-8b09-4bd1-8ef9-e670d58d07f8/images/1de98abd-c143-481a-b55f-e8d00eb96a38.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
## Epics
### Convert the Oracle database schema to PostgreSQL
| Task | Description | Skills required |
| --- | --- | --- |
| Set up AWS SCT. | Create a new report, and connect to Oracle as the source and PostgreSQL as the target. In **Project Settings**, go to the **SQL Scripting** tab. Change the **Target SQL Script** to **Multiple Files**. These files will be used later and named the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.html) | DBA |
| Convert the Oracle database schema. | In the **Action** tab, choose **Generate Report**. Then, choose **Convert Schema** and choose **Save as SQL**. | DBA |
| Modify the scripts. | For example, you might want to modify the script if a number in the source schema has been converted to numeric format in PostgreSQL, but you want to use **BIGINT **instead for better performance. | DBA |
### Create and configure the Amazon RDS DB instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Amazon RDS DB instance. | In the correct AWS Region, create a new PostgreSQL DB instance. For more information, see [Creating a PostgreSQL DB instance and connecting to a database on a PostgreSQL DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html) in the Amazon RDS documentation. | AWS SysAdmin, DBA |
| Configure DB instance specifications. | Specify the DB engine version, DB instance class, Multi-AZ deployment, storage type, and allocated storage. Enter the DB instance identifier, a primary user name, and a primary password. | AWS SysAdmin, DBA |
| Configure network and security. | Specify the virtual private cloud (VPC), subnet group, public accessibility, Availability Zone preference, and security groups. | DBA, SysAdmin |
| Configure database options. | Specify the database name, port, parameter group, encryption, and KMS key. | AWS SysAdmin, DBA |
| Configure backups. | Specify the backup retention period, backup window, start time, duration, and whether to copy tags to snapshots. | AWS SysAdmin, DBA |
| Configure monitoring options. | Activate or deactivate enhanced monitoring and performance insights. | AWS SysAdmin, DBA |
| Configure maintenance options. | Specify auto minor version upgrade, maintenance window, and start day, time, and duration. | AWS SysAdmin, DBA |
| Run the pre-migration scripts from AWS SCT. | On the Amazon RDS instance, run the following scripts generated by AWS SCT:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.html) | AWS SysAdmin, DBA |
### Configure the Oracle bystander in Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the network for Amazon EC2. | Create the new VPC, subnets, internet gateway, route tables, and security groups. | AWS SysAdmin |
| Create the EC2 instance. | In the appropriate AWS Region, create a new EC2 instance. Select the Amazon Machine Image (AMI), choose the instance size, and configure instance details: number of instances (1), the VPC and subnet you created in the previous task, auto-assign public IP, and other options. Add storage, configure security groups, and launch. When prompted, create and save a key pair for the next step. | AWS SysAdmin |
| Connect the Oracle source database to the EC2 instance. | Copy the IPv4 public IP address and DNS to a text file and connect by using SSH as follows: **ssh -i "your\$1file.pem" ec2-user@**. | AWS SysAdmin |
| Set up the initial host for a bystander in Amazon EC2. | Set up SSH keys, bash profile, ORATAB, and symbolic links. Create Oracle directories. | AWS SysAdmin, Linux Admin |
| Set up the database copy for a bystander in Amazon EC2 | Use RMAN to create a database copy, enable supplemental logging, and create the standby control file. After copying is complete, place the database in recovery mode. | AWS SysAdmin, DBA |
| Set up Oracle Data Guard. | Modify your **listener.ora** file and start the listener. Set up a new archive destination. Place the bystander in recovery mode, replace temporary files to avoid future corruption, install a crontab if necessary to prevent the archive directory from running out of space, and edit the **manage-trclog-files-oracle.cfg** file for the source and standby. | AWS SysAdmin, DBA |
| Prep the Oracle database to sync shipping. | Add the standby log files and change the recovery mode. Change the log shipping to **SYNC AFFIRM** on both the source primary and the source standby. Switch logs on primary, confirm via the Amazon EC2 bystander alert log that you are using the standby log files, and confirm that the redo stream is flowing in SYNC. | AWS SysAdmin, DBA |
### Migrate data with AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance in AWS DMS. | Complete the fields for the name, instance class, VPC (same as the Amazon EC2 instance), Multi-AZ, and public accessibility. Under **Advance**, specify allocated storage, subnet group, Availability Zone, VPC security groups, and AWS Key Management Service (AWS KMS) key. | AWS SysAdmin, DBA |
| Create the source database endpoint. | Specify the endpoint name, type, source engine (Oracle), server name (Amazon EC2 private DNS name), port, SSL mode, user name, password, SID, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose **Run Test**, and then create the endpoint. You can also configure the following advanced settings: **maxFileSize** and **numberDataTypeScale**. | AWS SysAdmin, DBA |
| Connect AWS DMS to Amazon RDS for PostgreSQL. | Create a migration security group for connections across VPCs. | AWS SysAdmin, DBA |
| Create the target database endpoint. | Specify the endpoint name, type, source engine (PostgreSQL), server name (Amazon RDS endpoint), port, SSL mode, user name, password, database name, VPC (specify the VPC that has the replication instance), and replication instance. To test the connection, choose **Run Test**, and then create the endpoint. You can also configure the following advanced settings: **maxFileSize **and **numberDataTypeScale**. | AWS SysAdmin, DBA |
| Create the AWS DMS replication task. | Specify the task name, replication instance, source and target endpoints, and replication instance. For migration type, choose **Migrate existing data and replicate ongoing changes**. Clear the **Start task on create** checkbox. | AWS SysAdmin, DBA |
| Configure the AWS DMS replication task settings. | For target table preparation mode, choose **Do nothing**. Stop task after full load completes (to create primary keys). Specify limited or full LOB mode, and activate control tables. Optionally, you can configure the **CommitRate** advance setting. | DBA |
| Configure table mappings. | In the **Table mappings** section, create an **Include** rule for all tables in all schemas included in the migration, and then create an **Exclude** rule. Add three transformation rules to convert the schema, table, and column names to lowercase, and add any other rules needed for this specific migration. | DBA |
| Start the task. | Start the replication task. Make sure that the full load is running. Run **ALTER SYSTEM SWITCH LOGFILE** on the primary Oracle database to kick-start the task. | DBA |
| Run the mid-migration scripts from AWS SCT. | In Amazon RDS for PostgreSQL, run the following scripts generated by AWS SCT: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.html) | DBA |
| Restart the task to continue change data capture (CDC). | Run **VACUUM **on the Amazon RDS for PostgreSQL DB instance, and restart the AWS DMS task to apply cached CDC changes. | DBA |
### Cut over to the PostgreSQL database
| Task | Description | Skills required |
| --- | --- | --- |
| Review the AWS DMS logs and validation tables for any errors. | Check and fix any replication or validation errors. | DBA |
| Stop all Oracle dependencies. | Stop all Oracle dependencies, shut down listeners on the Oracle database, and run **ALTER SYSTEM SWITCH LOGFILE**. Stop the AWS DMS task when it shows no activity. | DBA |
| Run the post-migration scripts from AWS SCT. | In Amazon RDS for PostgreSQL, run the following scripts generated by AWS SCT:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.html) | DBA |
| Complete additional Amazon RDS for PostgreSQL steps. | Increment sequences to match Oracle if needed, run **VACUUM** and **ANALYZE**, and take a snapshot for compliance. | DBA |
| Open the connections to Amazon RDS for PostgreSQL. | Remove the AWS DMS security groups from Amazon RDS for PostgreSQL, add production security groups, and point your applications to the new database. | DBA |
| Clean up AWS DMS objects. | Remove the endpoints, replication tasks, replication instances, and the EC2 instance. | SysAdmin, DBA |
## Related resources
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon RDS for PostgreSQL pricing](https://aws.amazon.com/rds/postgresql/pricing/)
# Migrate an Oracle Database to Amazon Redshift using AWS DMS and AWS SCT
*Piyush Goyal and Brian motzer, Amazon Web Services*
## Summary
This pattern provides guidance for migrating Oracle databases to an Amazon Redshift cloud data warehouse in the Amazon Web Services (AWS) Cloud by using AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT). The pattern covers source Oracle databases that are on premises or installed on an Amazon Elastic Compute Cloud (Amazon EC2) instance. It also covers Amazon Relational Database Service (Amazon RDS) for Oracle databases.
## Prerequisites and limitations
**Prerequisites **
+ An Oracle database that is running in an on-premises data center or in the AWS Cloud
+ An active AWS account
+ Familiarity with [using an Oracle database as a source for AWS DMS](http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [using an Amazon Redshift database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Redshift.html)
+ Knowledge of Amazon RDS, Amazon Redshift, the applicable database technologies, and SQL
+ Java Database Connectivity (JDBC) drivers for AWS SCT connectors, where AWS SCT is installed
**Product versions**
+ For self-managed Oracle databases, AWS DMS supports all Oracle database editions for versions 10.2 and later (for versions 10.*x*), 11g and up to 12.2, 18c, and 19c. For Amazon RDS for Oracle databases that AWS manages, AWS DMS supports all Oracle database editions for versions 11g (versions 11.2.0.4 and later) and up to 12.2, 18c, and 19c. We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support.
## Architecture
**Source technology stack **
One of the following:
+ An on-premises Oracle database
+ An Oracle database on an EC2 instance
+ An Amazon RDS for Oracle DB instance
**Target technology stack **
+ Amazon Redshift
**Target architecture **
*From an Oracle database running in the AWS Cloud to Amazon Redshift:*
![\[Migrating an Oracle database in the AWS Cloud to an Amazon Redshift data warehouse.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/22807be0-c7e0-49c6-8923-7d23bf83a50d/images/7140e819-81d6-45c4-805b-8e10828076a7.png)
*From an Oracle database running in an on-premises data center to Amazon Redshift:*
![\[Migrating an on-premises Oracle database to an Amazon Redshift data warehouse.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/22807be0-c7e0-49c6-8923-7d23bf83a50d/images/d6654b48-0e1b-4b01-a261-5a640be01fd7.png)
## Tools
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) - AWS Data Migration Service (AWS DMS) helps you migrate databases to AWS quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. AWS DMS can migrate your data to and from most widely used commercial and open-source databases.
+ [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) - AWS Schema Conversion Tool (AWS SCT) can be used to convert your existing database schema from one database engine to another. It supports various database engines, including Oracle, SQL Server, and PostgresSQL, as sources.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the database versions. | Validate the source and target database versions and make sure they are supported by AWS DMS. For information about supported Oracle Database versions, see [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). For information about using Amazon Redshift as a target, see [Using an Amazon Redshift database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Redshift.html). | DBA |
| Create a VPC and security group. | In your AWS account, create a virtual private cloud (VPC), if it doesn’t exist. Create a security group for outbound traffic to source and target databases. For more information, see the [Amazon Virtual Private Cloud (Amazon VPC) documentation](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html). | Systems administrator |
| Install AWS SCT. | Download and install the latest version of AWS SCT and its corresponding drivers. For more information, see [Installing, verifying, and updating the AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html). | DBA |
| Create a user for the AWS DMS task. | Create an AWS DMS user in the source database and grant it READ privileges. This user will be used by both AWS SCT and AWS DMS. | DBA |
| Test the DB connectivity. | Test the connectivity to the Oracle DB instance. | DBA |
| Create a new project in AWS SCT. | Open the AWS SCT tool and create a new project. | DBA |
| Analyze the Oracle schema to be migrated. | Use AWS SCT to analyze the schema to be migrated, and generate a database migration assessment report. For more information, see [Creating a database migration assessment report](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_AssessmentReport.Create.html) in the AWS SCT documentation. | DBA |
| Review the assessment report. | Review the report for migration feasibility. Some DB objects might require manual conversion. For more information about the report, see [Viewing the assessment report](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_AssessmentReport.View.html) in the AWS SCT documentation. | DBA |
### Prepare the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon Redshift cluster. | Create an Amazon Redshift cluster within the VPC that you created previously. For more information, see [Amazon Redshift clusters](https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-clusters.html) in the Amazon Redshift documentation. | DBA |
| Create database users. | Extract the list of users, roles, and grants from the Oracle source database. Create users in the target Amazon Redshift database and apply the roles from the previous step. | DBA |
| Evaluate database parameters. | Review the database options, parameters, network files, and database links from the Oracle source database, and evaluate their applicability to the target. | DBA |
| Apply any relevant settings to the target. | For more information about this step, see [Configuration reference](https://docs.aws.amazon.com/redshift/latest/dg/cm_chap_ConfigurationRef.html) in the Amazon Redshift documentation. | DBA |
### Create objects in the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS DMS user in the target database. | Create an AWS DMS user in the target database and grant it read and write privileges. Validate the connectivity from AWS SCT. | DBA |
| Convert the schema, review the SQL report, and save any errors or warnings. | For more information, see [Converting database schemas using the AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Converting.html) in the AWS SCT documentation. | DBA |
| Apply the schema changes to the target database or save them as a .sql file. | For instructions, see [Saving and applying your converted schema in the AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Converting.DW.html#CHAP_Converting.DW.SaveAndApply) in the AWS SCT documentation. | DBA |
| Validate the objects in the target database. | Validate the objects that were created in the previous step in the target database. Rewrite or redesign any objects that weren’t successfully converted. | DBA |
| Disable foreign keys and triggers. | Disable any foreign key and triggers. These can cause data loading issues during the full load process when running AWS DMS. | DBA |
### Migrate data using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS DMS replication instance. | Sign in to the AWS Management Console, and open the AWS DMS console. In the navigation pane, choose **Replication instances**, **Create replication instance**. For detailed instructions, see [step 1](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html#CHAP_GettingStarted.ReplicationInstance) in *Getting started with AWS DMS* in the AWS DMS documentation. | DBA |
| Create source and target endpoints. | Create source and target endpoints, Test the connection from the replication instance to both source and target endpoints. For detailed instructions, see [step 2](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html#CHAP_GettingStarted.Endpoints) in *Getting started with AWS DMS* in the AWS DMS documentation. | DBA |
| Create a replication task. | Create a replication task and select the appropriate migration method. For detailed instructions, see [step 3](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html#CHAP_GettingStarted.Tasks) in *Getting started with AWS DMS* in the AWS DMS documentation. | DBA |
| Start the data replication. | Start the replication task and monitor the logs for any errors. | DBA |
### Migrate your application
| Task | Description | Skills required |
| --- | --- | --- |
| Create application servers. | Create the new application servers on AWS. | Application owner |
| Migrate the application code. | Migrate the application code to the new servers. | Application owner |
| Configure the application server. | Configure the application server for the target database and drivers. | Application owner |
| Optimize the application code. | Optimize the application code for the target engine. | Application owner |
### Cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Validate users. | In the target Amazon Redshift database, validate users and grant them roles and privileges. | DBA |
| Validate that the application is locked. | Make sure that the application is locked, to prevent further changes. | Application owner |
| Validate the data. | Validate the data in the target Amazon Redshift database. | DBA |
| Enable foreign keys and triggers. | Enable foreign keys and triggers in the target Amazon Redshift database. | DBA |
| Connect to the new database. | Configure the application to connect to the new Amazon Redshift database. | Application owner |
| Perform final checks. | Perform a final, comprehensive system check before going live. | DBA, Application owner |
| Go live. | Go live with the target Amazon Redshift database. | DBA |
### Close the migration project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary AWS resources. | Shut down temporary AWS resources such as the AWS DMS replication instance and the EC2 instance used for AWS SCT. | DBA, Systems administrator |
| Review documents. | Review and validate the migration project documents. | DBA, Systems administrator |
| Gather metrics. | Collect information about the migration project, such as the time to migrate, the percentage of manual versus tool tasks, and total cost savings. | DBA, Systems administrator |
| Close out the project. | Close out the project and provide feedback. | DBA, Systems administrator |
## Related resources
**References**
+ [AWS DMS user guide](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [AWS SCT user guide](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon Redshift getting started guide](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html)
**Tutorials and videos**
+ [Dive Deep into AWS SCT and AWS DMS](https://www.youtube.com/watch?v=kJs9U4ys5FE) (presentation from AWS re:Invent 2019)
+ [Getting Started with AWS Database Migration Service](https://aws.amazon.com/dms/getting-started/)
# Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT
*Senthil Ramasamy, Amazon Web Services*
## Summary
This pattern describes how to migrate an Oracle database to Amazon Aurora PostgreSQL-Compatible Edition by using AWS Data Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT).
The pattern covers source Oracle databases that are on premises, Oracle databases that are installed on Amazon Elastic Compute Cloud (Amazon EC2) instances, and Amazon Relational Database Service (Amazon RDS) for Oracle databases. The pattern converts these databases to Aurora PostgreSQL-Compatible.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An Oracle database in an on-premises data center or in the AWS Cloud.
+ SQL clients installed either on a local machine or on an EC2 instance.
+ Java Database Connectivity (JDBC) drivers for AWS SCT connectors, installed on either a local machine or an EC2 instance where AWS SCT is installed.
**Limitations**
+ Database size limit: 128 TB
+ If the source database supports a commercial off-the-shelf (COTS) application or is vendor-specific, you might not be able to convert it to another database engine. Before using this pattern, confirm that the application supports Aurora PostgreSQL-Compatible.
**Product versions**
+ For self-managed Oracle databases, AWS DMS supports all Oracle database editions for versions 10.2 and later (for versions 10.x), 11g, and up to 12.2, 18c, and 19c. For the latest list of supported Oracle database versions (both self-managed and Amazon RDS for Oracle), see [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html).
+ We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support. For information about Oracle database versions supported by AWS SCT, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html).
+ Aurora supports the PostgreSQL versions listed in [Amazon Aurora PostgreSQL releases and engine versions](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Updates.20180305.html).
## Architecture
**Source technology stack**
One of the following:
+ An on-premises Oracle database
+ An Oracle database on an EC2 instance
+ An Amazon RDS for Oracle DB instance
**Target technology stack**
+ Aurora PostgreSQL-Compatible
**Target architecture**
![\[Target architecture for migrating Oracle databases to Aurora PostgreSQL-Compatible.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6de157c4-dcc9-4186-ae32-17efbbbee709/images/68beb634-926e-4908-97b1-edcd23e06a2b.png)
**Data migration architecture**
+ From an Oracle database running in the AWS Cloud
![\[Data migration architecture for an Oracle database on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6de157c4-dcc9-4186-ae32-17efbbbee709/images/7fc32019-3db1-485b-93e5-6d5539be048c.png)
+ From an Oracle database running in an on-premises data center
![\[Data migration architecture for an Oracle database in an on-premises data center.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6de157c4-dcc9-4186-ae32-17efbbbee709/images/c70d8774-aef7-4414-9766-ce8f25757c4b.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format compatible with the target database.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the source database. | To prepare the source database, see [Using Oracle Database as a source for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html) in the AWS SCT documentation. | DBA |
| Create an EC2 instance for AWS SCT. | Create and configure an EC2 instance for AWS SCT, if required. | DBA |
| Download AWS SCT. | Download the latest version of AWS SCT and associated drivers. For more information, see [Installing, verifying, and updating AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html) in the AWS SCT documentation. | DBA |
| Add users and permissions. | Add and validate the prerequisite users and permissions in the source database. | DBA |
| Create an AWS SCT project. | Create an AWS SCT project for the workload, and connect to the source database. For instructions, see [Creating an AWS SCT project](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.Project) and [Adding database servers](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.AddServers) in the AWS SCT documentation. | DBA |
| Evaluate feasibility. | Generate an assessment report, which summarizes action items for schemas that can’t be converted automatically and provides estimates for manual conversion efforts. For more information, see [Creating and reviewing the database migration assessment report](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.AssessmentReport) in the AWS SCT documentation. | DBA |
### Prepare the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create a target Amazon RDS DB instance. | Create a target Amazon RDS DB instance, using Amazon Aurora as the database engine. For instructions, see [Creating an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html) in the Amazon RDS documentation. | DBA |
| Extract users, roles, and permissions. | Extract the list of users, roles, and permissions from the source database. | DBA |
| Map users. | Map the existing database users to the new database users. | App owner |
| Create users. | Create users in the target database. | DBA, App owner |
| Apply roles. | Apply roles from the previous step to the target database. | DBA |
| Check options, parameters, network files, and database links. | Review the source database for options, parameters, network files, and database links, and then evaluate their applicability to the target database. | DBA |
| Apply settings. | Apply any relevant settings to the target database. | DBA |
### Transfer objects
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS SCT connectivity. | Configure AWS SCT connectivity to the target database. | DBA |
| Convert the schema using AWS SCT. | AWS SCT automatically converts the source database schema and most of the custom code to a format that is compatible with the target database. Any code that the tool cannot convert automatically is clearly marked so that you can convert it manually. | DBA |
| Review the report. | Review the generated SQL report and save any errors and warnings. | DBA |
| Apply automated schema changes. | Apply automated schema changes to the target database or save them as a .sql file. | DBA |
| Validate objects. | Validate that AWS SCT created the objects on the target. | DBA |
| Handle items that weren't converted. | Manually rewrite, reject, or redesign any items that failed to convert automatically. | DBA, App owner |
| Apply role and user permissions. | Apply the generated role and user permissions and review any exceptions. | DBA |
### Migrate the data
| Task | Description | Skills required |
| --- | --- | --- |
| Determine the method. | Determine the method for migrating data. | DBA |
| Create a replication instance. | Create a replication instance from the AWS DMS console. For more information, see [Working with an AWS DMS replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.html) in the AWS DMS documentation. | DBA |
| Create the source and target endpoints. | To create endpoints, follow the instructions in [Creating source and target endpoints in the AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html) documentation. | DBA |
| Create a replication task. | To create a task, see [Working with AWS DMS tasks](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.html) in the AWS DMS documentation. | DBA |
| Start the replication task and monitor the logs. | For more information about this step, see [Monitoring AWS DMS tasks](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html) in the AWS DMS documentation. | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Analyze and convert SQL items in the application code. | Use AWS SCT to analyze and convert the SQL items in the application code. When you convert your database schema from one engine to another, you also need to update the SQL code in your applications to interact with the new database engine instead of the old one. You can view, analyze, edit, and save the converted SQL code. | App owner |
| Create application servers. | Create the new application servers on AWS. | App owner |
| Migrate the application code. | Migrate the application code to the new servers. | App owner |
| Configure the application servers. | Configure the application servers for the target database and drivers. | App owner |
| Fix code. | Fix any code that’s specific to the source database engine in your application. | App owner |
| Optimize code. | Optimize your application code for the target database engine. | App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Cut over to the target database. | Perform the cutover to the new database. | DBA |
| Lock the application. | Lock the application from any further changes. | App owner |
| Validate changes. | Validate that all changes were propagated to the target database. | DBA |
| Redirect to the target database. | Point the new application servers to the target database. | App owner |
| Check everything. | Perform a final, comprehensive system check. | App owner |
| Go live. | Complete final cutover tasks. | App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary resources. | Shut down the temporary AWS resources such as the AWS DMS replication instance and the EC2 instance used for AWS SCT. | DBA, App owner |
| Update feedback. | Update feedback on the AWS DMS process for internal teams. | DBA, App owner |
| Revise process and templates. | Revise the AWS DMS process and improve the template if necessary. | DBA, App owner |
| Validate documents. | Review and validate the project documents. | DBA, App owner |
| Gather metrics. | Gather metrics to evaluate the time to migrate, percent of manual versus tool cost savings, and so on. | DBA, App owner |
| Close the project. | Close the migration project and provide feedback to stakeholders. | DBA, App owner |
## Related resources
**References**
+ [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Using a PostgreSQL Database as a Target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
+ [Oracle Database 11g/12c to Amazon Aurora with PostgreSQL Compatibility (9.6.x) Migration Playbook](https://d1.awsstatic.com/whitepapers/Migration/oracle-database-amazon-aurora-postgresql-migration-playbook.pdf)
+ [Oracle Database 19c to Amazon Aurora with PostgreSQL Compatibility (12.4) Migration Playbook](https://d1.awsstatic.com/whitepapers/Migration/oracle-database-amazon-aurora-postgresql-migration-playbook-12.4.pdf)
+ [Migrating an Amazon RDS for Oracle database to Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/dms/latest/sbs/chap-oracle-postgresql.html)
+ [AWS Data Migration Service](https://aws.amazon.com/dms/)
+ [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Migrate from Oracle to Amazon Aurora](https://aws.amazon.com/getting-started/projects/migrate-oracle-to-amazon-aurora/)
+ [Amazon RDS pricing](https://aws.amazon.com/rds/pricing/)
**Tutorials and videos**
+ [Database Migration Step-by-Step Walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html)
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [AWS Data Migration Service](https://www.youtube.com/watch?v=zb4GcjEdl8U) (video)
+ [Migrating an Oracle database to PostgreSQL](https://www.youtube.com/watch?v=ibtNkChGFkw) (video)
## Additional information
.
# Migrate data from an on-premises Oracle database to Aurora PostgreSQL
*Michelle Deng and Shunan Xiang, Amazon Web Services*
## Summary
This pattern provides guidance for data migration from an on-premises Oracle database to Amazon Aurora PostgreSQL-Compatible Edition. It targets an online data migration strategy with a minimal amount of downtime for multi-terabyte Oracle databases that contain large tables with high data manipulation language (DML) activities. An Oracle Active Data Guard standby database is used as the source to offload data migration from the primary database. The replication from the Oracle primary database to standby can be suspended during the full load to avoid ORA-01555 errors.
Table columns in primary keys (PKs) or foreign keys (FKs), with data type NUMBER, are commonly used to store integers in Oracle. We recommend that you convert these to INT or BIGINT in PostgreSQL for better performance. You can use the AWS Schema Conversion Tool (AWS SCT) to change the default data type mapping for PK and FK columns. (For more information, see the AWS blog post [Convert the NUMBER data type from Oracle to PostgreSQL](https://aws.amazon.com/blogs/database/convert-the-number-data-type-from-oracle-to-postgresql-part-2/).) The data migration in this pattern uses AWS Database Migration Service (AWS DMS) for both full load and change data capture (CDC).
You can also use this pattern to migrate an on-premises Oracle database to Amazon Relational Database Service (Amazon RDS) for PostgreSQL, or an Oracle database that's hosted on Amazon Elastic Compute Cloud (Amazon EC2) to either Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An Oracle source database in an on-premises data center with Active Data Guard standby configured
+ AWS Direct Connect configured between the on-premises data center and the AWS Cloud
+ Familiarity with [using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html)
**Limitations**
+ Amazon Aurora database clusters can be created with up to 128 TiB of storage. Amazon RDS for PostgreSQL database instances can be created with up to 64 TiB of storage. For the latest storage information, see [Amazon Aurora storage and reliability](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Overview.StorageReliability.html) and [Amazon RDS DB instance storage](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html) in the AWS documentation.
**Product versions**
+ AWS DMS supports all Oracle database editions for versions 10.2 and later (for versions 10.x), 11g and up to 12.2, 18c, and 19c. For the latest list of supported versions, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) in the AWS documentation.
## Architecture
**Source technology stack**
+ On-premises Oracle databases with Oracle Active Data Guard standby configured
**Target technology stack**
+ Aurora PostgreSQL-Compatible
**Data migration architecture**
![\[Migrating an Oracle database to Aurora PostgreSQL-Compatible\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/49f9b03e-6d33-4ac0-94ad-d3e6d02e6d63/images/0038a36b-fb7d-4f2d-8376-8d38290b0736.png)
## Tools
+ **AWS DMS** - [AWS Database Migration Service](https://docs.aws.amazon.com/dms/index.html) (AWS DMS) supports several source and target databases. See [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) in the AWS DMS documentation for a list of supported Oracle source and target database versions and editions. If the source database is not supported by AWS DMS, you must select another method for migrating the data in Phase 6 (in the *Epics *section). **Important note:** Because this is a heterogeneous migration, you must first check to see whether the database supports a commercial off-the-shelf (COTS) application. If the application is COTS, consult the vendor to confirm that Aurora PostgreSQL-Compatible is supported before proceeding. For more information, see [AWS DMS Step-by-Step Migration Walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html) in the AWS documentation.
+ **AWS SCT** - The [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/Welcome.htm) (AWS SCT) facilitates heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that's compatible with the target database. The custom code that the tool converts includes views, stored procedures, and functions. Any code that the tool cannot convert automatically is clearly marked so that you can convert it yourself.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions. | | DBA |
| Install AWS SCT and drivers. | | DBA |
| Add and validate the AWS SCT prerequisite users and grants-source database. | | DBA |
| Create an AWS SCT project for the workload, and connect to the source database. | | DBA |
| Generate an assessment report and evaluate feasibility. | | DBA, App owner |
### Prepare the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Aurora PostgreSQL-Compatible target database. | | DBA |
| Extract users, roles, and grants list from the source database. | | DBA |
| Map the existing database users to the new database users. | | App owner |
| Create users in the target database. | | DBA |
| Apply roles from the previous step to the target Aurora PostgreSQL-Compatible database. | | DBA |
| Review database options, parameters, network files, and database links from the source database, and evaluate their applicability to the target database. | | DBA, App owner |
| Apply any relevant settings to the target database. | | DBA |
### Prepare for database object code conversion
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS SCT connectivity to the target database. | | DBA |
| Convert the schema in AWS SCT, and save the converted code as a .sql file. | | DBA, App owner |
| Manually convert any database objects that failed to convert automatically. | | DBA, App owner |
| Optimize the database code conversion. | | DBA, App owner |
| Separate the .sql file into multiple .sql files based on the object type. | | DBA, App owner |
| Validate the SQL scripts in the target database. | | DBA, App owner |
### Prepare for data migration
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS DMS replication instance. | | DBA |
| Create the source and target endpoints. | If the data type of the PKs and FKs is converted from NUMBER in Oracle to BIGINT in PostgreSQL, consider specifying the connection attribute `numberDataTypeScale=-2` when you create the source endpoint. | DBA |
### Migrate data – full load
| Task | Description | Skills required |
| --- | --- | --- |
| Create the schema and tables in the target database. | | DBA |
| Create AWS DMS full-load tasks by either grouping tables or splitting a big table based on the table size. | | DBA |
| Stop the applications on the source Oracle databases for a short period. | | App owner |
| Verify that the Oracle standby database is synchronous with the primary database, and stop the replication from the primary database to the standby database. | | DBA, App owner |
| Start applications on the source Oracle database. | | App owner |
| Start the AWS DMS full-load tasks in parallel from the Oracle standby database to the Aurora PostgreSQL-Compatible database. | | DBA |
| Create PKs and secondary indexes after the full load is complete. | | DBA |
| Validate the data. | | DBA |
### Migrate data – CDC
| Task | Description | Skills required |
| --- | --- | --- |
| Create AWS DMS ongoing replication tasks by specifying a custom CDC start time or system change number (SCN) when the Oracle standby was synchronized with the primary database, and before the applications were restarted in the previous task. | | DBA |
| Start AWS DMS tasks in parallel to replicate ongoing changes from the Oracle standby database to the Aurora PostgreSQL-Compatible database. | | DBA |
| Re-establish the replication from the Oracle primary database to the standby database. | | DBA |
| Monitor the logs and stop the applications on the Oracle database when the target Aurora PostgreSQL-Compatible database is almost synchronous with the source Oracle database. | | DBA, App owner |
| Stop the AWS DMS tasks when the target is fully synchronized with the source Oracle database. | | DBA |
| Create FKs and validate the data in the target database. | | DBA |
| Create functions, views, triggers, sequences, and other object types in the target database. | | DBA |
| Apply role grants in the target database. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Use AWS SCT to analyze and convert the SQL statements inside the application code. | | App owner |
| Create new application servers on AWS. | | App owner |
| Migrate the application code to the new servers. | | App owner |
| Configure the application server for the target database and drivers. | | App owner |
| Fix any code that's specific to the source database engine in the application. | | App owner |
| Optimize the application code for the target database. | | App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Point the new application server to the target database. | | DBA, App owner |
| Perform sanity checks. | | DBA, App owner |
| Go live. | | DBA, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary AWS resources. | | DBA, Systems administrator |
| Review and validate the project documents. | | DBA, App owner |
| Gather metrics for time to migrate, percentage of manual versus tool use, cost savings, and similar data. | | DBA, App owner |
| Close out the project and provide feedback. | | DBA, App owner |
## Related resources
**References**
+ [Oracle Database to Aurora PostgreSQL-Compatible: Migration Playbook](https://d1.awsstatic.com/whitepapers/Migration/oracle-database-amazon-aurora-postgresql-migration-playbook.pdf)
+ [Migrating an Amazon RDS for Oracle Database to Amazon Aurora MySQL](https://docs.aws.amazon.com/dms/latest/sbs/chap-rdsoracle2aurora.html)
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [AWS SCT website](https://aws.amazon.com/dms/schema-conversion-tool/)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Migrate from Oracle to Amazon Aurora](https://aws.amazon.com/getting-started/projects/migrate-oracle-to-amazon-aurora/)
**Tutorials**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [AWS Database Migration Service Step-by-Step Walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/dms-sbs-welcome.html)
# Migrate from SAP ASE to Amazon RDS for SQL Server using AWS DMS
*Amit Kumar, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an SAP Adaptive Server Enterprise (ASE) database to an Amazon Relational Database Service (Amazon RDS) DB instance that's running Microsoft SQL Server. The source database can be located in an on-premises data center or on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The pattern uses AWS Database Migration Service (AWS DMS) to migrate data and (optionally) computer-aided software engineering (CASE) tools to convert the database schema.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An SAP ASE database in an on-premises data center or on an EC2 instance
+ A target Amazon RDS for SQL Server database that’s up and running
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ SAP ASE version 15.7 or 16.x only. For the latest information, see [Using an SAP Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SAP.html).
+ For Amazon RDS target databases, AWS DMS supports [Microsoft SQL Server versions on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.VersionSupport) for the Enterprise, Standard, Web, and Express editions. For the latest information about supported versions, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.SQLServer.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support.
## Architecture
**Source technology stack **
+ An SAP ASE database that's on premises or on an Amazon EC2 instance
**Target technology stack **
+ An Amazon RDS for SQL Server DB instance
**Source and target architecture**
*From an SAP ASE database on Amazon EC2 to an Amazon RDS for SQL Server DB instance:*
![\[Target architecture for SAP ASE on Amazon EC2 to Amazon RDS for SQL Server\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5ca697a2-9ca3-4231-b457-c1dc59ada5f1/images/957bdcf0-ab58-4b6d-a71a-d0ecbc31822c.png)
*From an on-premises SAP ASE database to an Amazon RDS for SQL Server DB instance:*
![\[Target architecture for on-premises SAP ASE to Amazon RDS for SQL Server\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5ca697a2-9ca3-4231-b457-c1dc59ada5f1/images/65aab2f5-0e63-4c34-97e2-cd4ac23751a4.png)
## Tools
+ [AWS Database Migration Service](https://docs.aws.amazon.com/dms/) (AWS DMS) is a web service you can use to migrate data from your database that is on-premises, on an Amazon RDS DB instance, or in a database on an EC2 instance, to a database on an AWS service such as Amazon RDS for SQL Server or an EC2 instance. You can also migrate a database from an AWS service to an on-premises database. You can migrate data between heterogeneous or homogeneous database engines.
+ For schema conversions, you can optionally use [erwin Data Modeler](https://erwin.com/products/erwin-data-modeler/) or [SAP PowerDesigner](https://www.sap.com/products/technology-platform/powerdesigner-data-modeling-tools.html).
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions. | | DBA |
| Identify the storage requirements (storage type and capacity). | | DBA, SysAdmin |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, SysAdmin |
| Identify the network access security requirements for the source and target databases. | | DBA, SysAdmin |
| Identify the application migration strategy. | | DBA, SysAdmin, App owner |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC) and subnets. | | SysAdmin |
| Create security groups and network access control lists (ACLs). | | SysAdmin |
| Configure and start an Amazon RDS DB instance. | | SysAdmin |
### Migrate data - option 1
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the database schema manually or use a CASE tool such as erwin Data Modeler or SAP PowerDesigner. | | DBA |
### Migrate data - option 2
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate data using AWS DMS. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | | DBA, SysAdmin, App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | | DBA, SysAdmin, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary AWS resources. | | DBA, SysAdmin |
| Review and validate the project documents. | | DBA, SysAdmin, App owner |
| Gather metrics such as time to migrate, percentage of manual versus automated tasks, and cost savings. | | DBA, SysAdmin, App owner |
| Close out the project and provide feedback. | | DBA, SysAdmin, App owner |
## Related resources
**References**
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [Amazon RDS Pricing](https://aws.amazon.com/rds/pricing/)
+ [Using a SAP ASE Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SAP.html)
+ [Limitations for RDS Custom for SQL Server](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-reqs-limits-MS.html)
**Tutorials and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [AWS DMS (video)](https://www.youtube.com/watch?v=zb4GcjEdl8U)
+ [Amazon RDS (video)](https://www.youtube.com/watch?v=igRfulrrYCo)
# Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS DMS
*Marcelo Fernandes, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an on-premises Microsoft SQL Server database to Amazon Redshift by using AWS Data Migration Service (AWS DMS).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Microsoft SQL Server database in an on-premises data center
+ Completed prerequisites for using an Amazon Redshift database as a target for AWS DMS, as discussed in the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Redshift.html#CHAP_Target.Redshift.Prerequisites)
**Product versions**
+ SQL Server 2005-2019, Enterprise, Standard, Workgroup, Developer, and Web editions. For the latest list of supported versions, see [Using a Microsoft SQL Server Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html) in the AWS documentation.
## Architecture
**Source technology stack**
+ An on-premises Microsoft SQL Server database
**Target technology stack**
+ Amazon Redshift
**Data migration architecture**
![\[Architecture for migrating an on-premises SQL Server database to Amazon Redshift using AWS DMS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/65b2be1b-740e-4d4d-99a8-f77c4ea6553d/images/3a094bf2-be31-4d83-8dd2-9dc078321055.png)
## Tools
+ [AWS DMS ](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)is a data migration service that supports several types of source and target databases. For information about the Microsoft SQL Server database versions and editions that are supported for use with AWS DMS, see [Using a Microsoft SQL Server Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html) in the AWS DMS documentation. If AWS DMS doesn't support your source database, you must select an alternative method for data migration.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database version and engine. | | DBA |
| Identify the hardware requirements for the target server instance. | | DBA, Systems administrator |
| Identify the storage requirements (storage type and capacity). | | DBA, Systems administrator |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, Systems administrator |
| Identify the network access security requirements for the source and target databases. | | DBA, Systems administrator |
| Identify the application migration strategy. | | DBA, App owner, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | For more information, see [Working with a DB instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html) in the AWS documentation. | Systems administrator |
| Create security groups. | | Systems administrator |
| Configure and start an Amazon Redshift cluster. | For more information, see [Create a sample Amazon Redshift cluster](https://docs.aws.amazon.com/redshift/latest/gsg/rs-gsg-launch-sample-cluster.html) in the Amazon Redshift documentation. | DBA, Systems administrator |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the data from the Microsoft SQL Server database by using AWS DMS. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | | DBA, App owner, Systems administrator |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | | DBA, App owner, Systems administrator |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary resources. | | DBA, Systems administrator |
| Review and validate the project documents. | | DBA, App owner, Systems administrator |
| Gather metrics such as time to migrate, percentage of manual versus automated tasks, and cost savings. | | DBA, App owner, Systems administrator |
| Close out the project and provide feedback. | | DBA, App owner, Systems administrator |
## Related resources
**References**
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/index.html)
+ [Amazon Redshift documentation](https://docs.aws.amazon.com/redshift/)
+ [Amazon Redshift Pricing](https://aws.amazon.com/redshift/pricing/)
**Tutorials and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html)
+ [Using an Amazon Redshift database as a target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Redshift.html)
+ [AWS DMS (video)](https://www.youtube.com/watch?v=zb4GcjEdl8U)
# Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS SCT data extraction agents
*Neha Thakur, Amazon Web Services*
## Summary
This pattern outlines steps for migrating an on-premises Microsoft SQL Server source database to an Amazon Redshift target database by using AWS Schema Conversion Tool (AWS SCT) data extraction agents. An agent is an external program that is integrated with AWS SCT but performs data transformation elsewhere and interacts with other AWS services on your behalf.
## Prerequisites and limitations
**Prerequisites**
+ A Microsoft SQL Server source database used for the data warehouse workload in an on-premises data center
+ An active AWS account
**Product versions**
+ Microsoft SQL Server version 2008 or later. For the latest list of supported versions, see [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html).
## Architecture
**technology stack****Source **
+ An on-premises Microsoft SQL Server database
**technology stack****Target **
+ Amazon Redshift
**Data migration architecture**
![\[Migrating a SQL Server database to Amazon Redshift by using AWS SCT data extraction agents.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6975f67a-0705-47b4-a1b8-90aaa2597a04/images/dbff958b-7601-442e-9e23-4d07edd0ccfd.png)
## Tools
+ [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) (AWS SCT) handles heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that's compatible with the target database. When the source and target databases are very different, you can use an AWS SCT agent to perform additional data transformation. For more information, see [Migrating Data from an On-Premises Data Warehouse to Amazon Redshift](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/agents.dw.html) in the AWS documentation.
## Best practices
+ [Best practices for AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_BestPractices.html)
+ [Best practices for Amazon Redshift ](https://docs.aws.amazon.com/redshift/latest/dg/best-practices.html)
## Epics
### Prepare for migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions and engines. | | DBA |
| Identify hardware requirements for the target server instance. | | DBA, SysAdmin |
| Identify storage requirements (storage type and capacity). | | DBA, SysAdmin |
| Choose the proper instance type (capacity, storage features, network features). | | DBA, SysAdmin |
| Identify network access security requirements for the source and target databases. | | DBA, SysAdmin |
| Choose an application migration strategy. | | DBA, SysAdmin, App owner |
### Configure infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC) and subnets. | | SysAdmin |
| Create security groups. | | SysAdmin |
| Configure and start the Amazon Redshift cluster. | | SysAdmin |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the data using the AWS SCT data extraction agents. | | DBA |
### Migrate applications
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the chosen application migration strategy. | | DBA, SysAdmin, App owner |
### Cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Switch application clients over to the new infrastructure. | | DBA, SysAdmin, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary AWS resources. | | DBA, SysAdmin |
| Review and validate the project documents. | | DBA, SysAdmin, App owner |
| Gather metrics such as time to migrate, percentage of manual versus automated tasks, and cost savings. | | DBA, SysAdmin, App owner |
| Close the project and provide any feedback. | | DBA, SysAdmin, App owner |
## Related resources
**References**
+ [AWS SCT User Guide](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Using Data Extraction Agents](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/agents.html)
+ [Amazon Redshift Pricing](https://aws.amazon.com/redshift/pricing/)
**Tutorials and videos**
+ [Getting Started with the AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_GettingStarted.html)
+ [Getting Started with Amazon Redshift](http://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html)
# Migrate legacy applications from Oracle Pro\$1C to ECPG
*Sai Parthasaradhi and Mahesh Balumuri, Amazon Web Services*
## Summary
Most legacy applications that have embedded SQL code use the Oracle Pro\$1C precompiler to access the database. When you migrate these Oracle databases to Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition, you have to convert your application code to a format that’s compatible with the precompiler in PostgreSQL, which is called ECPG. This pattern describes how to convert Oracle Pro\$1C code to its equivalent in PostgreSQL ECPG.
For more information about Pro\$1C, see the [Oracle documentation](https://docs.oracle.com/cd/E11882_01/appdev.112/e10825/pc_01int.htm#i2415). For a brief introduction to ECPG, see the [Additional information](#migrate-legacy-applications-from-oracle-pro-c-to-ecpg-additional) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ An Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database
+ An Oracle database running on premises
## Tools
+ The PostgreSQL packages listed in the next section.
+ [AWS CLI ](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)– The AWS Command Line Interface (AWS CLI) is an open-source tool for interacting with AWS services through commands in your command-line shell. With minimal configuration, you can run AWS CLI commands that implement functionality equivalent to that provided by the browser-based AWS Management Console from a command prompt.
## Epics
### Set the build environment on CentOS or RHEL
| Task | Description | Skills required |
| --- | --- | --- |
| Install PostgreSQL packages. | Install the required PostgreSQL packages by using the following commands.
| App developer, DevOps engineer |
| Install the header files and libraries. | Install the `postgresql12-devel` package, which contains header files and libraries, by using the following commands. Install the package in both the development and the runtime environments to avoid errors in the runtime environment.
For the development environment only, also run the following commands.
yum install zlib-devel make -y ln -s /usr/pgsql-12/bin/ecpg /usr/bin/
| App developer, DevOps engineer |
| Configure the environment path variable. | Set the environment path for PostgreSQL client libraries.
export PATH=$PATH:/usr/pgsql-12/bin
| App developer, DevOps engineer |
| Install additional software as necessary. | If required, install **pgLoader** as a replacement for **SQL\$1Loader** in Oracle.
If you’re calling any Java applications from Pro\$1C modules, install Java.
yum install java -y
Install **ant** to compile the Java code.
yum install ant -y
| App developer, DevOps engineer |
| Install the AWS CLI. | Install the AWS CLI to run commands to interact with AWS services such as AWS Secrets Manager and Amazon Simple Storage Service (Amazon S3) from your applications.
| App developer, DevOps engineer |
| Identify the programs to be converted. | Identify the applications that you want to convert from Pro\$1C to ECPG. | App developer, App owner |
### Convert Pro\$1C code to ECPG
| Task | Description | Skills required |
| --- | --- | --- |
| Remove unwanted headers. | Remove the `include `headers that are not required in PostgreSQL, such as `oci.h`, `oratypes`, and `sqlda`. | App owner, App developer |
| Update variable declarations. | Add `EXEC SQL` statements for all variable declarations that are used as host variables.Remove the `EXEC SQL VAR` declarations such as the following from your application.
EXEC SQL VAR query IS STRING(2048);
| App developer, App owner |
| Update ROWNUM functionality. | The `ROWNUM` function isn’t available in PostgreSQL. Replace this with the `ROW_NUMBER` window function in SQL queries.Pro\$1C code:
SELECT SUBSTR(RTRIM(FILE_NAME,'.txt'),12) INTO :gcpclFileseq FROM (SELECT FILE_NAME FROM DEMO_FILES_TABLE WHERE FILE_NAME LIKE '%POC%' ORDER BY FILE_NAME DESC) FL2 WHERE ROWNUM <=1 ORDER BY ROWNUM;
ECPG code:
SELECT SUBSTR(RTRIM(FILE_NAME,'.txt'),12) INTO :gcpclFileseq FROM (SELECT FILE_NAME , ROW_NUMBER() OVER (ORDER BY FILE_NAME DESC) AS ROWNUM FROM demo_schema.DEMO_FILES_TABLE WHERE FILE_NAME LIKE '%POC%' ORDER BY FILE_NAME DESC) FL2 WHERE ROWNUM <=1 ORDER BY ROWNUM;
| App developer, App owner |
| Update function parameters to use alias variables. | In PostgreSQL, function parameters can’t be used as host variables. Overwrite them by using an alias variable.Pro\$1C code:
int processData(int referenceId){ EXEC SQL char col_val[100]; EXEC SQL select column_name INTO :col_val from table_name where col=:referenceId; }
ECPG code:
int processData(int referenceIdParam){ EXEC SQL int referenceId = referenceIdParam; EXEC SQL char col_val[100]; EXEC SQL select column_name INTO :col_val from table_name where col=:referenceId; }
| App developer, App owner |
| Update struct types. | Define `struct` types in `EXEC SQL BEGIN` and `END` blocks with `typedef` if the `struct` type variables are used as host variables. If the `struct` types are defined in header (`.h`) files, include the files with `EXEC SQL` include statements.Pro\$1C code:Header file (`demo.h`)
struct s_partition_ranges { char sc_table_group[31]; char sc_table_name[31]; char sc_range_value[10]; }; struct s_partition_ranges_ind { short ss_table_group; short ss_table_name; short ss_range_value; };
ECPG code:Header file (`demo.h`)
EXEC SQL BEGIN DECLARE SECTION; typedef struct { char sc_table_group[31]; char sc_table_name[31]; char sc_range_value[10]; } s_partition_ranges; typedef struct { short ss_table_group; short ss_table_name; short ss_range_value; } s_partition_ranges_ind; EXEC SQL END DECLARE SECTION;
exec sql include "demo.h" EXEC SQL BEGIN DECLARE SECTION; s_partition_ranges gc_partition_data[MAX_PART_TABLE] ; s_partition_ranges_ind gc_partition_data_ind[MAX_PART_TABLE] ; EXEC SQL END DECLARE SECTION;
| App developer, App owner |
| Modify logic to fetch from cursors. | To fetch multiple rows from cursors by using array variables, change the code to use `FETCH FORWARD`.Pro\$1C code:
EXEC SQL char aPoeFiles[MAX_FILES][FILENAME_LENGTH]; EXEC SQL FETCH filename_cursor into :aPoeFiles;
ECPG code:
EXEC SQL char aPoeFiles[MAX_FILES][FILENAME_LENGTH]; EXEC SQL int fetchSize = MAX_FILES; EXEC SQL FETCH FORWARD :fetchSize filename_cursor into :aPoeFiles;
| App developer, App owner |
| Modify package calls that don't have return values. | Oracle package functions that don’t have return values should be called with an indicator variable. If your application includes multiple functions that have the same name or if the unknown type functions generate runtime errors, typecast the values to the data types.Pro\$1C code:
void ProcessData (char *data , int id) { EXEC SQL EXECUTE BEGIN pkg_demo.process_data (:data, :id); END; END-EXEC; }
ECPG code:
void ProcessData (char *dataParam, int idParam ) { EXEC SQL char *data = dataParam; EXEC SQL int id = idParam; EXEC SQL short rowInd; EXEC SQL short rowInd = 0; EXEC SQL SELECT pkg_demo.process_data ( inp_data => :data::text, inp_id => :id ) INTO :rowInd; }
| App developer, App owner |
| Rewrite SQL\$1CURSOR variables. | Rewrite the `SQL_CURSOR` variable and its implementation.Pro\$1C code:
EXEC SQL DECLARE demo_cursor CURSOR FOR SELECT * from pkg_demo.open_filename_rc( demo_cur=>refcursor ) ; EXEC SQL char open_filename_rcInd[100]; # As the below function returns cursor_name as # return we need to use char[] type as indicator. EXEC SQL SELECT pkg_demo.get_cursor ( demo_cur=>'demo_cursor' ) INTO :open_filename_rcInd;
| App developer, App owner |
| Apply common migration patterns. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-legacy-applications-from-oracle-pro-c-to-ecpg.html) | App developer, App owner |
| Enable debugging, if required. | To run the ECPG program in debug mode, add the following command inside the main function block.
ECPGdebug(1, stderr);
| App developer, App owner |
### Compile ECPG programs
| Task | Description | Skills required |
| --- | --- | --- |
| Create an executable file for ECPG. | If you have an embedded SQL C source file named `prog1.pgc`, you can create an executable program by using the following sequence of commands.
ecpg prog1.pgc cc -I/usr/local/pgsql/include -c prog1.c cc -o prog1 prog1.o -L/usr/local/pgsql/lib -lecpg
| App developer, App owner |
| Create a make file for compilation. | Create a make file to compile the ECPG program, as shown in the following sample file.
| App developer, App owner |
### Test the application
| Task | Description | Skills required |
| --- | --- | --- |
| Test the code. | Test the converted application code to make sure that it functions correctly. | App developer, App owner, Test engineer |
## Related resources
+ [ECPG - Embedded SQL in C](https://www.postgresql.org/docs/current/static/ecpg.html) (PostgreSQL documentation)
+ [Error Handling](https://www.postgresql.org/docs/12/ecpg-errors.html) (PostgreSQL documentation)
+ [Why Use the Oracle Pro\$1C/C\$1\$1 Precompiler](https://docs.oracle.com/cd/E11882_01/appdev.112/e10825/pc_01int.htm#i2415) (Oracle documentation)
## Additional information
PostgreSQL has an embedded SQL precompiler, ECPG, which is equivalent to the Oracle Pro\$1C precompiler. ECPG converts C programs that have embedded SQL statements to standard C code by replacing the SQL calls with special function calls. The output files can then be processed with any C compiler tool chain.
**Input and output files**
ECPG converts each input file you specify on the command line to the corresponding C output file. If an input file name doesn’t have a file extension, .pgc is assumed. The file's extension is replaced by `.c` to construct the output file name. However, you can override the default output file name by using the `-o` option.
If you use a dash (`-`) as the input file name, ECPG reads the program from standard input and writes to standard output, unless you override that by using the `-o` option.
**Header files**
When the PostgreSQL compiler compiles the pre-processed C code files, it looks for the ECPG header files in the PostgreSQL `include` directory. Therefore, you might have to use the `-I` option to point the compiler to the correct directory (for example, `-I/usr/local/pgsql/include`).
**Libraries**
Programs that use C code with embedded SQL have to be linked against the `libecpg` library. For example, you can use the linker options` -L/usr/local/pgsql/lib -lecpg`.
Converted ECPG applications call functions in the `libpq` library through the embedded SQL library (`ecpglib`), and communicate with the PostgreSQL server by using the standard frontend/backend protocol.
# Migrate virtual generated columns from Oracle to PostgreSQL
*Veeranjaneyulu Grandhi, Rajesh Madiwale, and Ramesh Pathuri, Amazon Web Services*
## Summary
In version 11 and earlier, PostgreSQL doesn’t provide a feature that is directly equivalent to an Oracle virtual column. Handling virtual generated columns while migrating from Oracle Database to PostgreSQL version 11 or earlier is difficult for two reasons:
+ Virtual columns aren’t visible during migration.
+ PostgreSQL doesn't support the `generate` expression before version 12.
However, there are workarounds to emulate similar functionality. When you use AWS Database Migration Service (AWS DMS) to migrate data from Oracle Database to PostgreSQL version 11 and earlier, you can use trigger functions to populate the values in virtual generated columns. This pattern provides examples of Oracle Database and PostgreSQL code that you can use for this purpose. On AWS, you can use Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition for your PostgreSQL database.
Starting with PostgreSQL version 12, generated columns are supported. Generated columns can either be calculated from other column values on the fly, or calculated and stored. [PostgreSQL generated columns](https://www.postgresql.org/docs/12/ddl-generated-columns.html) are similar to Oracle virtual columns.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Oracle database
+ Target PostgreSQL databases (on Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible)
+ [PL/pgSQL](https://www.postgresql.org/docs/current/plpgsql.html) coding expertise
**Limitations**
+ Applies only to PostgreSQL versions before version 12.
+ Applies to Oracle Database version 11g or later.
+ Virtual columns are not supported in data migration tools.
+ Applies only to columns defined in the same table.
+ If a virtual generated column refers to a deterministic user-defined function, it cannot be used as a partitioning key column.
+ The output of the expression must be a scalar value. It cannot return an Oracle supplied datatype, a user-defined type, `LOB`, or `LONG RAW`.
+ Indexes that are defined against virtual columns are equivalent to function-based indexes in PostgreSQL.
+ Table statistics must be gathered.
## Tools
+ [pgAdmin 4](https://www.pgadmin.org/) is an open source management tool for PostgreSQL. This tool provides a graphical interface that simplifies the creation, maintenance, and use of database objects.
+ [Oracle SQL Developer](https://www.oracle.com/database/sqldeveloper/) is a free, integrated development environment for working with SQL in Oracle databases in both traditional and cloud deployments.
## Epics
### Create source and target database tables
| Task | Description | Skills required |
| --- | --- | --- |
| Create a source Oracle Database table. | In Oracle Database, create a table with virtual generated columns by using the following statement.
CREATE TABLE test.generated_column ( CODE NUMBER, STATUS VARCHAR2(12) DEFAULT 'PreOpen', FLAG CHAR(1) GENERATED ALWAYS AS (CASE UPPER(STATUS) WHEN 'OPEN' THEN 'N' ELSE 'Y' END) VIRTUAL VISIBLE );
In this source table, the data in the `STATUS` column is migrated through AWS DMS to the target database. The `FLAG` column, however, is populated by using `generate by` functionality, so this column isn’t visible to AWS DMS during migration. To implement the functionality of `generated by`, you must use triggers and functions in the target database to populate the values in the `FLAG` column, as shown in the next epic. | DBA, App developer |
| Create a target PostgreSQL table on AWS. | Create a PostgreSQL table on AWS by using the following statement.
CREATE TABLE test.generated_column ( code integer not null, status character varying(12) not null , flag character(1) );
In this table, the `status` column is a standard column. The `flag` column will be a generated column based on the data in the `status` column. | DBA, App developer |
### Create a trigger function to handle the virtual column in PostgreSQL
| Task | Description | Skills required |
| --- | --- | --- |
| Create a PostgreSQL trigger. | In PostgreSQL, create a trigger.
CREATE TRIGGER tgr_gen_column AFTER INSERT OR UPDATE OF status ON test.generated_column FOR EACH ROW EXECUTE FUNCTION test.tgf_gen_column();
| DBA, App developer |
| Create a PostgreSQL trigger function. | In PostgreSQL, create a function for the trigger. This function populates a virtual column that is inserted or updated by the application or AWS DMS, and validates the data.
CREATE OR REPLACE FUNCTION test.tgf_gen_column() RETURNS trigger AS $VIRTUAL_COL$ BEGIN IF (TG_OP = 'INSERT') THEN IF (NEW.flag IS NOT NULL) THEN RAISE EXCEPTION 'ERROR: cannot insert into column "flag"' USING DETAIL = 'Column "flag" is a generated column.'; END IF; END IF; IF (TG_OP = 'UPDATE') THEN IF (NEW.flag::VARCHAR != OLD.flag::varchar) THEN RAISE EXCEPTION 'ERROR: cannot update column "flag"' USING DETAIL = 'Column "flag" is a generated column.'; END IF; END IF; IF TG_OP IN ('INSERT','UPDATE') THEN IF (old.flag is NULL) OR (coalesce(old.status,'') != coalesce(new.status,'')) THEN UPDATE test.generated_column SET flag = (CASE UPPER(status) WHEN 'OPEN' THEN 'N' ELSE 'Y' END) WHERE code = new.code; END IF; END IF; RETURN NEW; END $VIRTUAL_COL$ LANGUAGE plpgsql;
| DBA, App developer |
### Test data migration by using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance. | To create a replication instance, follow the [instructions](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Creating.html) in the AWS DMS documentation. The replication instance should be in the same virtual private cloud (VPC) as your source and target databases. | DBA, App developer |
| Create source and target endpoints. | To create the endpoints, follow the [instructions](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html) in the AWS DMS documentation. | DBA, App developer |
| Test the endpoint connections. | You can test the endpoint connections by specifying the VPC and replication instance and choosing **Run test**. | DBA, App developer |
| Create and start a full load task. | For instructions, see [Creating a Task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html) and [Full-load task settings](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.FullLoad.html) in the AWS DMS documentation. | DBA, App developer |
| Validate the data for the virtual column. | Compare the data in the virtual column in the source and target databases. You can validate the data manually or write a script for this step. | DBA, App developer |
## Related resources
+ [Getting started with AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html) (AWS DMS documentation)
+ [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) (AWS DMS documentation)
+ [Using a PostgreSQL database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html) (AWS DMS documentation)
+ [Generated columns in PostgreSQL](https://www.postgresql.org/docs/12/ddl-generated-columns.html) (PostgreSQL documentation)
+ [Trigger functions](https://www.postgresql.org/docs/12/plpgsql-trigger.html) (PostgreSQL documentation)
+ [Virtual columns](https://docs.oracle.com/database/121/SQLRF/statements_7002.htm#SQLRF01402) in Oracle Database (Oracle documentation)
# Set up Oracle UTL\$1FILE functionality on Aurora PostgreSQL-Compatible
*Rakesh Raghav and anuradha chintha, Amazon Web Services*
## Summary
As part of your migration journey from Oracle to Amazon Aurora PostgreSQL-Compatible Edition on the Amazon Web Services (AWS) Cloud, you might encounter multiple challenges. For example, migrating code that relies on the Oracle `UTL_FILE` utility is always a challenge. In Oracle PL/SQL, the `UTL_FILE` package is used for file operations, such as read and write, in conjunction with the underlying operating system. The `UTL_FILE` utility works for both server and client machine systems.
Amazon Aurora PostgreSQL-Compatible is a managed database offering. Because of this, it isn't possible to access files on the database server. This pattern walks you through the integration of Amazon Simple Storage Service (Amazon S3) and Amazon Aurora PostgreSQL-Compatible to achieve a subset of `UTL_FILE` functionality. Using this integration, we can create and consume files without using third-party extract, transform, and load (ETL) tools or services.
Optionally, you can set up Amazon CloudWatch monitoring and Amazon SNS notifications.
We recommend thoroughly testing this solution before implementing it in a production environment.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ AWS Database Migration Service (AWS DMS) expertise
+ Expertise in PL/pgSQL coding
+ An Amazon Aurora PostgreSQL-Compatible cluster
+ An S3 bucket
**Limitations **
This pattern doesn't provide the functionality to act as a replacement for the Oracle `UTL_FILE` utility. However, the steps and sample code can be enhanced further to achieve your database modernization goals.
**Product versions**
+ Amazon Aurora PostgreSQL-Compatible Edition 11.9
## Architecture
**Target technology stack**
+ Amazon Aurora PostgreSQL-Compatible
+ Amazon CloudWatch
+ Amazon Simple Notification Service (Amazon SNS)
+ Amazon S3
**Target architecture **
The following diagram shows a high-level representation of the solution.
![\[Data files are uploaded to an S3 bucket, processed using the aws_s3 extension, and sent to the Aurora instance.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/3aeecd46-1f87-41f9-a9cd-f8181f92e83f/images/4a6c5f5c-58fb-4355-b243-d09a15c1cec6.png)
1. Files are uploaded from the application into the S3 bucket.
1. The `aws_s3` extension accesses the data, using PL/pgSQL, and uploads the data to Aurora PostgreSQL-Compatible.
## Tools
+ [Amazon Aurora PostgreSQL-Compatible](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) – Amazon Aurora PostgreSQL-Compatible Edition is a fully managed, PostgreSQL-compatible, and ACID-compliant relational database engine. It combines the speed and reliability of high-end commercial databases with the cost-effectiveness of open-source databases.
+ [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) – The AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services. With only one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) – Amazon CloudWatch monitors Amazon S3 resources and use.
+ [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) – Amazon Simple Storage Service (Amazon S3) is storage for the internet. In this pattern, Amazon S3 provides a storage layer to receive and store files for consumption and transmission to and from the Aurora PostgreSQL-Compatible cluster.
+ [aws\$1s3](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Procedural.Importing.html#aws_s3.table_import_from_s3) – The `aws_s3` extension integrates Amazon S3 and Aurora PostgreSQL-Compatible.
+ [Amazon SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) – Amazon Simple Notification Service (Amazon SNS) coordinates and manages the delivery or sending of messages between publishers and clients. In this pattern, Amazon SNS is used to send notifications.
+ [pgAdmin](https://www.pgadmin.org/docs/) – pgAdmin is an open-source management tool for Postgres. pgAdmin 4 provides a graphical interface for creating, maintaining, and using database objects.
**Code**
To achieve the required functionality, the pattern creates multiple functions with naming similar to `UTL_FILE`. The *Additional information* section contains the code base for these functions.
In the code, replace `testaurorabucket` with the name of your test S3 bucket. Replace `us-east-1` with the AWS Region where your test S3 bucket is located.
## Epics
### Integrate Amazon S3 and Aurora PostgreSQL-Compatible
| Task | Description | Skills required |
| --- | --- | --- |
| Set up IAM policies. | Create AWS Identity and Access Management (IAM) policies that grant access to the S3 bucket and objects in it. For the code, see the *Additional information* section. | AWS administrator, DBA |
| Add Amazon S3 access roles to Aurora PostgreSQL. | Create two IAM roles: one role for read and one role for write access to Amazon S3. Attach the two roles to the Aurora PostgreSQL-Compatible cluster: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-oracle-utl_file-functionality-on-aurora-postgresql-compatible.html)For more information, see the Aurora PostgreSQL-Compatible documentation on [importing](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_PostgreSQL.S3Import.html) and [exporting](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/postgresql-s3-export.html) data to Amazon S3. | AWS administrator, DBA |
### Set up the extensions in Aurora PostgreSQL-Compatible
| Task | Description | Skills required |
| --- | --- | --- |
| Create the aws\$1commons extension. | The `aws_commons` extension is a dependency of the `aws_s3` extension. | DBA, Developer |
| Create the aws\$1s3 extension. | The `aws_s3` extension interacts with Amazon S3. | DBA, Developer |
### Validate Amazon S3 and Aurora PostgreSQL-Compatible integration
| Task | Description | Skills required |
| --- | --- | --- |
| Test importing files from Amazon S3 into Aurora PostgreSQL. | To test importing files into Aurora PostgreSQL-Compatible, create a sample CSV file and upload it into the S3 bucket. Create a table definition based on the CSV file, and load the file into the table by using the `aws_s3.table_import_from_s3` function. | DBA, Developer |
| Test exporting files from Aurora PostgreSQL to Amazon S3. | To test exporting files from Aurora PostgreSQL-Compatible, create a test table, populate it with data, and then export the data by using the `aws_s3.query_export_to_s3` function. | DBA, Developer |
### To mimic the UTL\$1FILE utility, create wrapper functions
| Task | Description | Skills required |
| --- | --- | --- |
| Create the utl\$1file\$1utility schema. | The schema keeps the wrapper functions together. To create the schema, run the following command.
CREATE SCHEMA utl_file_utility;
| DBA, Developer |
| Create the file\$1type type. | To create the `file_type` type, use the following code.
CREATE TYPE utl_file_utility.file_type AS ( p_path character varying(30), p_file_name character varying );
| DBA/Developer |
| Create the init function. | The `init` function initializes common variable such as `bucket` or `region`. For the code, see the *Additional information* section. | DBA/Developer |
| Create the wrapper functions. | Create the wrapper functions `fopen`, `put_line`, and `fclose`. For code, see the *Additional information* section. | DBA, Developer |
### Test the wrapper functions
| Task | Description | Skills required |
| --- | --- | --- |
| Test the wrapper functions in write mode. | To test the wrapper functions in write mode, use the code provided in the *Additional information* section. | DBA, Developer |
| Test the wrapper functions in append mode. | To test the wrapper functions in append mode, use the code provide in the *Additional information* section. | DBA, Developer |
## Related resources
+ [Amazon S3 integration](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_PostgreSQL.S3Import.html)
+ [Amazon S3](https://aws.amazon.com/s3/)
+ [Aurora](https://aws.amazon.com/rds/aurora/?nc2=h_ql_prod_db_aa&aurora-whats-new.sort-by=item.additionalFields.postDateTime&aurora-whats-new.sort-order=desc)
+ [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/)
+ [Amazon SNS](https://aws.amazon.com/sns/?nc2=h_ql_prod_ap_sns&whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc)
## Additional information
**Set up IAM policies**
Create the following policies.
|
|
| Policy name | JSON |
| --- |--- |
| S3IntRead |
|
**Create the init function**
To initialize common variables, such as `bucket` or `region`, create the `init` function by using the following code.
```
CREATE OR REPLACE FUNCTION utl_file_utility.init(
)
RETURNS void
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
BEGIN
perform set_config
( format( '%s.%s','UTL_FILE_UTILITY', 'region' )
, 'us-east-1'::text
, false );
perform set_config
( format( '%s.%s','UTL_FILE_UTILITY', 's3bucket' )
, 'testaurorabucket'::text
, false );
END;
$BODY$;
```
**Create the wrapper functions**
Create the `fopen`, `put_line`, and `fclose` wrapper functions.
*fopen*
```
CREATE OR REPLACE FUNCTION utl_file_utility.fopen(
p_file_name character varying,
p_path character varying,
p_mode character DEFAULT 'W'::bpchar,
OUT p_file_type utl_file_utility.file_type)
RETURNS utl_file_utility.file_type
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
declare
v_sql character varying;
v_cnt_stat integer;
v_cnt integer;
v_tabname character varying;
v_filewithpath character varying;
v_region character varying;
v_bucket character varying;
BEGIN
/*initialize common variable */
PERFORM utl_file_utility.init();
v_region := current_setting( format( '%s.%s', 'UTL_FILE_UTILITY', 'region' ) );
v_bucket := current_setting( format( '%s.%s', 'UTL_FILE_UTILITY', 's3bucket' ) );
/* set tabname*/
v_tabname := substring(p_file_name,1,case when strpos(p_file_name,'.') = 0 then length(p_file_name) else strpos(p_file_name,'.') - 1 end );
v_filewithpath := case when NULLif(p_path,'') is null then p_file_name else concat_ws('/',p_path,p_file_name) end ;
raise notice 'v_bucket %, v_filewithpath % , v_region %', v_bucket,v_filewithpath, v_region;
/* APPEND MODE HANDLING; RETURN EXISTING FILE DETAILS IF PRESENT ELSE CREATE AN EMPTY FILE */
IF p_mode = 'A' THEN
v_sql := concat_ws('','create temp table if not exists ', v_tabname,' (col1 text)');
execute v_sql;
begin
PERFORM aws_s3.table_import_from_s3
( v_tabname,
'',
'DELIMITER AS ''#''',
aws_commons.create_s3_uri
( v_bucket,
v_filewithpath ,
v_region)
);
exception
when others then
raise notice 'File load issue ,%',sqlerrm;
raise;
end;
execute concat_ws('','select count(*) from ',v_tabname) into v_cnt;
IF v_cnt > 0
then
p_file_type.p_path := p_path;
p_file_type.p_file_name := p_file_name;
else
PERFORM aws_s3.query_export_to_s3('select ''''',
aws_commons.create_s3_uri(v_bucket, v_filewithpath, v_region)
);
p_file_type.p_path := p_path;
p_file_type.p_file_name := p_file_name;
end if;
v_sql := concat_ws('','drop table ', v_tabname);
execute v_sql;
ELSEIF p_mode = 'W' THEN
PERFORM aws_s3.query_export_to_s3('select ''''',
aws_commons.create_s3_uri(v_bucket, v_filewithpath, v_region)
);
p_file_type.p_path := p_path;
p_file_type.p_file_name := p_file_name;
END IF;
EXCEPTION
when others then
p_file_type.p_path := p_path;
p_file_type.p_file_name := p_file_name;
raise notice 'fopenerror,%',sqlerrm;
raise;
END;
$BODY$;
```
*put\$1line*
```
CREATE OR REPLACE FUNCTION utl_file_utility.put_line(
p_file_name character varying,
p_path character varying,
p_line text,
p_flag character DEFAULT 'W'::bpchar)
RETURNS boolean
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
/**************************************************************************
* Write line, p_line in windows format to file, p_fp - with carriage return
* added before new line.
**************************************************************************/
declare
v_sql varchar;
v_ins_sql varchar;
v_cnt INTEGER;
v_filewithpath character varying;
v_tabname character varying;
v_bucket character varying;
v_region character varying;
BEGIN
PERFORM utl_file_utility.init();
/* check if temp table already exist */
v_tabname := substring(p_file_name,1,case when strpos(p_file_name,'.') = 0 then length(p_file_name) else strpos(p_file_name,'.') - 1 end );
v_sql := concat_ws('','select count(1) FROM pg_catalog.pg_class c LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace where n.nspname like ''pg_temp_%'''
,' AND pg_catalog.pg_table_is_visible(c.oid) AND Upper(relname) = Upper( '''
, v_tabname ,''' ) ');
execute v_sql into v_cnt;
IF v_cnt = 0 THEN
v_sql := concat_ws('','create temp table ',v_tabname,' (col text)');
execute v_sql;
/* CHECK IF APPEND MODE */
IF upper(p_flag) = 'A' THEN
PERFORM utl_file_utility.init();
v_region := current_setting( format( '%s.%s', 'UTL_FILE_UTILITY', 'region' ) );
v_bucket := current_setting( format( '%s.%s', 'UTL_FILE_UTILITY', 's3bucket' ) );
/* set tabname*/
v_filewithpath := case when NULLif(p_path,'') is null then p_file_name else concat_ws('/',p_path,p_file_name) end ;
begin
PERFORM aws_s3.table_import_from_s3
( v_tabname,
'',
'DELIMITER AS ''#''',
aws_commons.create_s3_uri
( v_bucket,
v_filewithpath,
v_region )
);
exception
when others then
raise notice 'Error Message : %',sqlerrm;
raise;
end;
END IF;
END IF;
/* INSERT INTO TEMP TABLE */
v_ins_sql := concat_ws('','insert into ',v_tabname,' values(''',p_line,''')');
execute v_ins_sql;
RETURN TRUE;
exception
when others then
raise notice 'Error Message : %',sqlerrm;
raise;
END;
$BODY$;
```
*fclose*
```
CREATE OR REPLACE FUNCTION utl_file_utility.fclose(
p_file_name character varying,
p_path character varying)
RETURNS boolean
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
DECLARE
v_filewithpath character varying;
v_bucket character varying;
v_region character varying;
v_tabname character varying;
v_sql character varying;
BEGIN
PERFORM utl_file_utility.init();
v_region := current_setting( format( '%s.%s', 'UTL_FILE_UTILITY', 'region' ) );
v_bucket := current_setting( format( '%s.%s', 'UTL_FILE_UTILITY', 's3bucket' ) );
v_tabname := substring(p_file_name,1,case when strpos(p_file_name,'.') = 0 then length(p_file_name) else strpos(p_file_name,'.') - 1 end );
v_filewithpath := case when NULLif(p_path,'') is null then p_file_name else concat_ws('/',p_path,p_file_name) end ;
raise notice 'v_bucket %, v_filewithpath % , v_region %', v_bucket,v_filewithpath, v_region ;
/* exporting to s3 */
perform aws_s3.query_export_to_s3
(concat_ws('','select * from ',v_tabname,' order by ctid asc'),
aws_commons.create_s3_uri(v_bucket, v_filewithpath, v_region)
);
v_sql := concat_ws('','drop table ', v_tabname);
execute v_sql;
RETURN TRUE;
EXCEPTION
when others then
raise notice 'error fclose %',sqlerrm;
RAISE;
END;
$BODY$;
```
**Test your setup and wrapper functions**
Use the following anonymous code blocks to test your setup.
*Test the write mode*
The following code writes a file named `s3inttest` in the S3 bucket.
```
do $$
declare
l_file_name varchar := 's3inttest' ;
l_path varchar := 'integration_test' ;
l_mode char(1) := 'W';
l_fs utl_file_utility.file_type ;
l_status boolean;
begin
select * from
utl_file_utility.fopen( l_file_name, l_path , l_mode ) into l_fs ;
raise notice 'fopen : l_fs : %', l_fs;
select * from
utl_file_utility.put_line( l_file_name, l_path ,'this is test file:in s3bucket: for test purpose', l_mode ) into l_status ;
raise notice 'put_line : l_status %', l_status;
select * from utl_file_utility.fclose( l_file_name , l_path ) into l_status ;
raise notice 'fclose : l_status %', l_status;
end;
$$
```
*Test the append mode*
The following code appends lines onto the `s3inttest` file that was created in the previous test.
```
do $$
declare
l_file_name varchar := 's3inttest' ;
l_path varchar := 'integration_test' ;
l_mode char(1) := 'A';
l_fs utl_file_utility.file_type ;
l_status boolean;
begin
select * from
utl_file_utility.fopen( l_file_name, l_path , l_mode ) into l_fs ;
raise notice 'fopen : l_fs : %', l_fs;
select * from
utl_file_utility.put_line( l_file_name, l_path ,'this is test file:in s3bucket: for test purpose : append 1', l_mode ) into l_status ;
raise notice 'put_line : l_status %', l_status;
select * from
utl_file_utility.put_line( l_file_name, l_path ,'this is test file:in s3bucket : for test purpose : append 2', l_mode ) into l_status ;
raise notice 'put_line : l_status %', l_status;
select * from utl_file_utility.fclose( l_file_name , l_path ) into l_status ;
raise notice 'fclose : l_status %', l_status;
end;
$$
```
**Amazon SNS notifications**
Optionally, you can set up Amazon CloudWatch monitoring and Amazon SNS notifications on the S3 bucket. For more information, see [Monitoring Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/monitoring-overview.html) and [Setting up Amazon SNS Notifications](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html).
# Validate database objects after migrating from Oracle to Amazon Aurora PostgreSQL
*Venkatramana Chintha and Eduardo Valentim, Amazon Web Services*
## Summary
This pattern describes a step-by-step approach to validate objects after migrating an Oracle database to Amazon Aurora PostgreSQL-Compatible Edition.
This pattern outlines usage scenarios and steps for database object validation; for more detailed information, see [Validating database objects after migration using AWS SCT and AWS DMS](https://aws.amazon.com/blogs/database/validating-database-objects-after-migration-using-aws-sct-and-aws-dms/) on the [AWS Database blog](https://aws.amazon.com/blogs/).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An on-premises Oracle database that was migrated to an Aurora PostgreSQL-Compatible database.
+ Sign-in credentials that have the [AmazonRDSDataFullAccess](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/query-editor.html) policy applied, for the Aurora PostgreSQL-Compatible database.
+ This pattern uses the [query editor for Aurora Serverless DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/query-editor.html), which is available in the Amazon Relational Database Service (Amazon RDS) console. However, you can use this pattern with any other query editor.
**Limitations**
+ Oracle SYNONYM objects are not available in PostgreSQL but can be partially validated through **views** or SET search\$1path queries.
+ The Amazon RDS query editor is available only in [certain AWS Regions and for certain MySQL and PostgreSQL versions](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/query-editor.html).
## Architecture
![\[Database migration workflow showing on-premises Oracle to AWSAurora PostgreSQL via client program and validation scripts.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7c028960-6dea-46ad-894d-e42cefd50c03/images/be5f8ae3-f5af-4c5e-9440-09ab410beaa1.png)
## Tools
**Tools**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) – Aurora PostgreSQL-Compatible is a fully managed, PostgreSQL-compatible, and ACID-compliant relational database engine that combines the speed and reliability of high-end commercial databases with the simplicity and cost-effectiveness of open-source databases.
+ [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) – Amazon Relational Database Service (Amazon RDS) makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks.
+ [Query Editor for Aurora Severless](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/query-editor.html) – Query editor helps you run SQL queries in the Amazon RDS console. You can run any valid SQL statement on the Aurora Serverless DB cluster, including data manipulation and data definition statements.
To validate the objects, use the full scripts in the "Object validation scripts" file in the "Attachments" section. Use the following table for reference.
|
|
| Oracle object | Script to use |
| --- |--- |
| Packages | Query 1 |
| Tables | Query 3 |
| Views | Query 5 |
| Sequences | Query 7 |
| Triggers | Query 9 |
| Primary keys | Query 11 |
| Indexes | Query 13 |
| Check constraints | Query 15 |
| Foreign keys | Query 17 |
|
|
| PostgreSQL object | Script to use |
| --- |--- |
| Packages | Query 2 |
| Tables | Query 4 |
| Views | Query 6 |
| Sequences | Query 8 |
| Triggers | Query 10 |
| Primary keys | Query 12 |
| Indexes | Query 14 |
| Check constraints | Query 16 |
| Foreign keys | Query 18 |
## Epics
### Validate objects in the source Oracle database
| Task | Description | Skills required |
| --- | --- | --- |
| Run the "packages" validation query in the source Oracle database. | Download and open the "Object validation scripts" file from the "Attachments" section. Connect to the source Oracle database through your client program. Run the "Query 1" validations script from the "Object validation scripts" file. Important: Enter your Oracle user name instead of "your\$1schema" in the queries. Make sure you record your query results. | Developer, DBA |
| Run the "tables" validation query. | Run the "Query 3" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "views" validation query. | Run the "Query 5" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "sequences" count validation. | Run the "Query 7" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "triggers" validation query. | Run the "Query 9" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "primary keys" validation query. | Run the "Query 11" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "indexes" validation query. | Run the "Query 13" validation script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "check constraints" validation query. | Run the "Query 15" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "foreign keys" validation query. | Run the "Query 17" validation script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
### Validate objects in the target Aurora PostgreSQL-Compatible database
| Task | Description | Skills required |
| --- | --- | --- |
| Connect to the target Aurora PostgreSQL-Compatible database by using the query editor. | Sign in to the AWS Management Console and open the Amazon RDS console. In the upper-right corner, choose the AWS Region in which you created the Aurora PostgreSQL-Compatible database. In the navigation pane, choose "Databases," and choose the target Aurora PostgreSQL-Compatible database. In "Actions," choose "Query." Important: If you haven't connected to the database before, the "Connect to database" page opens. You then need to enter your database information, such as user name and password. | Developer, DBA |
| Run the "packages" validation query. | Run the "Query 2" script from the "Object validation scripts" file in the "Attachments" section. Make sure you record your query results. | Developer, DBA |
| Run the "tables" validation query. | Return to the query editor for the Aurora PostgreSQL-Compatible database, and run the "Query 4" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "views" validation query. | Return to the query editor for the Aurora PostgreSQL-Compatible database, and run the "Query 6" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "sequences" count validation. | Return to the query editor for the Aurora PostgreSQL-Compatible database, and run the "Query 8" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "triggers" validation query. | Return to the query editor for the Aurora PostgreSQL-Compatible database, and run the "Query 10" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "primary keys" validation query. | Return to the query editor for the Aurora PostgreSQL-Compatible database, and run the "Query 12" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "indexes" validation query. | Return to the query editor for the Aurora PostgreSQL-Compatible database, and run the "Query 14" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "check constraints" validation query. | Run the "Query 16" script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
| Run the "foreign keys" validation query. | Run the "Query 18" validation script from the "Object validation scripts" file. Make sure you record your query results. | Developer, DBA |
### Compare source and target database validation records
| Task | Description | Skills required |
| --- | --- | --- |
| Compare and validate both query results. | Compare the query results of the Oracle and Aurora PostgreSQL-Compatible databases to validate all objects. If they all match, then all objects have been successfully validated. | Developer, DBA |
## Related resources
+ [Validating database objects after a migration using AWS SCT and AWS DMS](https://aws.amazon.com/blogs/database/validating-database-objects-after-migration-using-aws-sct-and-aws-dms/)
+ [Amazon Aurora Features: PostgreSQL-Compatible Edition](https://aws.amazon.com/rds/aurora/postgresql-features/)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/7c028960-6dea-46ad-894d-e42cefd50c03/attachments/attachment.zip)
# Rehost
**Topics**
+ [Accelerate the discovery and migration of Microsoft workloads to AWS](accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.md)
+ [Create an approval process for firewall requests during a rehost migration to AWS](create-an-approval-process-for-firewall-requests-during-a-rehost-migration-to-aws.md)
+ [Ingest and migrate EC2 Windows instances into an AWS Managed Services account](ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.md)
+ [Migrate a Couchbase Server database to Amazon EC2](migrate-couchbase-server-ec2.md)
+ [Migrate Db2 for LUW to Amazon EC2 by using log shipping to reduce outage time](migrate-db2-for-luw-to-amazon-ec2-by-using-log-shipping-to-reduce-outage-time.md)
+ [Migrate Db2 for LUW to Amazon EC2 with high availability disaster recovery](migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery.md)
+ [Migrate IIS-hosted applications to Amazon EC2 by using appcmd.exe](migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon EC2 using Application Migration Service](migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.md)
+ [Migrate an F5 BIG-IP workload to F5 BIG-IP VE on the AWS Cloud](migrate-an-f5-big-ip-workload-to-f5-big-ip-ve-on-the-aws-cloud.md)
+ [Migrate an on-premises Go web application to AWS Elastic Beanstalk by using the binary method](migrate-an-on-premises-go-web-application-to-aws-elastic-beanstalk-by-using-the-binary-method.md)
+ [Migrate an on-premises SFTP server to AWS using AWS Transfer for SFTP](migrate-an-on-premises-sftp-server-to-aws-using-aws-transfer-for-sftp.md)
+ [Migrate an on-premises VM to Amazon EC2 by using AWS Application Migration Service](migrate-an-on-premises-vm-to-amazon-ec2-by-using-aws-application-migration-service.md)
+ [Migrate small sets of data from on premises to Amazon S3 using AWS SFTP](migrate-small-sets-of-data-from-on-premises-to-amazon-s3-using-aws-sftp.md)
+ [Migrate an on-premises Oracle database to Oracle on Amazon EC2](migrate-an-on-premises-oracle-database-to-oracle-on-amazon-ec2.md)
+ [Migrate an on-premises Oracle database to Amazon EC2 by using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.md)
+ [Migrate RHEL BYOL systems to AWS License-Included instances by using AWS MGN](migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon EC2](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-ec2.md)
+ [Rehost on-premises workloads in the AWS Cloud: migration checklist](rehost-on-premises-workloads-in-the-aws-cloud-migration-checklist.md)
+ [Set up Multi-AZ infrastructure for a SQL Server Always On FCI by using Amazon FSx](set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.md)
+ [Use BMC Discovery queries to extract migration data for migration planning](use-bmc-discovery-queries-to-extract-migration-data-for-migration-planning.md)
# Accelerate the discovery and migration of Microsoft workloads to AWS
*Ali Alzand, Amazon Web Services*
## Summary
This pattern shows you how to use the [Migration Validator Toolkit PowerShell module](https://github.com/aws-samples/migration-validator-toolkit-for-microsoft-workloads) to discover and migrate your Microsoft workloads to AWS. The module works by performing multiple checks and validations for common tasks associated with any Microsoft workload. For example, the module checks for instances that might have multiple disks attached to it or instances that use many IP addresses. For a full list of checks that the module can perform, see the [Checks](https://github.com/aws-samples/migration-validator-toolkit-for-microsoft-workloads#checks) section on the module's GitHub page.
The Migration Validator Toolkit PowerShell module can help your organization reduce the time and effort involved in discovering what applications and services are running on your Microsoft workloads. The module can also help you identify the configurations of your workloads so that you can find out if your configurations are supported on AWS. The module also provides recommendations for next steps and mitigation actions, so that you can avoid any misconfigurations before, during, or after your migration.
## Prerequisites and limitations
**Prerequisites**
+ Local administrator account
+ PowerShell 4.0
**Limitations**
+ Works only on Microsoft Windows Server 2012 R2 or later
## Tools
**Tools**
+ PowerShell 4.0
**Code repository**
The Migration Validator Toolkit PowerShell module for this pattern is available in the GitHub [migration-validator-toolkit-for-microsoft-workloads](https://github.com/aws-samples/migration-validator-toolkit-for-microsoft-workloads) repository.
## Epics
### Run the Migration Validator Toolkit PowerShell module on a single target
| Task | Description | Skills required |
| --- | --- | --- |
| Download, extract, import, and invoke the module. | Choose one of the following methods to download and deploy the module:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html)**Run the PowerShell script**In PowerShell, run the following example code:
The code downloads the module from a .zip file. Then, the code extracts, imports, and invokes the module.**Download and extract the .zip file**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html)**Clone the GitHub repository**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html) | System Administrator |
| Invoke the module manually. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html)[Format-Table](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/format-table?view=powershell-7.3) format:
| System Administrator |
### Run the Migration Validator Toolkit PowerShell module on multiple targets
| Task | Description | Skills required |
| --- | --- | --- |
| Download the .zip file or clone the GitHub repository. | Choose one of the following options:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html)
| System Administrator |
| Update the server.csv list. | If you downloaded the .zip file, follow these steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html) | System Administrator |
| Invoke the module. | You can use any computer within the domain that uses a domain user that has administrator access to target computers.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html)
The output .csv file is saved in `MigrationValidatorToolkit\Outputs\folder` with the prefix name `DomainComputers_MigrationAutomations_YYYY-MM-DDTHH-MM-SS`. | System Administrator |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| `MigrationValidatorToolkit` writes information about executions, commands, and errors to log files on the running host. | You can view log files manually in the following location:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.html) |
## Related resources
+ [Options, tools, and best practices for migrating Microsoft workloads to AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-microsoft-workloads-aws/introduction.html) (AWS Prescriptive Guidance)
+ [Microsoft migration patterns](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migration-migration-patterns-by-workload-microsoft-pattern-list.html) (AWS Prescriptive Guidance)
+ [Free Cloud Migration Services on AWS](https://aws.amazon.com/free/migration/) (AWS documentation)
+ [Predefined post-launch actions](https://docs.aws.amazon.com/mgn/latest/ug/predefined-post-launch-actions.html) (Application marketing documentation)
## Additional information
**Frequently asked questions**
*Where can I run the Migration Validator Toolkit PowerShell module?*
You can run the module on Microsoft Windows Server 2012 R2 or later.
*When do I run this module?*
We recommend that you run the module during the [assess phase](https://aws.amazon.com/cloud-migration/how-to-migrate/) of the migration journey.
*Does the module modify my existing servers?*
No. All actions in this module are read-only.
*How long does it take to run the module?*
It typically takes 1–5 minutes to run the module, but it depends on the resource allocation of your server.
*What permissions does the module need to run?*
You must run the module from a local administrator account.
*Can I run the module on physical servers?*
Yes, as long as the operating system is Microsoft Windows Server 2012 R2 or later.
*How do I run the module at scale for multiple servers?*
To run the module on multiple domain-joined computers at scale, follow the steps in the *Run the Migration Validator Toolkit PowerShell module on multiple targets *epic of this guide. For non domain-joined computers, use a remote invocation or run the module locally by following the steps in the *Run the Migration Validator Toolkit PowerShell module on a single target* epic of this guide.
# Create an approval process for firewall requests during a rehost migration to AWS
*Srikanth Rangavajhala, Amazon Web Services*
## Summary
If you want to use [AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) or [Cloud Migration Factory on AWS](https://aws.amazon.com/solutions/implementations/cloud-migration-factory-on-aws/) for a rehost migration to the AWS Cloud, one of the prerequisites is that you must keep TCP ports 443 and 1500 open. Typically, opening these firewall ports requires approval from your information security (InfoSec) team.
This pattern outlines the process to obtain a firewall request approval from an InfoSec team during a rehost migration to the AWS Cloud. You can use this process to avoid rejections of your firewall request by the InfoSec team, which can become expensive and time consuming. The firewall request process has two review and approval steps between AWS migration consultants and leads who work with your InfoSec and application teams to open the firewall ports.
This pattern assumes that you are planning a rehost migration with AWS consultants or migration specialists from your organization. You can use this pattern if your organization doesn’t have a firewall approval process or firewall request blanket approval form. For more information about this, see the *Limitations* section of this pattern. For more information on network requirements for Application Migration Service, see [Network requirements](https://docs.aws.amazon.com/mgn/latest/ug/Network-Requirements.html) in the Application Migration Service documentation.
## Prerequisites and limitations
**Prerequisites **
+ A planned rehost migration with AWS consultants or migration specialists from your organization
+ The required port and IP information to migrate the stack
+ Existing and future state architecture diagrams
+ Firewall information about the on-premises and destination infrastructure, ports, and zone-to-zone traffic flow
+ A firewall request review checklist (attached)
+ A firewall request document, configured according to your organization’s requirements
+ A contact list for the firewall reviewers and approvers, including the following roles:
+ **Firewall request submitter** – AWS migration specialist or consultant. The firewall request submitter can also be a migration specialist from your organization.
+ **Firewall request reviewer** – Typically, this is the single point of contact (SPOC) from AWS.
+ **Firewall request approver** – An InfoSec team member.
**Limitations **
+ This pattern describes a generic firewall request approval process. Requirements can vary for individual organizations.
+ Make sure that you track changes to your firewall request document.
The following table shows the use cases for this pattern.
|
|
| Does your organization have an existing firewall approval process? | Does your organization have an existing firewall request form? | Suggested action |
| --- |--- |--- |
| Yes | Yes | Collaborate with AWS consultants or your migration specialists to implement your organization’s process. |
| No | Yes | Use this pattern’s firewall approval process. Use either an AWS consultant or a migration specialist from your organization to submit the firewall request blanket approval form. |
| No | No | Use this pattern’s firewall approval process. Use either an AWS consultant or a migration specialist from your organization to submit the firewall request blanket approval form. |
## Architecture
The following diagram shows the steps for the firewall request approval process.
![\[Process for firewall request approval from an InfoSec team during a rehost migration to AWS Cloud.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/cf9b58ad-ab6f-43d3-92da-968529c8d042/images/c672f7ce-6e9f-4dbc-bf2c-4272a6c4432b.png)
## Tools
You can use scanner tools such as [Palo Alto Networks](https://www.paloaltonetworks.com/) or [SolarWinds](https://www.solarwinds.com/) to analyze and validate firewalls and IP addresses.
## Epics
### Analyze the firewall request
| Task | Description | Skills required |
| --- | --- | --- |
| Analyze the ports and IP addresses. | The firewall request submitter completes an initial analysis to understand the required firewall ports and IP addresses. After this is complete, they request that your InfoSec team open the required ports and map the IP addresses. | AWS Cloud engineer, migration specialist |
### Validate the firewall request
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the firewall information. | The AWS Cloud engineer schedules a meeting with your InfoSec team. During this meeting, the engineer examines and validates the firewall request information.Typically, the firewall request submitter is the same person as the firewall requester. This validation phase can become iterative based on the feedback given by the approver if anything is observed or recommended. | AWS Cloud engineer, migration specialist |
| Update the firewall request document. | After the InfoSec team shares their feedback, the firewall request document is edited, saved, and re-uploaded. This document is updated after each iteration.We recommend that you store this document in a version-controlled storage folder. This means that all changes are tracked and correctly applied. | AWS Cloud engineer, migration specialist |
### Submit the firewall request
| Task | Description | Skills required |
| --- | --- | --- |
| Submit the firewall request. | After the firewall request approver has approved the firewall blanket approval request, the AWS Cloud engineer submits the firewall request. The request specifies the ports that must be open and IP addresses that are required to map and update the AWS account.You can make suggestions or provide feedback after the firewall request is submitted. We recommend that you automate this feedback process and send any edits through a defined workflow mechanism. | AWS Cloud engineer, migration specialist |
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/cf9b58ad-ab6f-43d3-92da-968529c8d042/attachments/attachment.zip)
# Ingest and migrate EC2 Windows instances into an AWS Managed Services account
*Anil Kunapareddy and Venkatramana Chintha, Amazon Web Services*
## Summary
This pattern explains the step-by-step process of migrating and ingesting Amazon Elastic Compute Cloud (Amazon EC2) Windows instances into an Amazon Web Services (AWS) Managed Services (AMS) account. AMS can help you manage the instance more efficiently and securely. AMS provides operational flexibility, enhances security and compliance, and helps you optimize capacity and reduce costs.
This pattern starts with an EC2 Windows instance that you have migrated to a staging subnet in your AMS account. A variety of migration services and tools are available to perform this task, such as AWS Application Migration Service.
To make a change to your AMS-managed environment, you create and submit a request for change (RFC) for a particular operation or action. Using an AMS workload ingest (WIGS) RFC, you ingest the instance into the AMS account and create a custom Amazon Machine Image (AMI). You then create the AMS-managed EC2 instance by submitting another RFC to create an EC2 stack. For more information, see [AMS Workload Ingest](https://docs.aws.amazon.com/managedservices/latest/appguide/ams-workload-ingest.html) in the AMS documentation.
## Prerequisites and limitations
**Prerequisites**
+ An active, AMS-managed AWS account
+ An existing landing zone
+ Permissions to make changes in the AMS-managed VPC
+ An Amazon EC2 Windows instance in a staging subnet in your AMS account
+ Completion of the [general prerequisites](https://docs.aws.amazon.com/managedservices/latest/appguide/ex-migrate-instance-prereqs.html) for migrating workloads using AMS WIGS
+ Completion of the [Windows prerequisites](https://docs.aws.amazon.com/managedservices/latest/appguide/ex-migrate-prereqs-win.html) for migrating workloads using AMS WIGS
**Limitations**
+ This pattern is for EC2 instances operating Windows Server. This pattern doesn’t apply to instances running other operating systems, such as Linux.
## Architecture
**Source technology stack**
Amazon EC2 Windows instance in a staging subnet in your AMS account
**Target technology stack**
Amazon EC2 Windows instance managed by AWS Managed Services (AMS)
**Target architecture**
![\[Process to migrate and ingest Amazon EC2 Windows instances into an AWS Managed Services account.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/393c21cb-b6c6-4446-b597-b62e29fdb7f8/images/0b2fa855-7460-49f8-9e7f-3485e6ce1745.png)
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can use Amazon EC2 to launch as many or as few virtual servers as you need, and you can scale out or scale in.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Managed Services (AMS)](https://docs.aws.amazon.com/managedservices/?id=docs_gateway) helps you operate more efficiently and securely by providing ongoing management of your AWS infrastructure, including monitoring, incident management, security guidance, patch support, and backup for AWS workloads.
**Other services**
+ [PowerShell](https://learn.microsoft.com/en-us/powershell/) is a Microsoft automation and configuration management program that runs on Windows, Linux, and macOS.
## Epics
### Configure settings on the instance
| Task | Description | Skills required |
| --- | --- | --- |
| Change the DNS Client settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
| Change the Windows Update settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
| Enable the firewall. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
### Prepare the instance for AMS WIGS
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up and prepare the instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
| Repair the sppnp.dll file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
| Run the pre-WIG validation script. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
| Create the failsafe AMI. | After the pre-WIG validation passes, create a pre-ingestion AMI as follows:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html)For more information, see [AMI \$1 Create](https://docs.aws.amazon.com/managedservices/latest/ctref/deployment-advanced-ami-create.html) in the AMS documentation. | Migration engineer |
### Ingest and validate the instance
| Task | Description | Skills required |
| --- | --- | --- |
| Submit the RFC to create the workload ingest stack. | Submit a request for change (RFC) to start the AMS WIGS. For instructions, see [Workload Ingest Stack: Creating](https://docs.aws.amazon.com/managedservices/latest/appguide/ex-workload-ingest-col.html) in the AMS documentation. This starts the workload ingestion and installs all the software required by AMS, including backup tools, Amazon EC2 management software, and antivirus software. | Migration engineer |
| Validate successful migration. | After the workload ingestion is complete, you can see the AMS-managed instance and AMS-ingested AMI.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
### Launch the instance in the target AMS account
| Task | Description | Skills required |
| --- | --- | --- |
| Submit the RFC to create an EC2 stack. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.html) | Migration engineer |
## Related resources
**AWS Prescriptive Guidance**
+ [Automate pre-workload ingestion activities for AWS Managed Services on Windows](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-pre-workload-ingestion-activities-for-aws-managed-services-on-windows.html)
+ [Automatically create an RFC in AMS using Python](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-create-an-rfc-in-ams-using-python.html?did=pg_card&trk=pg_card)
**AMS documentation**
+ [AMS Workload Ingest](https://docs.aws.amazon.com/managedservices/latest/appguide/ams-workload-ingest.html)
+ [How Migration Changes Your Resource](https://docs.aws.amazon.com/managedservices/latest/appguide/ex-migrate-changes.html)
+ [Migrating Workloads: Standard Process](https://docs.aws.amazon.com/managedservices/latest/appguide/mp-migrate-stack-process.html)
**Marketing resources**
+ [AWS Managed Services](https://aws.amazon.com/managed-services/)
+ [AWS Managed Services FAQs](https://aws.amazon.com/managed-services/faqs/)
+ [AWS Managed Services Resources](https://aws.amazon.com/managed-services/resources/)
+ [AWS Managed Services Features](https://aws.amazon.com/managed-services/features/)
# Migrate a Couchbase Server database to Amazon EC2
*Subhani Shaik, Amazon Web Services*
## Summary
This pattern describes how you can migrate Couchbase Server from an on-premises environment to Amazon Elastic Compute Cloud (Amazon EC2) on AWS.
Couchbase Server is a distributed NoSQL (JSON document) database that provides relational database capabilities. Migrating a Couchbase Server database to AWS can provide increased scalability, improved performance, cost efficiency, enhanced security, simplified management, and global reach, which can benefit applications that require high availability and low latency data access. You also gain access to advanced features through AWS managed services.
Couchbase Server on AWS provides the following key features:
+ Memory-first architecture
+ High availability, disaster recovery, and load balancing
+ Multi-master, multi-Region deployment for optimal performance
For more information about key benefits, see the [Additional information](#migrate-couchbase-server-ec2-additional) section and the [Couchbase website](https://www.couchbase.com/partners/amazon/).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account with a virtual private cloud (VPC), two Availability Zones, private subnets, and a security group. For instructions, see [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html) in the Amazon Virtual Private Cloud (Amazon VPC) documentation.
+ Connectivity enabled between source and target environments. For information about the TCX ports used by Couchbase Server, see the [Couchbase documentation](https://docs.couchbase.com/server/current/install/install-ports.html).
## Architecture
The following diagram shows the high-level architecture for migrating Couchbase Server to AWS.
![\[Migration architecture for rehosting Couchbase Server on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4cedced2-3528-4f12-b19e-7d389e820cc1/images/ac22133a-895f-4999-b1e1-57f69e83a326.png)
From the on-premises Couchbase cluster, data moves through a customer gateway by using [AWS Direct Connect](https://aws.amazon.com/directconnect/). The data passes through a router and an Direct Connect route and reaches the VPC through an [AWS Virtual Private Network (Site-to-Site VPN)](https://aws.amazon.com/vpn/) gateway. The VPC contains an EC2 instance that is running Couchbase Server. The AWS infrastructure also includes [AWS Identity and Access Management (IAM)](https://aws.amazon.com/iam/) for access control, [AWS Key Management Service (AWS KMS)](https://aws.amazon.com/kms/) for data encryption, [Amazon Elastic Block Store (Amazon EBS)](https://aws.amazon.com/ebs/) for block storage, and [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) for data storage.
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
## Best practices
+ [Installing and configuring Couchbase](https://docs.couchbase.com/server/current/install/install-intro.html) on different operating platforms
+ [Best practices](https://docs.couchbase.com/server/current/cloud/couchbase-cloud-deployment.html#aws-best-practices) for deploying Couchbase Server on AWS
+ [Creating a Couchbase cluster](https://docs.couchbase.com/server/current/manage/manage-nodes/create-cluster.html)
+ [Performance best practices](https://docs.couchbase.com/dotnet-sdk/current/project-docs/performance.html) for Couchbase applications
+ [Security best practices](https://docs.couchbase.com/server/current/learn/security/security-overview.html) for Couchbase Server
+ [Storage best practices](https://www.couchbase.com/forums/t/what-is-the-best-document-storage-strategy-in-couchbase/1573) for Couchbase Server databases
## Epics
### Deploy an Amazon EC2 instance for Couchbase Server
| Task | Description | Skills required |
| --- | --- | --- |
| Open the Amazon EC2 console. | Sign in to the [AWS Management Console](https://console.aws.amazon.com/) and open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/). | DevOps engineer, Couchbase administrator |
| Deploy an Amazon EC2 instance. | Launch an EC2 instance that matches the on-premises Couchbase Server configurations. For more information about how to deploy an EC2 instance, see [Launch an Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html) in the Amazon EC2 documentation. | DevOps engineer, Couchbase administrator |
### Install and configure Couchbase Server on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Install a Couchbase cluster. | Review [Couchbase Server deployment guidelines](https://docs.couchbase.com/server/current/install/install-production-deployment.html) before you install Couchbase Server on Amazon EC2.To install Couchbase Server, see the [Couchbase Server documentation](https://docs.couchbase.com/server/current/install/install-intro.html) | Couchbase administrator |
| Configure the cluster. | To configure the cluster, see [Cluster Configuration Options](https://docs.couchbase.com/cloud/clusters/databases.html#cluster-configuration-options) in the Couchbase documentation. | Couchbase administrator |
### Add a new node and rebalance the Couchbase cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Add a node for the EC2 instance. | Add the newly deployed EC2 instance that has Couchbase installed to the existing on-premises cluster. For instructions, see [Add a Node and Rebalance](https://docs.couchbase.com/server/current/manage/manage-nodes/add-node-and-rebalance.html) in the Couchbase Server documentation. | Couchbase administrator |
| Rebalance the cluster. | The rebalancing process makes the newly added node with the EC2 instance an active member of the Couchbase cluster. For instructions, see [Add a Node and Rebalance](https://docs.couchbase.com/server/current/manage/manage-nodes/add-node-and-rebalance.html) in the Couchbase Server documentation | Couchbase administrator |
### Reconfigure connections
| Task | Description | Skills required |
| --- | --- | --- |
| Remove the on-premises nodes and rebalance. | You can now remove the on-premises nodes from the cluster. After removing the nodes, follow the rebalance process to redistribute data, indexes, event processing, and query processing among available nodes in the cluster. For instructions, see [Remove a Node and Rebalance](https://docs.couchbase.com/server/current/manage/manage-nodes/remove-node-and-rebalance.html) in the Couchbase Server documentation. | Couchbase administrator |
| Update connection parameters. | Update your application's connection parameters to use the new Amazon EC2 IP address, so your application can connect to the new node. | Couchbase application developer |
## Related resources
+ [Couchbase Server Services](https://docs.couchbase.com/server/current/learn/services-and-indexes/services/services.html)
+ [Deploy Couchbase Server Using AWS Marketplace](https://docs.couchbase.com/server/current/cloud/couchbase-aws-marketplace.html)
+ [Connect to Couchbase Server](https://docs.couchbase.com/server/current/guides/connect.html)
+ [Manage Buckets](https://docs.couchbase.com/server/current/manage/manage-buckets/bucket-management-overview.html)
+ [Cross Data Center Replication (XDCR)](https://docs.couchbase.com/server/current/learn/clusters-and-availability/xdcr-overview.html)
+ [Couchbase Inc. License Agreement](https://www.couchbase.com/LA20190115/)
## Additional information
**Key benefits**
Migrating your Couchbase database to AWS provides the following advantages:
**Scalability**. You can scale your Couchbase cluster up or down based on demand without having to manage physical hardware, so you can easily accommodate fluctuating data volumes and application usage. AWS provides:
+ Vertical and horizontal scaling options
+ [Global deployment](https://aws.amazon.com/about-aws/global-infrastructure/) capabilities
+ Load balancing across AWS Regions
+ [Database scaling solutions](https://aws.amazon.com/blogs/database/scaling-your-amazon-rds-instance-vertically-and-horizontally/)
+ [Content delivery](https://aws.amazon.com/solutions/content-delivery/) optimization
**Performance optimization**. AWS provides a high-performance network infrastructure and [optimized instance types](https://aws.amazon.com/ec2/instance-types/) to ensure fast data access and low latency for your Couchbase database.
+ [High performance computing (HPC)](https://aws.amazon.com/hpc/) options
+ Global content delivery through [Amazon CloudFront](https://aws.amazon.com/cloudfront/)
+ Multiple [storage options](https://aws.amazon.com/products/storage/)
+ Advanced [database services](https://aws.amazon.com/products/databases/), including Amazon Relational Database Service (Amazon RDS) and Amazon DynamoDB
+ Low-latency connections with [Direct Connect](https://aws.amazon.com/directconnect/)
**Cost optimization**. Select the appropriate instance type and configuration to balance performance and cost based on your workload. Pay only for the resources you use. This can potentially reduce your operational costs by eliminating the need to manage on-premises hardware and taking advantage of AWS Cloud economies of scale.
+ [Reserved instances ](https://aws.amazon.com/ec2/pricing/reserved-instances/)can help you plan ahead and reduce your costs substantially when you use Couchbase on AWS.
+ [Automatic scaling](https://aws.amazon.com/autoscaling/) prevents over-provisioning and helps you optimize your utilization and cost efficiencies.
**Enhanced security**. Benefit from the robust security features on AWS, such as data encryption, access controls, and security groups to help protect the sensitive data you store in Couchbase. Additional benefits:
+ The [AWS Shared Responsibility Model](https://aws.amazon.com/compliance/shared-responsibility-model/) clearly differentiates between the security *of* the cloud (AWS responsibility) and security *in* the cloud (customer responsibility).
+ [AWS compliance](https://aws.amazon.com/compliance/) supports major security standards.
+ AWS provides advanced [encryption](https://docs.aws.amazon.com/prescriptive-guidance/latest/encryption-best-practices/welcome.html) options.
+ [AWS Identity and Access Management (IAM)](https://aws.amazon.com/iam/) helps you manage secure access to your resources.
**Simplified management**. AWS provides managed services for Couchbase, so you can focus on application development instead of managing the underlying infrastructure.
**Global reach**. You can deploy your Couchbase cluster across multiple AWS Regions to achieve low latency for users around the world. You can deploy your databases entirely in the cloud or in a hybrid environment. You can safeguard your data with built-in enterprise-grade security and fast, efficient bidirectional synchronization of data from the edge to the cloud. At the same time, you can simplify development with a consistent programming model for building web and mobile apps.
**Business continuity**:
+ **Data backup and recovery**. In case of an issue, you can use [AWS Backup](https://aws.amazon.com/backup/) to ensure data resiliency and easy recovery. For disaster recovery options, see the [AWS Well-Architected Framework documentation](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html).
+ **Couchbase multi-Region deployment**: To deploy a Couchbase database in a multi-Region AWS environment, you can subscribe to Couchbase Server in [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-zy5g2wqmqdyzw), use [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)templates to create separate Couchbase clusters in each Region, and then configure cross-Region replication to synchronize data across Regions. This configuration ensures high availability and geographic redundancy across multiple Regions. For more information, see [Deploy Couchbase Server Using AWS Marketplace](https://docs.couchbase.com/server/current/cloud/couchbase-aws-marketplace.html) in the Couchbase documentation.
**Infrastructure agility**:
+ Rapid [resource provisioning](https://aws.amazon.com/products/management-and-governance/use-cases/provisioning-and-orchestration/) and deprovisioning
+ [Global infrastructure](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) reach
+ [Automatic scaling ](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scale-based-on-demand.html)based on demand
+ [Infrastructure as Code (IaC)](https://aws.amazon.com/what-is/iac/) for consistent deployments
+ Multiple[ instance types ](https://aws.amazon.com/ec2/instance-types/)that are optimized for different workloads
**Innovation enablement**:
+ Access to the latest technologies, including [AI/ML](https://aws.amazon.com/ai/generative-ai/),[ IoT](https://aws.amazon.com/iot/), and [analytics](https://aws.amazon.com/big-data/datalakes-and-analytics/)
+ [Managed services](https://aws.amazon.com/blogs/architecture/reduce-operational-load-using-aws-managed-services-for-your-data-solutions/), which reduce operational overhead
+ [Modern application](https://aws.amazon.com/modern-apps/) development practices
+ [Serverless ](https://aws.amazon.com/serverless/)computing options
**Operational excellence**:
+ [Centralized monitoring and logging](https://docs.aws.amazon.com/prescriptive-guidance/latest/designing-control-tower-landing-zone/logging-monitoring.html)
+ [Automated resource management](https://aws.amazon.com/systems-manager/)
+ [Predictive maintenance](https://aws.amazon.com/what-is/predictive-maintenance/) capabilities
+ [Enhanced visibility](https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-cloudwatch-provides-centralized-visibility-telemetry-configurations/) into resource usage
+ [Streamlined deployment processes](https://aws.amazon.com/blogs/mt/streamline-change-processes-and-improve-governance-with-aws-well-architected/)
**Modernization opportunities**:
+ [Microservices](https://aws.amazon.com/microservices/) architecture
+ [DevOps](https://aws.amazon.com/devops/) practices implementation
+ [Cloud-native](https://aws.amazon.com/what-is/cloud-native/) application development
+ [Legacy application modernization](https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-modernizing-applications/welcome.html)
**Competitive advantages**:
+ [Faster time to market](https://aws.amazon.com/blogs/smb/accelerate-time-to-market-and-business-growth-with-an-automated-software-as-a-service-platform/)
+ Improved[ customer experience](https://aws.amazon.com/blogs/publicsector/improving-customer-experience-for-the-public-sector-using-aws-services/)
+ [Data-driven](https://aws.amazon.com/data/data-driven-decision-making/) decision-making
+ Enhanced [business intelligence](https://aws.amazon.com/what-is/business-intelligence/)
# Migrate Db2 for LUW to Amazon EC2 by using log shipping to reduce outage time
*Feng Cai, Ambarish Satarkar, and Saurabh Sharma, Amazon Web Services*
## Summary
When customers migrate their IBM Db2 for LUW (Linux, UNIX, and Windows) workloads to Amazon Web Services (AWS), using Amazon Elastic Compute Cloud (Amazon EC2) with the Bring Your Own License (BYOL) model is the fastest way. However, migrating large amounts of data from on-premises Db2 into AWS can be a challenge, especially when the outage window is short. Many customers try to set the outage window to less than 30 minutes, which leaves little time for the database itself.
This pattern covers how to accomplish a Db2 migration with a short outage window by using transaction log shipping. This approach applies to Db2 on a little-endian Linux platform.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A Db2 instance running on EC2 instance that matches the on-premises file system layouts
+ An Amazon Simple Storage Service (Amazon S3) bucket accessible to the EC2 instance
+ An AWS Identity and Access Management (IAM) policy and role to make programmatic calls to Amazon S3
+ Synchronized time zone and system clocks on Amazon EC2 and the on-premises server
+ The on-premises network connected to AWS through [AWS Site-to-Site VPN](https://aws.amazon.com/vpn/) or [AWS Direct Connect](https://aws.amazon.com/directconnect/)
**Limitations**
+ The Db2 on-premises instance and Amazon EC2 must be on the same [platform family](https://www.ibm.com/docs/en/db2/11.1?topic=dbrs-backup-restore-operations-between-different-operating-systems-hardware-platforms).
+ The Db2 on-premises workload must be logged. To block any unlogged transaction, set `blocknonlogged=yes` in the database configuration.
**Product versions**
+ Db2 for LUW version 11.5.9 and later
## Architecture
**Source technology stack**
+ Db2 on Linux** **x86\$164
** Target technology stack**
+ Amazon EBS
+ Amazon EC2
+ AWS Identity and Access Management (IAM)
+ Amazon S3
+ AWS Site-to-Site VPN or Direct Connect
**Target architecture**
The following diagram shows one Db2 instance running on-premises with a virtual private network (VPN) connection to Db2 on Amazon EC2. The dotted lines represent the VPN tunnel between your data center and the AWS Cloud.
![\[Workflow to accomplish a Db2 migration within short outage window using transaction log shipping.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7dec6e4a-a92e-4204-9e42-f89d7dcafbfa/images/a7e1c1d6-2ec1-4271-952d-a58260ad7c81.png)
## Tools
**AWS services**
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) provides block-level storage volumes for use with Amazon Elastic Compute Cloud (Amazon EC2) instances.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Site-to-Site VPN](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html) helps you pass traffic between instances that you launch on AWS and your own remote network.
**Other tools**
+ [db2cli](https://www.ibm.com/docs/en/db2/11.5?topic=commands-db2cli-db2-interactive-cli) is the Db2 interactive CLI command.
## Best practices
+ On the target database, use [gateway endpoints for Amazon S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) to access the database backup image and log files in Amazon S3.
+ On the source database, use [AWS PrivateLink for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) to send the database backup image and log files to Amazon S3.
## Epics
### Set environment variables
| Task | Description | Skills required |
| --- | --- | --- |
| Set environment variables. | This pattern uses the following names:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-db2-for-luw-to-amazon-ec2-by-using-log-shipping-to-reduce-outage-time.html)You can change them to fit your environment. | DBA |
### Configure the on-premises Db2 server
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the AWS CLI. | To download and install the latest version of the AWS CLI, run the following commands:
| Linux administrator |
| Set up a local destination for Db2 archive logs. | To keep the target database on Amazon EC2 in sync with the on-premises source database, the latest transaction logs need be retrieved from the source.In this setup, `/db2logs` is set by `LOGARCHMETH2` on the source as a staging area. The archived logs in this directory will be synced into Amazon S3 and accessed by Db2 on Amazon EC2. The pattern uses `LOGARCHMETH2` because `LOGARCHMETH1` might have been configured to use a third-party vendor tool that AWS CLI command cannot access. To retrieve the logs, run the following command:
db2 connect to sample db2 update db cfg for SAMPLE using LOGARCHMETH2 disk:/db2logs
| DBA |
| Run an online database backup. | Run an online database backup, and save it to the local backup file system:
db2 backup db sample online to /backup
| DBA |
### Set up the S3 bucket and IAM policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | Create an S3 bucket for the on-premises server to send the backup Db2 images and log files to on AWS. The bucket will also be accessed by Amazon EC2:
| AWS administrator, AWS systems administrator |
| Attach the IAM policy to the IAM role used by the EC2 instance. | In most AWS environments, a running EC2 instance has an IAM Role set by your systems administrator. If the IAM role is not set, create the role and choose **Modify IAM role** on the EC2 console to associate the role with the EC2 instance that hosts the Db2 database. Attach the IAM policy to the IAM role with the policy ARN:
aws iam attach-role-policy \ --policy-arn "arn:aws:iam::aws_account_id:policy/db2s3policy" \ --role-name db2s3role
After the policy is attached, any EC2 instance associated with the IAM role can access the S3 bucket. | AWS administrator, AWS systems administrator |
### Send the source database backup image and log files to Amazon S3
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the AWS CLI on the on-premises Db2 server. | Configure the AWS CLI with the `Access Key ID` and `Secret Access Key` generated in the earlier step:
$ aws configure AWS Access Key ID [None]: ************* AWS Secret Access Key [None]: *************************** Default region name [None]: us-east-1 Default output format [None]: json
| AWS administrator, AWS systems administrator |
| Send the backup image to Amazon S3. | Earlier, an online database backup was saved to the `/backup` local directory. To send that backup image to the S3 bucket, run the following command:
| AWS administrator, Migration engineer |
| Send the Db2 archive logs to Amazon S3. | Sync the on-premises Db2 archive logs with the S3 bucket that can be accessed by the target Db2 instance on Amazon EC2:
Run this command periodically by using cron or other scheduling tools. The frequency depends on how often the source database archives transaction log files. | AWS administrator, Migration engineer |
### Connect Db2 on Amazon EC2 to Amazon S3 and start the database sync
| Task | Description | Skills required |
| --- | --- | --- |
| Create a PKCS12 keystore. | Db2 uses a Public-Key Cryptography Standards (PKCS) encryption keystore to keep the AWS access key secure. Create a keystore and configure the source Db2 instance to use it:
db2 "update dbm cfg using keystore_location /home/db2inst1/.keystore/db2s3.p12 keystore_type pkcs12"
| DBA |
| Create the Db2 storage access alias. | To create the [storage access alias](https://www.ibm.com/docs/en/db2/11.5?topic=commands-catalog-storage-access), use the following script syntax:`db2 "catalog storage access alias vendor S3 server container ''"`For example, your script might look like the following: `db2 "catalog storage access alias DB2AWSS3 vendor S3 server s3.us-east-1.amazonaws.com container 'logshipmig-db2'" ` | DBA |
| Set the staging area. | By default, Db2 uses `DB2_OBJECT_STORAGE_LOCAL_STAGING_PATH` as the staging area to upload and download files to and from Amazon S3. The default path is `sqllib/tmp/RemoteStorage.xxxx` under the instance home directory, with `xxxx` referring to the Db2 partition number. Note that the staging area must have enough capacity to hold the backup images and log files. You can use the registry to point the staging area into a different directory.We also recommend using `DB2_ENABLE_COS_SDK=ON`, `DB2_OBJECT_STORAGE_SETTINGS=EnableStreamingRestore`, and the link to the `awssdk` library to bypass the Amazon S3 staging area for database backup and restore:
| DBA |
| Restore the database from the backup image. | Restore the target database on Amazon EC2 from the backup image in the S3 bucket:
db2 restore db sample from DB2REMOTE://DB2AWSS3/logshipmig-db2/SAMPLE_backup replace existing
| DBA |
| Roll forward the database. | After the restore is complete, the target database will be put into rollforward pending state. Configure `LOGARCHMETH1` and `LOGARCHMETH2` so that Db2 knows where to get the transaction log files:
db2 update db cfg for SAMPLE using LOGARCHMETH1 'DB2REMOTE://DB2AWSS3//SAMPLE_LOGS/' db2 update db cfg for SAMPLE using LOGARCHMETH2 OFF
Start database rollforward:
db2 ROLLFORWARD DATABASE sample to END OF LOGS
This command processes all log files that have been transferred to the S3 bucket. Run it periodically based on the frequency of the `s3 sync` command on the on-premises Db2 servers. For example, if `s3 sync` runs at each hour, and it takes 10 minutes to sync all the log files, set the command to run at 10 minutes after each hour. | DBA |
### Bring Db2 on Amazon EC2 online during the cutover window
| Task | Description | Skills required |
| --- | --- | --- |
| Bring the target database online. | During the cutover window, do one of the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-db2-for-luw-to-amazon-ec2-by-using-log-shipping-to-reduce-outage-time.html)After the last transaction log is synced into Amazon S3, run the `ROLLFORWARD` command for the final time:
db2 rollforward DB sample to END OF LOGS db2 rollforward DB sample complete
Rollforward Status .... Rollforward status = not pending .... DB20000I The ROLLFORWARD command completed successfully.
db2 activate db sample DB20000I The ACTIVATE DATABASE command completed successfully.
Bring the target database online, and point the application connections to Db2 on Amazon EC2. | DBA |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| If multiple databases have the same instance name and database name on different hosts (DEV, QA, PROD), backups and logs might go to the same subdirectory. | Use different S3 buckets for DEV, QA, and PROD, and add the hostname as subdirectory prefix to avoid confusion. |
| If there are multiple backup images in the same location, you will get the following error when you restore:`SQL2522N More than one backup file matches the time stamp value provided for the backed up database image.` | In the `restore` command, add the timestamp of the backup:`db2 restore db sample from DB2REMOTE://DB2AWSS3/logshipmig-db2/SAMPLE_backup taken at 20230628164042 replace existing` |
## Related resources
+ [Db2 backup and restore operations between different operating systems and hardware platforms](https://www.ibm.com/docs/en/db2/11.5?topic=dbrs-backup-restore-operations-between-different-operating-systems-hardware-platforms)
+ [Set up Db2 STORAGE ACCESS ALIAS and DB2REMOTE](https://www.ibm.com/docs/en/db2/11.5?topic=commands-catalog-storage-access)
+ [Db2 ROLLFORWARD command](https://www.ibm.com/docs/en/db2/11.5?topic=commands-rollforward-database)
+ [Db2 secondary log archive method](https://www.ibm.com/docs/en/db2/11.5?topic=parameters-logarchmeth2-secondary-log-archive-method)
# Migrate Db2 for LUW to Amazon EC2 with high availability disaster recovery
*Feng Cai, Aruna Gangireddy, and Venkatesan Govindan, Amazon Web Services*
## Summary
When customers migrate their IBM Db2 LUW (Linux, UNIX, and Windows) workload to Amazon Web Services (AWS), using Amazon Elastic Compute Cloud (Amazon EC2) with the Bring Your Own License (BYOL) model is the fastest way. However, migrating large amounts of data from on-premises Db2 into AWS can be a challenge, especially when the outage window is short. Many customers try to set the outage window to less than 30 minutes, which leaves little time for the database itself.
This pattern covers how to accomplish a Db2 migration with a short outage window by using Db2 high availability disaster recovery (HADR). This approach applies to Db2 databases that are on the little-endian Linux platform and are not using Data Partitioning Feature (DPF).
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A Db2 instance running on an Amazon EC2 instance that matches the on-premises file system layouts
+ An Amazon Simple Storage Service (Amazon S3) bucket accessible to the EC2 instance
+ An AWS Identity and Access Management (IAM) policy and role to make programmatic calls to Amazon S3
+ Synchronized time zone and system clocks on Amazon EC2 and the on-premises server
+ The on-premises network connected to AWS through [AWS Site-to-Site VPN](https://aws.amazon.com/vpn/) or [AWS Direct Connect](https://aws.amazon.com/directconnect/)
+ Communication between the on-premises server and Amazon EC2 on HADR ports
**Limitations **
+ The Db2 on-premises instance and Amazon EC2 must be on the same [platform family](https://www.ibm.com/docs/en/db2/11.1?topic=dbrs-backup-restore-operations-between-different-operating-systems-hardware-platforms).
+ HADR is not supported in a partitioned database environment.
+ HADR doesn’t support the use of raw I/O (direct disk access) for database log files.
+ HADR doesn’t support infinite logging.
+ `LOGINDEXBUILD` must be set to `YES`, which will increase the log usage for rebuilding the index.
+ The Db2 on-premises workload must be logged. Set `blocknonlogged=yes` in the database configuration to block any unlogged transactions.
**Product versions**
+ Db2 for LUW version 11.5.9 and later
## Architecture
**Source technology stack**
+ Db2 on Linux** **x86\$164
**Target technology stack**
+ Amazon EC2
+ AWS Identity and Access Management (IAM)
+ Amazon S3
+ AWS Site-to-Site VPN
**Target architecture**
In the following diagram, Db2 on premises is running on `db2-server1` as the primary. It has two HADR standby targets. One standby target is on premises and is optional. The other standby target, `db2-ec2`, is on Amazon EC2. After the database is cut over to AWS, `db2-ec2` becomes the primary.
![\[Workflow to migrate with a short outage window an on-premises Db2 by using Db2 HADR.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2db43e4b-f0ea-4a92-96da-4cafb7d3368b/images/5295420e-3cd8-4127-9a18-ade971c36339.png)
1. Logs are streamed from the primary on-premises database to the standby on-premises database.
1. Using Db2 HADR, logs are streamed from the primary on-premises database through Site-to-Site VPN to Db2 on Amazon EC2.
1. Db2 backup and archive logs are sent from the primary on-premises database to the S3 bucket on AWS.
## Tools
**AWS services**
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Site-to-Site VPN](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html) helps you pass traffic between instances that you launch on AWS and your own remote network.
**Other tools**
+ [db2cli](https://www.ibm.com/docs/en/db2/11.5?topic=commands-db2cli-db2-interactive-cli) is the Db2 interactive CLI command.
## Best practices
+ On the target database, use [gateway endpoints for Amazon S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) to access the database backup image and log files in Amazon S3.
+ On the source database, use [AWS PrivateLink for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) to send the database backup image and log files to Amazon S3.
## Epics
### Set environment variables
| Task | Description | Skills required |
| --- | --- | --- |
| Set environment variables. | This pattern uses the following names and ports:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery.html)You can change them to fit your environment. | DBA |
### Configure the on-premises Db2 server
| Task | Description | Skills required |
| --- | --- | --- |
| Set up AWS CLI. | To download and install the latest version of AWS CLI, run the following commands:
| Linux administrator |
| Set up a local destination for Db2 archive logs. | Conditions such as heavy update batch jobs and network slowdowns can cause the HADR standby server to have a lag. To catch up, the standby server needs the transaction logs from the primary server. The sequence of places to request logs is the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery.html)In this setup, `/db2logs` is set by `LOGARCHMETH2` on the source as a staging area. The archived logs in this directory will be synced into Amazon S3 and accessed by Db2 on Amazon EC2. The pattern uses `LOGARCHMETH2` because `LOGARCHMETH1` might have been configured to use a third-party vendor tool that the AWS CLI command cannot access:
db2 connect to sample db2 update db cfg for SAMPLE using LOGARCHMETH2 disk:/db2logs
| DBA |
| Run an online database backup. | Run an online database backup, and save it to the local backup file system:
db2 backup db sample online to /backup
| DBA |
### Set up the S3 bucket and IAM policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | Create an S3 bucket for the on-premises server to send the backup Db2 images and log files to on AWS. The bucket will be accessed by Amazon EC2:
| AWS administrator, AWS systems administrator |
| Attach the IAM policy to the IAM role. | Usually, the EC2 instance with Db2 running would have an IAM role assigned by the systems administrator. If no IAM role is assigned, you can choose **Modify IAM role** on the Amazon EC2 console.Attach the IAM policy to the IAM role associated with the EC2 instance. After the policy is attached, the EC2 instance can access the S3 bucket:
aws iam attach-role-policy --policy-arn "arn:aws:iam::aws_account_id:policy/db2s3hapolicy" --role-name db2s3harole
| |
### Send the source database backup image and log files to Amazon S3
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS CLI on the on-premises Db2 server. | Configure AWS CLI with the `Access Key ID` and `Secret Access Key` that you generated earlier:
$ aws configure AWS Access Key ID [None]: ************* AWS Secret Access Key [None]: *************************** Default region name [None]: us-east-1 Default output format [None]: json
| AWS administrator, AWS systems administrator |
| Send the backup image to Amazon S3. | Earlier, an online database backup was saved to the `/backup` local directory. To send that backup image to the S3 bucket, run the following command:
| AWS administrator, AWS systems administrator |
| Send the Db2 archive logs to Amazon S3. | Sync the on-premises Db2 archive logs with the Amazon S3 bucket that can be accessed by the target Db2 instance on Amazon EC2:
aws s3 sync /db2logs s3://hadrmig-db2/SAMPLE_LOGS
Run this command periodically by using cron or other scheduling tools. The frequency depends on how often the source database archives transaction log files. | |
### Connect Db2 on Amazon EC2 to Amazon S3 and start the initial database sync
| Task | Description | Skills required |
| --- | --- | --- |
| Create a PKCS12 keystore. | Db2 uses a Public-Key Cryptography Standards (PKCS) encryption keystore to keep the AWS access key secure. Create a keystore, and configure the source Db2 to use it:
db2 "update dbm cfg using keystore_location /home/db2inst1/.keystore/db2s3.p12 keystore_type pkcs12"
| DBA |
| Create the Db2 storage access alias. | Db2 uses a storage access alias to access Amazon S3 directly with the `INGEST`, `LOAD`, `BACKUP DATABASE`, or `RESTORE DATABASE` commands. Because you assigned an IAM role to the EC2 instance, `USER` and `PASSWORD` are not required:`db2 "catalog storage access alias vendor S3 server container ''"`For example, your script might look like the following: `db2 "catalog storage access alias DB2AWSS3 vendor S3 server s3.us-east-1.amazonaws.com container 'hadrmig-db2'" ` | DBA |
| Set the staging area. | We recommend using `DB2_ENABLE_COS_SDK=ON`, `DB2_OBJECT_STORAGE_SETTINGS=EnableStreamingRestore`, and the link to the `awssdk` library to bypass the Amazon S3 staging area for database backup and restore:
| DBA |
| Restore the database from the backup image. | Restore the target database on Amazon EC2 from the backup image in the S3 bucket:
db2 create db sample on /data1 db2 restore db sample from DB2REMOTE://DB2AWSS3/hadrmig-db2/SAMPLE_backup replace existing
| DBA |
### Set up HADR with no HADR on premises
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the on-premises Db2 server as the primary. | Update the database configuration settings for HADR on `db2-server1` (the on-premises source) as the primary. Set `HADR_SYNCMODE` to `SUPERASYNC` mode, which has the shortest transaction response time:`db2 update db cfg for sample using HADR_LOCAL_HOST db2-server1 HADR_LOCAL_SVC 50010 HADR_REMOTE_HOST db2-ec2 HADR_REMOTE_SVC 50012 HADR_REMOTE_INST db2inst1 HADR_SYNCMODE SUPERASYNC DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully`Some network delays between the on-premises data center and AWS are expected. (You can set a different `HADR_SYNCMODE` value based on network reliability. For more information, see the [Related resources](#migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery-resources) section). | DBA |
| Change the target database log archive destination. | Change the target database log archive destination to match the Amazon EC2 environment:
db2 update db cfg for SAMPLE using LOGARCHMETH1 'DB2REMOTE://DB2AWSS3//SAMPLE_LOGS/' LOGARCHMETH2 OFF DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully
| DBA |
| Configure HADR for Db2 on the Amazon EC2 server. | Update database configuration for HADR on `db2-ec2` as standby:`db2 update db cfg for sample using HADR_LOCAL_HOST db2-ec2 HADR_LOCAL_SVC 50012 HADR_REMOTE_HOST db2-server1 HADR_REMOTE_SVC 50010 HADR_REMOTE_INST db2inst1 HADR_SYNCMODE SUPERASYNC DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully` | DBA |
| Verify HADR setup. | Verify the HADR parameters on the source and target Db2 servers.To verify the setup on `db2-server1`, run the following command:
db2 get db cfg for sample|grep HADR HADR database role = PRIMARY HADR local host name (HADR_LOCAL_HOST) = db2-server1 HADR local service name (HADR_LOCAL_SVC) = 50010 HADR remote host name (HADR_REMOTE_HOST) = db2-ec2 HADR remote service name (HADR_REMOTE_SVC) = 50012 HADR instance name of remote server (HADR_REMOTE_INST) = db2inst1 HADR timeout value (HADR_TIMEOUT) = 120 HADR target list (HADR_TARGET_LIST) = HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC HADR spool log data limit (4KB) (HADR_SPOOL_LIMIT) = AUTOMATIC(52000) HADR log replay delay (seconds) (HADR_REPLAY_DELAY) = 0 HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0 HADR SSL certificate label (HADR_SSL_LABEL) = HADR SSL Hostname Validation (HADR_SSL_HOST_VAL) = OFF
To verify the setup on `db2-ec2`, run the following command:
db2 get db cfg for sample|grep HADR HADR database role = STANDBY HADR local host name (HADR_LOCAL_HOST) = db2-ec2 HADR local service name (HADR_LOCAL_SVC) = 50012 HADR remote host name (HADR_REMOTE_HOST) = db2-server1 HADR remote service name (HADR_REMOTE_SVC) = 50010 HADR instance name of remote server (HADR_REMOTE_INST) = db2inst1 HADR timeout value (HADR_TIMEOUT) = 120 HADR target list (HADR_TARGET_LIST) = HADR log write synchronization mode (HADR_SYNCMODE) = SUPERASYNC HADR spool log data limit (4KB) (HADR_SPOOL_LIMIT) = AUTOMATIC(52000) HADR log replay delay (seconds) (HADR_REPLAY_DELAY) = 0 HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0 HADR SSL certificate label (HADR_SSL_LABEL) = HADR SSL Hostname Validation (HADR_SSL_HOST_VAL) = OFF
The `HADR_LOCAL_HOST`, `HADR_LOCAL_SVC`, `HADR_REMOTE_HOST`, and `HADR_REMOTE_SVC` parameters indicate the one primary and one standby HADR setup. | DBA |
| Start up the Db2 HADR instance. | Start the Db2 HADR instance on the standby server `db2-ec2` first:
db2 start hadr on db sample as standby DB20000I The START HADR ON DATABASE command completed successfully.
Start Db2 HADR on the primary (source) server `db2-server1`:
db2 start hadr on db sample as primary DB20000I The START HADR ON DATABASE command completed successfully.
The HADR connection between Db2 on premises and on Amazon EC2 has now been successfully established. The Db2 primary server `db2-server1` starts streaming transaction log records to `db2-ec2` in real time. | DBA |
### Set up HADR when HADR exists on premises
| Task | Description | Skills required |
| --- | --- | --- |
| Add Db2 on Amazon EC2 as an auxiliary standby. | If HADR is running on the on-premises Db2 instance, you can add Db2 on Amazon EC2 as an auxiliary standby using `HADR_TARGET_LIST` by running the following commands on `db2-ec2`:`db2 update db cfg for sample using HADR_LOCAL_HOST db2-ec2 HADR_LOCAL_SVC 50012 HADR_REMOTE_HOST db2-server1 HADR_REMOTE_SVC 50010 HADR_REMOTE_INST db2inst1 HADR_SYNCMODE SUPERASYNC DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. db2 update db cfg for sample using HADR_TARGET_LIST "db2-server1:50010\|db2-server2:50011" DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.` | DBA |
| Add the auxiliary standby information to the on-premises servers. | Update `HADR_TARGET_LIST` on the two on-premises servers (primary and standby).On `db2-server1`, run the following code:`db2 update db cfg for sample using HADR_TARGET_LIST "db2-server2:50011\|db2-ec2:50012" DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. SQL1363W One or more of the parameters submitted for immediate modification were not changed dynamically. For these configuration parameters, the database must be shutdown and reactivated before the configuration parameter changes become effective.`On `db2-server2`, run the following code:`db2 update db cfg for sample using HADR_TARGET_LIST "db2-server1:50010\|db2-ec2:50012" DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. SQL1363W One or more of the parameters submitted for immediate modification were not changed dynamically. For these configuration parameters, the database must be shutdown and reactivated before the configuration parameter changes become effective.` | DBA |
| Verify HADR setup. | Verify the HADR parameters on the source and target Db2 servers.On `db2-server1`, run the following code:
db2 get db cfg for sample|grep HADR HADR database role = PRIMARY HADR local host name (HADR_LOCAL_HOST) = db2-server1 HADR local service name (HADR_LOCAL_SVC) = 50010 HADR remote host name (HADR_REMOTE_HOST) = db2-server2 HADR remote service name (HADR_REMOTE_SVC) = 50011 HADR instance name of remote server (HADR_REMOTE_INST) = db2inst1 HADR timeout value (HADR_TIMEOUT) = 120 HADR target list (HADR_TARGET_LIST) = db2-server2:50011|db2-ec2:50012 HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC HADR spool log data limit (4KB) (HADR_SPOOL_LIMIT) = AUTOMATIC(52000) HADR log replay delay (seconds) (HADR_REPLAY_DELAY) = 0 HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0 HADR SSL certificate label (HADR_SSL_LABEL) = HADR SSL Hostname Validation (HADR_SSL_HOST_VAL) = OFF
On `db2-server2`, run the following code:
db2 get db cfg for sample|grep HADR HADR database role = STANDBY HADR local host name (HADR_LOCAL_HOST) = db2-server2 HADR local service name (HADR_LOCAL_SVC) = 50011 HADR remote host name (HADR_REMOTE_HOST) = db2-server1 HADR remote service name (HADR_REMOTE_SVC) = 50010 HADR instance name of remote server (HADR_REMOTE_INST) = db2inst1 HADR timeout value (HADR_TIMEOUT) = 120 HADR target list (HADR_TARGET_LIST) = db2-server1:50010|db2-ec2:50012 HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC HADR spool log data limit (4KB) (HADR_SPOOL_LIMIT) = AUTOMATIC(52000) HADR log replay delay (seconds) (HADR_REPLAY_DELAY) = 0 HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0 HADR SSL certificate label (HADR_SSL_LABEL) = HADR SSL Hostname Validation (HADR_SSL_HOST_VAL) = OFF
On `db2-ec2`, run the following code:
db2 get db cfg for sample|grep HADR HADR database role = STANDBY HADR local host name (HADR_LOCAL_HOST) = db2-ec2 HADR local service name (HADR_LOCAL_SVC) = 50012 HADR remote host name (HADR_REMOTE_HOST) = db2-server1 HADR remote service name (HADR_REMOTE_SVC) = 50010 HADR instance name of remote server (HADR_REMOTE_INST) = db2inst1 HADR timeout value (HADR_TIMEOUT) = 120 HADR target list (HADR_TARGET_LIST) = db2-server1:50010|db2-server2:50011 HADR log write synchronization mode (HADR_SYNCMODE) = SUPERASYNC HADR spool log data limit (4KB) (HADR_SPOOL_LIMIT) = AUTOMATIC(52000) HADR log replay delay (seconds) (HADR_REPLAY_DELAY) = 0 HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0 HADR SSL certificate label (HADR_SSL_LABEL) = HADR SSL Hostname Validation (HADR_SSL_HOST_VAL) = OFF
The `HADR_LOCAL_HOST`, `HADR_LOCAL_SVC`, `HADR_REMOTE_HOST`, `HADR_REMOTE_SVC`, and `HADR_TARGET_LIST` parameters indicate the one primary and two standby HADR setup. | |
| Stop and start Db2 HADR. | `HADR_TARGET_LIST` is now set up on all three servers. Each Db2 server is aware of the other two. Stop and restart HADR (brief outage) to take advantage of the new configuration.On `db2-server1`, run the following commands:
db2 stop hadr on db sample db2 deactivate db sample db2 activate db sample
On `db2-server2`, run the following commands:
db2 deactivate db sample db2 start hadr on db sample as standby SQL1766W The command completed successfully
On `db2-ec2`, run the following commands:
db2 start hadr on db sample as standby SQL1766W The command completed successfully
On `db2-server1`, run the following commands:
db2 start hadr on db sample as primary SQL1766W The command completed successfully
The HADR connection between Db2 on premises and on Amazon EC2 is now successfully established. The Db2 primary server `db2-server1` starts streaming transaction log records to both `db2-server2` and `db2-ec2` in real time. | DBA |
### Make Db2 on Amazon EC2 as primary during the cutover window
| Task | Description | Skills required |
| --- | --- | --- |
| Make sure that there is no HADR lag on the standby server. | Check HADR status from the primary server `db2-server1`. Don’t be alarmed when `HADR_STATE` is in `REMOTE_CATCHUP` status, which is normal when `HADR_SYNCMODE` is set to `SUPERASYNC`. The `PRIMARY_LOG_TIME` and `STANDBY_REPLAY_LOG_TIME` show that they are in sync:
| DBA |
| Run HADR takeover. | To complete the migration, make `db2-ec2` the primary database by running the HADR takeover command. Use the command `db2pd` to verify the `HADR_ROLE` value:
db2 TAKEOVER HADR ON DATABASE sample DB20000I The TAKEOVER HADR ON DATABASE command completed successfully.
db2pd -hadr -db sample Database Member 0 -- Database SAMPLE -- Active -- Up 0 days 00:03:25 -- Date 2022-10-26-02.46.45.048988
HADR_ROLE = PRIMARY REPLAY_TYPE = PHYSICAL
To complete the migration to AWS, point the application connections to Db2 on Amazon EC2. | |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| If you use NAT for firewall and security reasons, the host can have two IP addresses (one internal and one external), which can cause an HADR IP address check failure. The `START HADR ON DATABASE` command will return the following message:`HADR_LOCAL_HOST:HADR_LOCAL_SVC (-xx-xx-xx-xx.:50011 (xx.xx.xx.xx:50011)) on remote database is different from HADR_REMOTE_HOST:HADR_REMOTE_SVC (xx-xx-xx-xx.:50011 (x.x.x.x:50011)) on local database.` | To [support HADR in a NAT environment,](https://www.ibm.com/docs/en/db2/11.5?topic=support-hadr-nat) you can configure the `HADR_LOCAL_HOST` with both the internal and external address. For example, if the Db2 server has the internal name `host1` and the external name `host1E`, `HADR_LOCAL_HOST` can be `HADR_LOCAL_HOST: "host1 \| host1E"`. |
## Related resources
+ [Db2 backup and restore operations between different operating systems and hardware platforms](https://www.ibm.com/docs/en/db2/11.5?topic=dbrs-backup-restore-operations-between-different-operating-systems-hardware-platforms)
+ [Set up Db2 STORAGE ACCESS ALIAS and DB2REMOTE](https://www.ibm.com/docs/en/db2/11.5?topic=commands-catalog-storage-access)
+ [Db2 high availability disaster recovery](https://www.ibm.com/docs/en/db2/11.5?topic=server-high-availability-disaster-recovery-hadr)
+ [hadr\$1syncmode - HADR synchronization mode for log writes in peer state configuration parameter](https://www.ibm.com/docs/en/db2/11.5?topic=dcp-hadr-syncmode-hadr-synchronization-mode-log-writes-in-peer-state)
# Migrate IIS-hosted applications to Amazon EC2 by using appcmd.exe
*Deepak Kumar, Amazon Web Services*
## Summary
When you migrate Internet Information Services (IIS)-hosted applications to Amazon Elastic Compute Cloud (Amazon EC2) instances, you need to address several authentication challenges. These challenges include re-entering domain credentials for application pool identities and potentially regenerating machine keys for proper website functionality. You can use AWS Directory Service to establish trust relationships with your on-premises Active Directory or create a new managed Active Directory in AWS. This pattern describes a clean migration approach that uses the backup and restore functionality of IIS on Amazon EC2 instances. The approach uses appcmd.exe to uninstall and reinstall IIS on the target EC2 instances, enabling successful migration of IIS-hosted websites, application pool identities, and machine keys.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account for the target server.
+ A functional source IIS server with websites hosted on it.
+ Understanding of IIS working principles such as administration and configuration.
+ System administrator access on both the source and target servers.
+ Completed migration of the source IIS server to the target AWS account. You can use migration tools such as AWS Application Migration Service, an Amazon Machine Image (AMI) snapshot-based approach, or other migration tools.
**Limitations**
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
**Product versions**
+ IIS 8.5 or IIS 10.0
## Architecture
**Source technology stack **
+ Windows Server with IIS 8.5 or IIS 10.0 installed
**Target technology stack **
+ Windows Server with IIS 8.5 or IIS 10.0 installed
+ Application Migration Service
**Target architecture**
The following diagram shows the workflow and architecture components for this pattern.
![\[Workflow to migrate IIS-hosted applications to Amazon EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2f9f7757-b2bc-4077-b51a-700de521424c/images/36aa9b7a-d0aa-4fa4-be47-9fee43b53c22.png)
The solution includes the following steps:
1. [Install](https://docs.aws.amazon.com/mgn/latest/ug/agent-installation.html) and configure the AWS Replication Agent on the source IIS server in your corporate data center. This agent initiates the replication process and manages data transfer to AWS.
1. The AWS Replication Agent establishes a [secure connection ](https://docs.aws.amazon.com/mgn/latest/ug/Agent-Related-FAQ.html#How-Communication-Secured)to Application Migration Service and begins replicating the source server data, including IIS configurations, websites, and application files.
1. Application Migration Service launches EC2 instances in the application subnet with the replicated data. The target EC2 instance runs IIS and contains the migrated applications with their associated Amazon Elastic Block Store (Amazon EBS) volumes. After the initial replication, Application Migration Service continues to sync changes until you're [ready to cut over](https://docs.aws.amazon.com/mgn/latest/ug/migration-dashboard.html#ready-for-cutover1) to the new environment.
## Tools
**AWS services**
+ [AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) helps you rehost (*lift and shift*) applications to the AWS Cloud without change and with minimal downtime.
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/ebs/latest/userguide/what-is-ebs.html) provides block-level storage volumes for use with Amazon EC2 instances.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
**Other tools**
+ [Internet Information Services (IIS)](https://www.iis.net/overview) for Windows Server is a web server with a scalable and open architecture for hosting anything on the Web. IIS provides a set of administration tools, including administration and command line tools (for example, appcmd.exe), managed code and scripting APIs, and Windows PowerShell support.
## Epics
### Back up IIS at source prior to migration
| Task | Description | Skills required |
| --- | --- | --- |
| Create backups of IIS-hosted websites, configuration key, and `WAS` key. | To create backups for IIS-hosted websites, the configuration key (`iisConfigurationKey`), and the `WAS` key (`iisWasKey`), use appcmd.exe on the source server. Use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.html)To export the configuration key and the `WAS` key, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.html) | IIS Administrator |
### Uninstall and reinstall IIS on the target server
| Task | Description | Skills required |
| --- | --- | --- |
| Uninstall IIS on the target server. | To uninstall IIS on the target server, use the following steps: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.html) | IIS Administrator |
| Install IIS on the target server. | To install IIS on the target server, use the following steps: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.html) | IIS Administrator |
### Restore IIS websites and configuration from the backups
| Task | Description | Skills required |
| --- | --- | --- |
| Restore IIS websites and configuration. | To restore the IIS backups that you created from the source server on the target server, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.html) | IIS Administrator |
## Related resources
**AWS documentation**
+ [Installing the AWS Replication Agent](https://docs.aws.amazon.com/mgn/latest/ug/agent-installation.html) (AWS Application Migration Service documentation)
**AWS Prescriptive Guidance**
+ [Migrate an on-premises VM to Amazon EC2 by using AWS Application Migration Service](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-vm-to-amazon-ec2-by-using-aws-application-migration-service.html)
+ [Using AMIs or Amazon EBS snapshots for backups](https://docs.aws.amazon.com/prescriptive-guidance/latest/backup-recovery/ec2-backup.html#amis-snapshots)
**Microsoft resources**
+ [Application pool identities](https://learn.microsoft.com/en-us/troubleshoot/developer/webapps/iis/was-service-svchost-process-operation/understanding-identities#application-pool-identities)
+ [IIS documentation](https://learn.microsoft.com/en-us/iis/)
+ [IIS 8 appcmd.exe documentation](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/jj635852(v=ws.11))
+ [IIS 10 appcmd.exe documentation](https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/new-features-introduced-in-iis-10)
+ [Powerful Admin Tools](https://learn.microsoft.com/en-us/iis/overview/powerful-admin-tools)
# Migrate an on-premises Microsoft SQL Server database to Amazon EC2 using Application Migration Service
*Senthil Ramasamy, Amazon Web Services*
## Summary
This pattern describes the steps for migrating a Microsoft SQL Server database from an on-premises data center to an Amazon Elastic Compute Cloud (Amazon EC2) instance. It uses the AWS Application Migration Service (AWS MGN) to rehost your database using an automated lift-and-shift migration. AWS MGN performs block-level replication of your source database server.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A source Microsoft SQL Server database in an on-premises data center
**Limitations**
+ Your network bandwidth may be limited between the on-premises data center and AWS.
+ AWS MGN is limited to databases that are hosted on standalone servers with dedicated storage. It doesn’t support migrating clustered database systems and database systems where the rate of change exceeds a network’s throughput.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas page](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/), and choose the link for the service.
**Product versions**
+ All versions of Microsoft SQL Server database
+ Windows and Linux operating systems that [support AWS MGN](https://docs.aws.amazon.com/mgn/latest/ug/Supported-Operating-Systems.html)
## Architecture
**Source technology stack**
An on-premises Microsoft SQL Server database
**Target technology stack**
A Microsoft SQL Server database on an Amazon EC2 instance
**Target architecture**
![\[Replicate data from an on-premises corporate data center to AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a459eaef-c256-4691-a7ec-2304f634228c/images/d8d6cee7-f42c-4686-bf92-6e6d39adfb17.png)
This architecture uses AWS MGN to replicate data from an on-premises corporate data center to AWS. The diagram shows the data replication process, API communications, and the test and cutover phases.
1. Data replication:
+ AWS MGN replicates data from the on-premises corporate data center to AWS and initiates ongoing replication of changes.
+ Replication servers in the staging subnet receive and process the data.
1. API communication:
+ Replication servers connect to AWS MGN, Amazon EC2, and Amazon Simple Storage Service (Amazon S3) API endpoints through TCP port 443.
+ AWS MGN manages the migration.
+ Amazon EC2 manages instance operations.
1. Test and cutover:
+ Test instances launch in the operational subnet using replicated data.
+ After successful testing, AWS MGN creates cutover instances for the final migration.
## Tools
+ [AWS Application Migration Service (AWS MGN)](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) helps you rehost (*lift and shift*) applications to the AWS Cloud without change and with minimal downtime.
+ [Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
## Best practices
+ Set up API regional endpoints for AWS MGN, Amazon EC2, and Amazon S3 in the virtual private cloud (VPC) to prohibit public access from the internet.
+ Set up AWS MGN launch settings to launch target database servers in a private subnet.
+ Allow only required ports in database security groups.
+ Follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation.
## Epics
### Set up AWS MGN
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS MGN. | Search for the AWS Application Migration Service in the AWS Management Console, and initiate the setup process. This will create a replication template and redirect you to the MGN console **Source servers** page. As you configure the MGN service, choose a service role from the generated list. | DBA, Migration engineer |
| Add source server. | Add details of your on-premises source database server, and then add the server. | DBA, Migration engineer |
| Install the AWS MGN agent on the source server. | Download the AWS MGN agent installer to your local system, and transfer the installer to your source database server. To validate the installer hash, see Validating the downloaded [AWS Replication Agent installer for Windows 2012](https://docs.aws.amazon.com/mgn/latest/ug/windows-agent.html#installer-hash-table-2012). | DBA, Migration engineer |
### Install AWS MGN agent on source machines
| Task | Description | Skills required |
| --- | --- | --- |
| Generate client IAM credentials. | Before you install the AWS MGN agent, generate AWS credentials by creating a new IAM user with the appropriate permissions.For more information, see [AWS managed policies for AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/security-iam-awsmanpol.html) and [Generating the required AWS credentials](https://docs.aws.amazon.com/mgn/latest/ug/credentials.html). | DBA, Migration engineer |
| Install the agent on the source server. | Install the agent on the source machine that hosts the Microsoft SQL Server database. For more information, see [Installing the AWS Replication Agent on Windows servers](https://docs.aws.amazon.com/mgn/latest/ug/windows-agent.html).Provide the following AWS credentials:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.html)Your unique AWS credentials enable the AWS MGN agent to authenticate and perform migration tasks. | App owner, DBA, Migration engineer |
| Choose disks to replicate. | After entering your AWS credentials, the installer verifies that your server meets the minimum requirements for agent installation (for example, whether the server has enough disk space to install the AWS MGN agent). The installer displays the volume labels and storage details.To replicate your database using AWS MGN service, select the applicable disks on your source server. Enter the path of each disk, separated by commas. If you want to replicate all of the disks, leave the path blank. After you confirm the selected disks, the installation proceeds. | DBA, Migration engineer |
| Monitor synchronization progress. | AWS Replication Agent initiates the synchronization process by first taking a snapshot of the selected disks and then replicating the data.You can monitor the synchronization progress from the **Source server** page in the AWS MGN console. For more information, see [Monitor the server in the migration lifecycle](https://docs.aws.amazon.com/mgn/latest/ug/migration-dashboard.html). | DBA, Migration engineer |
### Replication using AWS MGN
| Task | Description | Skills required |
| --- | --- | --- |
| Manage replication progress. | After you start the initial synchronization, your source server appears in the AWS MGN console, where you can manage and monitor the migration. The console displays an estimated time for complete replication, which is based on the total size of selected disks and available network bandwidth. | DBA, Migration engineer |
| Verify the synchronization. | After the disks on the source server are fully synchronized, verify that all selected disks are listed as fully synced and no errors are reported in the console.The AWS MGN console will then automatically transition the migration lifecycle status to **Ready for testing**, indicating that the replicated environment in AWS is prepared for performance and functionality testing. | App owner, DBA, Migration engineer |
### Test and cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Configure launch settings. | Choose the source server in the AWS MGN console, and update the launch settings for the target test instance. From the source **Server details** page, navigate to the **Launch settings** tab to configure the test instance.Choose a cost-effective instance type and Amazon Elastic Block Store (Amazon EBS) volume type, and then configure the security groups and network requirements. For more information, see [Launch settings](https://docs.aws.amazon.com/mgn/latest/ug/launch-settings.html). | DBA, Migration engineer |
| Launch the target test instance. | Navigate to the AWS MGN console of your synchronized source machine, and launch a target test instance by choosing **Test and cut over** and then **Launch test instances**.This creates a launch job that deploys the test instance using your configured settings. The instance launches in the AWS Cloud and replicates your source database server's environment. Monitor the launch progress from the **Launch history** page, where you can track the instance creation and address any issues. | DBA, Migration engineer |
| Validate the target test instance. | Validate the Amazon EC2 database server:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.html)Conduct validation tests to ensure the database functions as expected. | DBA, Migration engineer |
| Rename the server. | AWS MGN migration involves a storage-level copy of your on-premises source server. Your SQL Server EC2 instance contains only the original source server's details in its binaries, so update the binary information to reflect the new server's name.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.html) | DBA, Migration engineer |
| Launch the cutover instance. | In the AWS MGN console, on the **Source servers** page, confirm that the migration lifecycle status of the server is **Ready for cutover**. Configure the launch settings for the cutover instance, ensuring that the settings mirror your on-premises environment.Before initiating the cutover, shut down your on-premises database, which ensures the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.html)Initiate the cutover instance in the AWS MGN console. When the cutover instance is operational, log in to the instance and perform the following tests:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.html) | App owner, DBA, Migration engineer, Migration lead |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The initial synchronization fails at the authentication step. | This is a network connectivity issue. The replication server can’t connect to AWS MGN. |
## Related resources
**AWS documentation**
+ [Getting started with AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/getting-started.html)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon EC2](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-microsoft-sql-server-database-to-amazon-ec2.html)
+ [What is Microsoft SQL Server on Amazon EC2?](https://docs.aws.amazon.com/sql-server-ec2/latest/userguide/sql-server-on-ec2-overview.html)
**Videos**
+ [Performing a Lift and Shift Migration with AWS Application Migration Service](https://www.youtube.com/watch?v=tB0sAR3aCb4) (video)
# Migrate an F5 BIG-IP workload to F5 BIG-IP VE on the AWS Cloud
*Deepak Kumar, Amazon Web Services*
## Summary
Organizations are looking to migrate to the AWS Cloud to increase their agility and resilience. After you migrate your [F5 BIG-IP ](https://www.f5.com/products/big-ip-services)security and traffic management solutions to the AWS Cloud, you can focus on agility and adoption of high-value operational models across your enterprise architecture.
This pattern describes how to migrate an F5 BIG-IP workload to an [F5 BIG-IP Virtual Edition (VE)](https://www.f5.com/products/big-ip-services/virtual-editions) workload on the AWS Cloud. The workload will be migrated by rehosting the existing environment and deploying aspects of replatforming, such as service discovery and API integrations. [AWS CloudFormation templates](https://github.com/F5Networks/f5-aws-cloudformation) accelerate your workload’s migration to the AWS Cloud.
This pattern is intended for technical engineering and architectural teams that are migrating F5 security and traffic management solutions, and accompanies the guide [Migrating from F5 BIG-IP to F5 BIG-IP VE on the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-f5-big-ip/welcome.html) on the AWS Prescriptive Guidance website.
## Prerequisites and limitations
**Prerequisites **
+ An existing on-premises F5 BIG-IP workload.
+ Existing F5 licenses for BIG-IP VE versions.
+ An active AWS account.
+ An existing virtual private cloud (VPC) configured with an egress through a NAT gateway or Elastic IP address, and configured with access to the following endpoints: Amazon Simple Storage Service (Amazon S3), Amazon Elastic Compute Cloud (Amazon EC2), AWS Security Token Service (AWS STS), and Amazon CloudWatch. You can also modify the [Modular and scalable VPC architecture](https://aws.amazon.com/quickstart/architecture/vpc/) Quick Start as a building block for your deployments.
+ One or two existing Availability Zones, depending on your requirements.
+ Three existing private subnets in each Availability Zone.
+ AWS CloudFormation templates, [available in the F5 GitHub repository](https://github.com/F5Networks/f5-aws-cloudformation/blob/master/template-index.md).
During the migration, you might also use the following, depending on your requirements:
+ An [F5 Cloud Failover Extension](https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/) to manage Elastic IP address mapping, secondary IP mapping, and route table changes.
+ If you use multiple Availability Zones, you will need to use the F5 Cloud Failover Extensions to handle the Elastic IP mapping to virtual servers.
+ You should consider using [F5 Application Services 3 (AS3)](https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/), [F5 Application Services Templates (FAST)](https://clouddocs.f5.com/products/extensions/f5-appsvcs-templates/latest/), or another infrastructure as code (IaC) model to manage the configurations. Preparing the configurations in an IaC model and using code repositories will help with the migration and your ongoing management efforts.
**Expertise**
+ This pattern requires familiarity with how one or more VPCs can be connected to existing data centers. For more information about this, see [Network-to-Amazon VPC connectivity options](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/network-to-amazon-vpc-connectivity-options.html) in the Amazon VPC documentation.
+ Familiarity is also required with F5 products and modules, including [Traffic Management Operating System (TMOS)](https://www.f5.com/services/resources/white-papers/tmos-redefining-the-solution), [Local Traffic Manager (LTM)](https://www.f5.com/products/big-ip-services/local-traffic-manager), [Global Traffic Manager (GTM)](https://techdocs.f5.com/kb/en-us/products/big-ip_gtm/manuals/product/gtm-concepts-11-5-0/1.html#unique_9842886), [Access Policy Manager (APM)](https://www.f5.com/products/security/access-policy-manager), [Application Security Manager (ASM)](https://www.f5.com/pdf/products/big-ip-application-security-manager-overview.pdf), [Advanced Firewall Manager (AFM)](https://www.f5.com/products/security/advanced-firewall-manager), and [BIG-IQ](https://www.f5.com/products/automation-and-orchestration/big-iq).
**Product versions**
+ We recommend that you use F5 BIG-IP [version 13.1](https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-bigip-ve-13-1-0.html) or later, although the pattern supports F5 BIG-IP [version 12.1](https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-bigip-12-1-4.html) or later.
## Architecture
**Source technology stack**
+ F5 BIG-IP workload
**Target technology stack **
+ Amazon CloudFront
+ CloudWatch
+ Amazon EC2
+ Amazon S3
+ Amazon VPC
+ AWS Global Accelerator
+ AWS STS
+ AWS Transit Gateway
+ F5 BIG-IP VE
**Target architecture **
![\[Architecture to migrate an F5 BIG-IP workload to an F5 BIG-IP VE workload.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/586fe806-fac1-48d3-9eb1-45a6c86430dc/images/16d7fc09-1ffe-4721-b503-d971db84cbae.png)
## Tools
+ [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions.
+ [Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html) speeds up distribution of your web content by delivering it through a worldwide network of data centers, which lowers latency and improves performance.
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) helps you monitor the metrics of your AWS resources and the applications you run on AWS in real time.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Security Token Service (AWS STS)](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) helps you request temporary, limited-privilege credentials for users.
+ [AWS Transit Gateway](https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html) is a central hub that connects virtual private clouds (VPCs) and on-premises networks.
+ [Amazon Virtual Private Cloud (Amazon VPC) ](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html)helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
## Epics
### Discovery and assessment
| Task | Description | Skills required |
| --- | --- | --- |
| Assess the performance of F5 BIG-IP. | Collect and record the performance metrics of the applications on the virtual server, and metrics of systems that will be migrated. This will help to correctly size the target AWS infrastructure for better cost optimization. | F5 Architect, Engineer and Network Architect, Engineer |
| Evaluate the F5 BIG-IP operating system and configuration. | Evaluate which objects will be migrated and if a network structure needs to be maintained, such as VLANs. | F5 Architect, Engineer |
| Evaluate F5 license options. | Evaluate which license and consumption model you will require. This assessment should be based on your evaluation of the F5 BIG-IP operating system and configuration. | F5 Architect, Engineer |
| Evaluate the public applications. | Determine which applications will require public IP addresses. Align those applications to the required instances and clusters to meet performance and service-level agreement (SLA) requirements. | F5 Architect, Cloud Architect, Network Architect, Engineer, App Teams |
| Evaluate internal applications. | Evaluate which applications will be used by internal users. Make sure you know where those internal users sit in the organization and how those environments connect to the AWS Cloud. You should also make sure those applications can use domain name system (DNS) as part of the default domain. | F5 Architect, Cloud Architect, Network Architect, Engineer, App Teams |
| Finalize the AMI. | Not all F5 BIG-IP versions are created as Amazon Machine Images (AMIs). You can use the F5 BIG-IP Image Generator Tool if you have specific required quick-fix engineering (QFE) versions. For more information about this tool, see the "Related resources" section. | F5 Architect, Cloud Architect, Engineer |
| Finalize the instance types and architecture. | Decide on the instance types, VPC architecture, and interconnected architecture. | F5 Architect, Cloud Architect, Network Architect, Engineer |
### Complete security and compliance-related activities
| Task | Description | Skills required |
| --- | --- | --- |
| Document the existing F5 security policies. | Collect and document existing F5 security policies. Make sure you create a copy of them in a secure code repository. | F5 Architect, Engineer |
| Encrypt the AMI. | (Optional) Your organization might require encryption of data at rest. For more information about creating a custom Bring Your Own License (BYOL) image, see the "Related resources" section. | F5 Architect, Engineer Cloud Architect, Engineer |
| Harden the devices. | This will help protect against potential vulnerabilities. | F5 Architect, Engineer |
### Configure your new AWS environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create edge and security accounts. | Sign in to the AWS Management Console and create the AWS accounts that will provide and operate the edge and security services. These accounts might be different from the accounts that operate VPCs for shared services and applications. This step can be completed as part of a landing zone. | Cloud Architect, Engineer |
| Deploy edge and security VPCs. | Set up and configure the VPCs required to deliver edge and security services. | Cloud Architect, Engineer |
| Connect to the source data center. | Connect to the source data center that hosts your F5 BIG-IP workload. | Cloud Architect, Network Architect, Engineer |
| Deploy the VPC connections. | Connect the edge and security service VPCs to the application VPCs. | Network Architect, Engineer |
| Deploy the instances. | Deploy the instances by using the CloudFormation templates from the "Related resources" section. | F5 Architect, Engineer |
| Test and configure instance failover. | Make sure that the AWS Advanced HA iAPP template or F5 Cloud Failover Extension is configured and operating correctly. | F5 Architect, Engineer |
### Configure networking
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the VPC topology. | Open the Amazon VPC console and make sure that your VPC has all the required subnets and protections for the F5 BIG-IP VE deployment. | Network Architect, F5 Architect, Cloud Architect, Engineer |
| Prepare your VPC endpoints. | Prepare the VPC endpoints for Amazon EC2, Amazon S3, and AWS STS if an F5 BIG-IP workload does not have access to a NAT Gateway or Elastic IP address on a TMM interface. | Cloud Architect, Engineer |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the configuration. | Migrate the F5 BIG-IP configuration to F5 BIG-IP VE on the AWS Cloud. | F5 Architect, Engineer |
| Associate the secondary IPs. | Virtual server IP addresses have a relationship with the secondary IP addresses assigned to the instances. Assign secondary IP addresses and make sure "Allow remap/reassignment" is selected. | F5 Architect, Engineer |
### Test configurations
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the virtual server configurations. | Test the virtual servers. | F5 Architect, App Teams |
### Finalize operations
| Task | Description | Skills required |
| --- | --- | --- |
| Create the backup strategy. | Systems must be shut down to create a full snapshot. For more information, see "Updating an F5 BIG-IP virtual machine" in the "Related resources" section. | F5 Architect, Cloud Architect, Engineer |
| Create the cluster failover runbook. | Make sure that the failover runbook process is complete. | F5 Architect, Engineer |
| Set up and validate logging. | Configure F5 Telemetry Streaming to send logs to the required destinations. | F5 Architect, Engineer |
### Complete the cutover
| Task | Description | Skills required |
| --- | --- | --- |
| Cut over to the new deployment. | | F5 Architect, Cloud Architect, Network Architect, Engineer, AppTeams |
## Related resources
**Migration guide**
+ [Migrating from F5 BIG-IP to F5 BIG-IP VE on the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-f5-big-ip/welcome.html)
**F5 resources**
+ [CloudFormation templates in the F5 GitHub repository](https://github.com/F5Networks/f5-aws-cloudformation)
+ [F5 in AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=74d946f0-fa54-4d9f-99e8-ff3bd8eb2745)
+ [F5 BIG-IP VE overview](https://www.f5.com/products/big-ip-services/virtual-editions)
+ [Example Quickstart - BIG-IP Virtual Edition with WAF (LTM \$1 ASM)](https://github.com/F5Networks/f5-aws-cloudformation-v2/tree/main/examples/quickstart)
+ [F5 Application services on AWS: an overview (video)](https://www.youtube.com/watch?v=kutVjRHOAXo)
+ [F5 Application Services 3 Extension User Guide ](https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/)
+ [F5 cloud documentation](https://clouddocs.f5.com/training/community/public-cloud/html/intro.html)
+ [F5 iControl REST wiki](https://clouddocs.f5.com/api/icontrol-rest/)
+ [F5 Overview of single configuration files (11.x - 15.x)](https://support.f5.com/csp/article/K13408)
+ [F5 whitepapers](https://www.f5.com/services/resources/white-papers)
+ [F5 BIG-IP Image Generator Tool](https://clouddocs.f5.com/cloud/public/v1/ve-image-gen_index.html)
+ [Updating an F5 BIG-IP VE virtual machine](https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-ve-setup-vmware-esxi-11-5-0/3.html)
+ [Overview of the UCS archive "platform-migrate" option](https://support.f5.com/csp/article/K82540512)
# Migrate an on-premises Go web application to AWS Elastic Beanstalk by using the binary method
*Suhas Basavaraj and Shumaz Mukhtar Kazi, Amazon Web Services*
## Summary
This pattern describes how to migrate an on-premises Go web application to AWS Elastic Beanstalk. After the application is migrated, Elastic Beanstalk builds the binary for the source bundle and deploys it to an Amazon Elastic Compute Cloud (Amazon EC2) instance.
As a rehost migration strategy, this pattern’s approach is fast and requires no code changes, which means less testing and migration time.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ An on-premises Go web application.
+ A GitHub repository that contains your Go application’s source code. If you do not use GitHub, there are other ways to [create an application source bundle for Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/applications-sourcebundle.html).
**Product versions**
+ The most recent Go version supported by Elastic Beanstalk. For more information, see the [Elastic Beanstalk documentation](https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platforms-supported.html#platforms-supported.go).
## Architecture
**Source technology stack **
+ An on-premises Go web application
**Target technology stack**
+ AWS Elastic Beanstalk
+ Amazon CloudWatch
**Target architecture*** *
![\[Architecture for migrating a Go application to Elastic Beanstalk\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/cd8d660d-5621-4ea7-8f97-7a1e321c57d3/images/1df543d9-7073-43d8-abd3-f1f7e57278eb.png)
## Tools
+ [AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html) quickly deploys and manages applications in the AWS Cloud without users having to learn about the infrastructure that runs those applications. Elastic Beanstalk reduces management complexity without restricting choice or control.
+ [GitHub](https://github.com/) is an open-source distributed version control system.
## Epics
### Create the Go web application source bundle .zip file
| Task | Description | Skills required |
| --- | --- | --- |
| Create the source bundle for the Go application. | Open the GitHub repository that contains your Go application’s source code and prepare the source bundle. The source bundle contains an `application.go` source file in the root directory, which hosts the main package for your Go application. If you do not use GitHub, see the *Prerequisites* section earlier in this pattern for other ways to create your application source bundle. | System Admin, Application Developer |
| Create a configuration file. | Create an `.ebextensions` folder in your source bundle, and then create an `options.config` file inside this folder. For more information, see the [Elastic Beanstalk documentation](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html). | System Admin, Application Developer |
| Create the source bundle .zip file. | Run the following command.
git archive -o ../godemoapp.zip HEAD
This creates the source bundle .zip file. Download and save the .zip file as a local file. The .zip file cannot exceed 512 MB and cannot include a parent folder or top-level directory. | System Admin, Application Developer |
### Migrate the Go web application to Elastic Beanstalk
| Task | Description | Skills required |
| --- | --- | --- |
| Choose the Elastic Beanstalk application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-go-web-application-to-aws-elastic-beanstalk-by-using-the-binary-method.html)For instructions on how to create an Elastic Beanstalk application, see the [Elastic Beanstalk documentation](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.CreateApp.html). | System Admin, Application Developer |
| Initiate the Elastic Beanstalk web server environment. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-go-web-application-to-aws-elastic-beanstalk-by-using-the-binary-method.html) | System Admin, Application Developer |
| Upload the source bundle .zip file to Elastic Beanstalk. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-go-web-application-to-aws-elastic-beanstalk-by-using-the-binary-method.html) | System Admin, Application Developer |
| Test the deployed Go web application. | You will be redirected to the Elastic Beanstalk application's overview page. At the top of the overview, next to **Environment ID**, choose the URL that ends in `elasticbeanstalk.com` to navigate to your application. Your application must use this name in its configuration file as an environment variable and display it on the web page. | System Admin, Application Developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Unable to access the application through an Application Load Balancer. | Check the target group that contains your Elastic Beanstalk application. If it’s unhealthy, log in to your Elastic Beanstalk instance and check the `nginx.conf` file configuration to verify that it routes to the correct health status URL. You might need to change the target group health check URL. |
## Related resources
+ [Go platform versions supported by Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platforms-supported.html#platforms-supported.go)
+ [Using configuration files with Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html)
+ [Creating an example application in Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.CreateApp.html)
# Migrate an on-premises SFTP server to AWS using AWS Transfer for SFTP
*Akash Kumar, Amazon Web Services*
## Summary
This pattern describes how to migrate an on-premises file transfer solution that uses the Secure Shell (SSH) File Transfer Protocol (SFTP) to the AWS Cloud by using the AWS Transfer for SFTP service. Users generally connect to an SFTP server either through its domain name or by fixed IP. This pattern covers both cases.
AWS Transfer for SFTP is a member of the AWS Transfer Family. It is a secure transfer service that you can use to transfer files into and out of AWS storage services over SFTP. You can use AWS Transfer for SFTP with Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS). This pattern uses Amazon S3 for storage.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An existing SFTP domain name or fixed SFTP IP.
**Limitations**
+ The largest object that you can transfer in one request is currently 5 GiB. For files that are larger than 100 MiB, consider using [Amazon S3 multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html).
## Architecture
**Source technology stack **
+ On-premises flat files or database dump files.
**Target technology stack **
+ AWS Transfer for SFTP
+ Amazon S3
+ Amazon Virtual Private Cloud (Amazon VPC)
+ AWS Identity and Access Management (IAM) roles and policies
+ Elastic IP addresses
+ Security groups
+ Amazon CloudWatch Logs (optional)
**Target architecture **
![\[Use AWS Transfer for SFTP to migrate an on-premises SFTP server to the AWS Cloud.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/ec0a905c-edef-48ba-9b5e-ea4a4040d320/images/f42aa711-bfe0-4ac6-9f66-5c18a1dd1c7a.png)
**Automation and scale**
To automate the target architecture for this pattern, use the attached CloudFormation templates:
+ `amazon-vpc-subnets.yml` provisions a virtual private cloud (VPC) with two public and two private subnets.
+ `amazon-sftp-server.yml` provisions the SFTP server.
+ `amazon-sftp-customer.yml` adds users.
## Tools
**AWS services**
+ [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) helps you centralize the logs from all your systems, applications, and AWS services so you can monitor them and archive them securely.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. This pattern uses Amazon S3 as the storage system for file transfers.
+ [AWS Transfer for SFTP](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-family.html) helps you transfer files into and out of AWS storage services over the SFTP protocol.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
## Epics
### Create a VPC
| Task | Description | Skills required |
| --- | --- | --- |
| Create a VPC with subnets. | Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/). Create a virtual private cloud (VPC) with two public subnets. (The second subnet provides high availability.)—or—You can deploy the attached CloudFormation template, `amazon-vpc-subnets.yml`, in the [CloudFormation console](https://console.aws.amazon.com/cloudformation) to automate the tasks in this epic. | Developer, Systems administrator |
| Add an internet gateway. | Provision an internet gateway and attach it to the VPC. | Developer, Systems administrator |
| Migrate an existing IP. | Attach an existing IP to the Elastic IP address. You can create an Elastic IP address from your address pool and use it. | Developer, Systems administrator |
### Provision an SFTP server
| Task | Description | Skills required |
| --- | --- | --- |
| Create an SFTP server. | Open the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/). Follow the instructions in [Create an internet-facing endpoint for your server](https://docs.aws.amazon.com/transfer/latest/userguide/create-server-in-vpc.html#create-internet-facing-endpoint) in the AWS Transfer Family documentation to create an SFTP server with an internet-facing endpoint. For **Endpoint type**, choose **VPC hosted**. For **Access**, choose **Internet Facing**. For **VPC**, choose the VPC you created in the previous epic.—or—You can deploy the attached CloudFormation template, `amazon-sftp-server.yml`, in the [CloudFormation console](https://console.aws.amazon.com/cloudformation) to automate the tasks in this epic. | Developer, Systems administrator |
| Migrate the domain name. | Attach the existing domain name to the custom hostname. If you're using a new domain name, use the **Amazon Route 53 DNS **alias. For an existing domain name, choose **Other DNS**. For more information, see [Working with custom hostnames](https://docs.aws.amazon.com/transfer/latest/userguide/requirements-dns.html) in the AWS Transfer Family documentation. | Developer, Systems administrator |
| Add a CloudWatch logging role. | (Optional) if you want to enable CloudWatch logging, create a `Transfer` role with the CloudWatch Logs API operations `logs:CreateLogGroup`, `logs:CreateLogStream`,` logs:DescribeLogStreams`, and `logs:PutLogEvents`. For more information, see[ Log activity with CloudWatch](https://docs.aws.amazon.com/transfer/latest/userguide/monitoring.html#monitoring-enabling) in the AWS Transfer Family documentation. | Developer, system administrator |
| Save and submit. | Choose **Save**. For **Actions**, choose **Start **and wait for the SFTP server to be created with the status **Online**. | Developer, Systems administrator |
### Map Elastic IP addresses to the SFTP server
| Task | Description | Skills required |
| --- | --- | --- |
| Stop the server so you can modify settings. | On the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/), choose **Servers**, and then select the SFTP server you created. For **Actions**, choose **Stop**. When the server is offline, choose **Edit **to modify its settings. | Developer, system administrator |
| Choose Availability Zones and subnets. | In the **Availability Zones** section, choose the Availability Zones and subnets for your VPC. | Developer, Systems administrator |
| Add Elastic IP addresses. | For **IPv4 Addresses**, choose an Elastic IP address for each subnet, and then choose **Save**. | Developer, Systems administrator |
### Add users
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM role for users to access the S3 bucket. | Create a IAM role for `Transfer`** **and add` s3:ListBucket`,` s3:GetBucketLocation`, and `s3:PutObject` with the S3 bucket name as a resource. For more information, see [Create an IAM role and policy](https://docs.aws.amazon.com/transfer/latest/userguide/requirements-roles.html) in the AWS Transfer Family documentation.—or—You can deploy the attached CloudFormation template, `amazon-sftp-customer.yml`, in the [CloudFormation console](https://console.aws.amazon.com/cloudformation) to automate the tasks in this epic. | Developer, Systems administrator |
| Create an S3 bucket. | Create a S3 bucket for the application. | Developer, Systems administrator |
| Create optional folders. | (Optional) If you want to store files for users separately, in specific Amazon S3 folders, add folders as appropriate. | Developer, Systems administrator |
| Create an SSH public key. | To create an SSH key pair, see [Generate SSH keys](https://docs.aws.amazon.com/transfer/latest/userguide/key-management.html#sshkeygen) in the AWS Transfer Family documentation. | Developer, Systems administrator |
| Add users. | On the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/), choose **Servers**, select the SFTP server you created, and then choose **Add user**. For **Home directory**, choose the S3 bucket you created. For **SSH public key**, specify the public key portion of the SSH key pair. Add users for the SFTP server, and then choose **Add**. | Developer, Systems administrator |
### Test the SFTP server
| Task | Description | Skills required |
| --- | --- | --- |
| Update the security group. | In the **Security Groups** section of your SFTP server, add your test machine's IP to gain SFTP access. | Developer |
| Use an SFTP client utility to test the server. | Test file transfers by using any SFTP client utility. For a list of clients and instructions, see [Transferring files using a client](https://docs.aws.amazon.com/transfer/latest/userguide/transfer-file.html) in the AWS Transfer Family documentation. | Developer |
## Related resources
+ [AWS Transfer Family User Guide](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-for-sftp.html)
+ [Amazon S3 User Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)
+ [Elastic IP addresses](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html) in the Amazon EC2 documentation
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/ec0a905c-edef-48ba-9b5e-ea4a4040d320/attachments/attachment.zip)
# Migrate an on-premises VM to Amazon EC2 by using AWS Application Migration Service
*Thanh Nguyen, Amazon Web Services*
## Summary
When it comes to application migration, organizations can take different approaches to rehost (lift and shift) the application’s servers from the on-premises environment to the Amazon Web Services (AWS) Cloud. One way is to provision new Amazon Elastic Compute Cloud (Amazon EC2) instances and then install and configure the application from scratch. Another approach is to use third-party or AWS native migration services to migrate multiple servers at the same time.
This pattern outlines the steps for migrating a supported virtual machine (VM) to an Amazon EC2 instance on the AWS Cloud by using AWS Application Migration Service. You can use the approach in this pattern to migrate one or multiple virtual machines manually, one by one, or automatically by creating appropriate automation scripts based on the outlined steps.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account in one of the AWS Regions that support Application Migration Service
+ Network connectivity between the source server and target EC2 server through a private network by using AWS Direct Connect or a virtual private network (VPN), or through the internet
**Limitations**
+ For the latest list of supported Regions, see the [Supported AWS Regions](https://docs.aws.amazon.com/mgn/latest/ug/supported-regions.html).
+ For a list of supported operating systems, see the [Supported operating systems](https://docs.aws.amazon.com/mgn/latest/ug/Supported-Operating-Systems.html) and the *General *section of [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/).
## Architecture
**Source technology stack**
+ A physical, virtual, or cloud-hosted server running an operating system supported by Amazon EC2
**Target technology stack**
+ An Amazon EC2 instance running the same operating system as the source VM
+ Amazon Elastic Block Store (Amazon EBS)
**Source and target architecture**
The following diagram shows the high-level architecture and main components of the solution. In the on-premises data center, there are virtual machines with local disks. On AWS, there is a staging area with replication servers and a migrated resources area with EC2 instances for test and cutover. Both subnets contain EBS volumes.
![\[Main components to migrate a supported VM to an Amazon EC2 instance on the AWS Cloud.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/58c8bafd-9a6d-42d4-a5ce-08c4b9a286a3/images/f8396fad-7ee9-4f75-800f-e819f509e151.png)
1. Initialize AWS Application Migration Service.
1. Set up the staging area server configuration and reporting, including staging area resources.
1. Install agents on source servers, and use continuous block-level data replication (compressed and encrypted).
1. Automate orchestration and system conversion to shorten the cutover window.
**Network architecture**
The following diagram shows the high-level architecture and main components of the solution from the networking perspective, including required protocols and ports for communication between primary components in the on-premises data center and on AWS.
![\[Networking components including protocols and ports for communication between data center and AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/58c8bafd-9a6d-42d4-a5ce-08c4b9a286a3/images/2f594daa-ddba-4841-8785-6067e8d83c2f.png)
## Tools
+ [AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) helps you rehost (*lift and shift*) applications to the AWS Cloud without change and with minimal downtime.
## Best practices
+ Do not take the source server offline or perform a reboot until the cutover to the target EC2 instance is complete.
+ Provide ample opportunity for the users to perform user acceptance testing (UAT) on the target server to identify and resolve any issues. Ideally, this testing should start at least two weeks before cutover.
+ Frequently monitor the server replication status on the Application Migration Service console to identify issues early on.
+ Use temporary AWS Identity and Access Management (IAM) credentials for agent installation instead of permanent IAM user credentials.
## Epics
### Generate AWS credentials
| Task | Description | Skills required |
| --- | --- | --- |
| Create the AWS Replication Agent IAM role. | Sign in with administrative permissions to the AWS account.On the AWS Identity and Access Management (IAM) [console](https://console.aws.amazon.com/iam/), create an IAM role:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-vm-to-amazon-ec2-by-using-aws-application-migration-service.html) | AWS administrator, Migration engineer |
| Generate temporary security credentials. | On a machine with AWS Command Line Interface (AWS CLI) installed, sign in with administrative permissions. Or alternatively (within a supported AWS Region), on the AWS Management Console, sign in with administrative permissions to the AWS account, and open AWS CloudShell.Generate temporary credentials with the following command, replacing `` with the AWS account ID.`aws sts assume-role --role-arn arn:aws:iam:::role/MGN_Agent_Installation_Role --role-session-name mgn_installation_session_role`From the output of the command, copy the values for `AccessKeyId`,** **`SecretAccessKey`, and** **`SessionToken`.** **Store them in a safe location for later use.These temporary credentials will expire after one hour. If you need credentials after one hour, repeat the previous steps. | AWS administrator, Migration engineer |
### Initialize Application Migration Service and create the Replication Settings template
| Task | Description | Skills required |
| --- | --- | --- |
| Initialize the service. | On the console, sign in with administrative permissions to the AWS account.Choose **Application Migration Service**, and then choose **Get started**. | AWS administrator, Migration engineer |
| Create and configure the Replication Settings template. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-vm-to-amazon-ec2-by-using-aws-application-migration-service.html)Application Migration Service will automatically create all the IAM roles required to facilitate data replication and the launching of migrated servers. | AWS administrator, Migration engineer |
### Install AWS Replication Agents on source machines
| Task | Description | Skills required |
| --- | --- | --- |
| Have the required AWS credentials ready. | When you run the installer file on a source server, you will need to enter the temporary credentials that you generated earlier, including `AccessKeyId`, `SecretAccessKey`, and `SessionToken`. | Migration engineer, AWS administrator |
| For Linux servers, install the agent. | Copy the installer command, log in to your source servers, and run the installer. For detailed instructions, see the [AWS documentation](https://docs.aws.amazon.com/mgn/latest/ug/linux-agent.html). | AWS administrator, Migration engineer |
| For Windows servers, install the agent. | Download the installer file to each server, and then run the installer command. For detailed instructions, see the [AWS documentation](https://docs.aws.amazon.com/mgn/latest/ug/windows-agent.html). | AWS administrator, Migration engineer |
| Wait for initial data replication to be completed. | When the agent has been installed, the source server will appear on the Application Migration Service console, in the **Source servers** section. Wait while the server undergoes initial data replication. | AWS administrator, Migration engineer |
### Configure launch settings
| Task | Description | Skills required |
| --- | --- | --- |
| Specify the server details. | On the Application Migration Service console, choose the **Source servers** section, and then choose a server name from the list to access the server details. | AWS administrator, Migration engineer |
| Configure the launch settings. | Choose the **Launch settings** tab. You can configure a variety of settings, including general launch settings and EC2 launch template settings. For detailed instructions, see the [AWS documentation](https://docs.aws.amazon.com/mgn/latest/ug/launch-settings.html). | AWS administrator, Migration engineer |
### Perform a test
| Task | Description | Skills required |
| --- | --- | --- |
| Test the source servers. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-vm-to-amazon-ec2-by-using-aws-application-migration-service.html)The servers will be launched. | AWS administrator, Migration engineer |
| Verify that the test completed successfully. | After the test server is completely launched, the **Alerts** status on the page will show **Launched** for each server. | AWS administrator, Migration engineer |
| Test the server. | Perform testing against the test server to ensure that it functions as expected. | AWS administrator, Migration engineer |
### Schedule and perform a cutover
| Task | Description | Skills required |
| --- | --- | --- |
| Schedule a cutover window. | Schedule an appropriate cutover timeframe with relevant teams. | AWS administrator, Migration engineer |
| Perform the cutover. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-vm-to-amazon-ec2-by-using-aws-application-migration-service.html)The source server's **Migration lifecycle** will change to **Cutover in progress**. | AWS administrator, Migration engineer |
| Verify that the cutover completed successfully. | After the cutover servers are completely launched, the **Alerts** status on the **Source Servers** page will show **Launched** for each server. | AWS administrator, Migration engineer |
| Test the server. | Perform testing against the cutover server to ensure that it functions as expected. | AWS administrator, Migration engineer |
| Finalize the cutover. | Choose **Test and Cutover**, and then select **Finalize cutover** to finalize the migration process. | AWS administrator, Migration engineer |
## Related resources
+ [AWS Application Migration Service](https://aws.amazon.com/application-migration-service/)
+ [AWS Application Migration Service User Guide](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html)
# Migrate small sets of data from on premises to Amazon S3 using AWS SFTP
*Charles Gibson and Sergiy Shevchenko, Amazon Web Services*
## Summary
This pattern describes how to migrate small sets of data (5 TB or less) from on-premises data centers to Amazon Simple Storage Service (Amazon S3) by using AWS Transfer for SFTP (AWS SFTP). The data can be either database dumps or flat files.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An AWS Direct Connect link established between your data center and AWS
**Limitations**
+ The data files must be less than 5 TB. For files over 5 TB, you can perform a multipart upload to Amazon S3 or choose another data transfer method.
## Architecture
**Source technology stack**
+ On-premises flat files or database dumps
**Target technology stack**
+ Amazon S3
**Source and target architecture**
![\[Diagram showing data flow from on-premises servers to AWS Cloud services via Direct Connect and VPN.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a9c016ff-3e68-4714-ac51-46cb4727397a/images/5c5bb9ea-d552-44e8-8d0d-df341f84f55d.png)
## Tools
+ [AWS SFTP](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-for-sftp.html) – Enables the transfer of files directly into and out of Amazon S3 using Secure File Transfer Protocol (SFTP).
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) – Establishes a dedicated network connection from your on-premises data centers to AWS.
+ [VPC endpoints](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html) – Enable you to privately connect a VPC to supported AWS services and VPC endpoint services powered by AWS PrivateLink without an internet gateway, network address translation (NAT) device, VPN connection, or Direct Connect connection. Instances in a VPC don't require public IP addresses to communicate with resources in the service.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Document the current SFTP requirements. | | Application owner, SA |
| Identify the authentication requirements. | Requirements may include key-based authentication, user name or password, or identity provider (IdP). | Application owner, SA |
| Identify the application integration requirements. | | Application owner |
| Identify the users who require the service. | | Application owner |
| Determine the DNS name for the SFTP server endpoint. | | Networking |
| Determine the backup strategy. | | SA, DBA (if data is transferred) |
| Identify the application migration or cutover strategy. | | Application owner, SA, DBA |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create one or more virtual private clouds (VPCs) and subnets in your AWS account. | | Application owner, AMS |
| Create the security groups and network access control list (ACL). | | Security, Networking, AMS |
| Create the Amazon S3 bucket. | | Application owner, AMS |
| Create the AWS Identity and Access Management (IAM) role. | Create an IAM policy that includes the permissions to enable AWS SFTP to access your Amazon S3 bucket. This IAM policy determines what level of access you provide SFTP users. Create another IAM policy to establish a trust relationship with AWS SFTP. | Security, AMS |
| Associate a registered domain (optional). | If you have your own registered domain, you can associate it with the SFTP server. You can route SFTP traffic to your SFTP server endpoint from a domain or from a subdomain. | Networking, AMS |
| Create an SFTP server. | Specify the identity provider type used by the service to authenticate your users. | Application owner, AMS |
| Open an SFTP client. | Open an SFTP client and configure the connection to use the SFTP endpoint host. AWS SFTP supports any standard SFTP client. Commonly used SFTP clients include OpenSSH, WinSCP, Cyberduck, and FileZilla. You can get the SFTP server host name from the AWS SFTP console. | Application owner, AMS |
### Plan and test
| Task | Description | Skills required |
| --- | --- | --- |
| Plan the application migration. | Plan for any application configuration changes required, set the migration date, and determine the test schedule. | Application owner, AMS |
| Test the infrastructure. | Test in a non-production environment. | Application owner, AMS |
## Related resources
**References**
+ [AWS Transfer for SFTP User Guide](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-for-sftp.html)
+ [AWS Direct Connect resources](https://aws.amazon.com/directconnect/resources/)
+ [VPC Endpoints](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html)
**Tutorials and videos**
+ [AWS Transfer for SFTP (video)](https://www.youtube.com/watch?v=wcnGez5PP1E)
+ [AWS Transfer for SFTP user guide](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-for-sftp.html)
+ [AWS SA Whiteboarding - Direct Connect (video) ](https://www.youtube.com/watch?v=uP68iqyuqTg)
# Migrate an on-premises Oracle database to Oracle on Amazon EC2
*Baji Shaik and Pankaj Choudhary, Amazon Web Services*
## Summary
This pattern walks you through the steps for migrating an on-premises Oracle database to Oracle on an Amazon Elastic Compute Cloud (Amazon EC2) instance. It describes two options for migration: using AWS Data Migration Service (AWS DMS) or using native Oracle tools such as RMAN, Data Pump import/export, transportable tablespaces, and Oracle GoldenGate.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Oracle database in an on-premises data center
**Limitations**
+ The target operating system (OS) must be supported by Amazon EC2. For a complete list of supported systems, see [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/).
**Product versions**
+ Oracle versions 10.2 and later (for versions 10.x), 11g and up to 12.2, and 18c for the Enterprise, Standard, Standard One, and Standard Two editions. For the latest list of versions supported by AWS DMS, see "On-premises and Amazon EC2 instance databases" in [Sources for Data Migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html) in the AWS DMS documentation.
## Architecture
**Source technology stack**
+ An on-premises Oracle database
**Target technology stack**
+ An Oracle database instance on Amazon EC2
**Target architecture**
![\[Setting up replication for an Oracle database on Amaozn EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66c98694-6580-4ffb-9f16-84de58cf8b07/images/386d5b14-8633-4ecc-98fb-59872de99d41.png)
**Data migration architecture**
*Using AWS DMS:*
![\[Migrating an on-premises Oracle database to Amazon EC2 with AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66c98694-6580-4ffb-9f16-84de58cf8b07/images/14954066-d22b-486a-a432-265296752878.png)
*Using native Oracle tools:*
![\[Migrating an on-premises Oracle database to Amazon EC2 with Oracle tools.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66c98694-6580-4ffb-9f16-84de58cf8b07/images/82ba5fcb-8640-45fa-b432-2702dedc0774.png)
## Tools
+ **AWS DMS - **[AWS Database Migration Services](https://docs.aws.amazon.com/dms/index.html) (AWS DMS) supports several types of source and target databases. For information about the database versions and editions that are supported, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support.
+ **Native Oracle tools - **RMAN, Data Pump import/export, transportable tablespaces, Oracle GoldenGate
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the versions of the source and target databases. | | DBA |
| Identify the version of the target OS. | | DBA, SysAdmin |
| Identify hardware requirements for the target server instance based on the Oracle compatibility list and capacity requirements. | | DBA, SysAdmin |
| Identify storage requirements (storage type and capacity). | | DBA, SysAdmin |
| Identify network requirements (latency and bandwidth). | | DBA, SysAdmin |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, SysAdmin |
| Identify network/host access security requirements for source and target databases. | | DBA, SysAdmin |
| Identify a list of OS users required for Oracle software installation. | | DBA, SysAdmin |
| Download AWS Schema Conversion Tool (AWS SCT) and drivers. | | DBA |
| Create an AWS SCT project for the workload, and connect to the source database. | | DBA |
| Generate SQL files for the creation of objects (tables, indexes, sequences, etc.). | | DBA |
| Determine a backup strategy. | | DBA, SysAdmin |
| Determine availability requirements. | | DBA |
| Identify the application migration/switch-over strategy. | | DBA, SysAdmin, App owner |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC) and subnets in your AWS account. | | SysAdmin |
| Create security groups and network access control lists (ACLs). | | SysAdmin |
| Configure and start the EC2 instance. | | SysAdmin |
### Install the Oracle software
| Task | Description | Skills required |
| --- | --- | --- |
| Create the OS users and groups required for the Oracle software. | | DBA, SysAdmin |
| Download the required version of Oracle software. | | |
| Install the Oracle software on the EC2 instance. | | DBA, SysAdmin |
| Create objects like tables, primary keys, views, and sequences by using the scripts generated by AWS SCT. | | DBA |
### Migrate data - option 1
| Task | Description | Skills required |
| --- | --- | --- |
| Use native Oracle tools or third-party tools to migrate database objects and data. | Oracle tools include Data Pump import/export, RMAN, transportable tablespaces, and GoldenGate. | DBA |
### Migrate data - option 2
| Task | Description | Skills required |
| --- | --- | --- |
| Determine the migration method. | | DBA |
| Create a replication instance in the AWS DMS console. | | DBA |
| Create source and target endpoints. | | DBA |
| Create a replication task. | | DBA |
| Enable change data capture (CDC) to capture changes for a continuous replication. | | DBA |
| Run the replication task and monitor logs. | | DBA |
| Create secondary objects like indexes and foreign keys when the full load is done. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | | DBA, SysAdmin, App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application cutover/switch-over strategy. | | DBA, SysAdmin, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary AWS Secrets Manager resources. | | DBA, SysAdmin |
| Review and validate the project documents. | | DBA, SysAdmin, App owner |
| Gather metrics around time to migrate, % of manual vs. tool, cost savings, etc. | | DBA, SysAdmin, App owner |
| Close out the project and provide feedback. | | |
## Related resources
**References**
+ [Strategies for Migrating Oracle Databases to AWS](https://docs.aws.amazon.com/whitepapers/latest/strategies-migrating-oracle-db-to-aws/strategies-migrating-oracle-db-to-aws.html)
+ [Migrating Oracle databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/)
+ [Amazon EC2 website](https://aws.amazon.com/ec2/)
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [AWS DMS blog posts](https://aws.amazon.com/blogs/database/category/dms/)
+ [Amazon EC2 Pricing](https://aws.amazon.com/ec2/pricing/)
+ [Licensing Oracle Software in the Cloud Computing Environment](http://www.oracle.com/us/corporate/pricing/cloud-licensing-070579.pdf)
**Tutorials and videos**
+ [Getting Started with Amazon EC2](https://aws.amazon.com/ec2/getting-started/)
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Introduction to Amazon EC2 - Elastic Cloud Server & Hosting with AWS (video)](https://www.youtube.com/watch?v=TsRBftzZsQo)
# Migrate an on-premises Oracle database to Amazon EC2 by using Oracle Data Pump
*Navakanth Talluri, Amazon Web Services*
## Summary
When migrating databases, you must consider factors such as the source and target database engines and versions, migration tools and services, and acceptable downtime periods. If you’re migrating an on-premises Oracle database to Amazon Elastic Compute Cloud (Amazon EC2), you can use Oracle tools, such as Oracle Data Pump and Oracle Recovery Manager (RMAN). For more information about strategies, see [Migrating Oracle databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html).
Oracle Data Pump helps you extract the logical, consistent backup of the database and restore it to the target EC2 instance. This pattern describes how to migrate an on-premises Oracle database to an EC2 instance by using Oracle Data Pump and the `NETWORK_LINK` parameter, with minimal downtime. The `NETWORK_LINK` parameter starts an import through a database link. The Oracle Data Pump Import (impdp) client on the target EC2 instance connects to the source database, retrieves data from it, and writes the data directly to the database on the target instance. There are no backup, or *dump*, files used in this solution.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An on-premises Oracle database that:
+ Isn’t an Oracle Real Application Clusters (RAC) database
+ Isn’t an Oracle Automatic Storage Management (Oracle ASM) database
+ Is in read-write mode.
+ You have created an AWS Direct Connect link between your on-premises data center and AWS. For more information, see [Create a connection](https://docs.aws.amazon.com/directconnect/latest/UserGuide/create-connection.html) (Direct Connect documentation).
**Product versions**
+ Oracle Database 10g release 1 (10.1) and later
## Architecture
**Source technology stack**
+ A standalone (non-RAC and non-ASM) Oracle database server in an on-premises data center
**Target technology stack**
+ An Oracle database running on Amazon EC2
**Target architecture**
The [reliability pillar](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html) of the AWS Well-Architected Framework recommends creating data backups to help provide high availability and resiliency. For more information, see [Architecting for high availability](https://docs.aws.amazon.com/whitepapers/latest/oracle-database-aws-best-practices/architecting-for-high-availability.html#amazon-ec2) in *Best Practices for Running Oracle Database on AWS*. This pattern sets up primary and standby databases on EC2 instances by using Oracle Active Data Guard. For high availability, the EC2 instances should be in different Availability Zones. However, the Availability Zones can be in the same AWS Region or in different AWS Regions.
Active Data Guard provides read-only access to a physical standby database and applies redo changes continuously from the primary database. Based on your recovery point objective (RPO) and recovery time objective (RTO), you can choose between synchronous and asynchronous redo transport options.
The following image shows the target architecture if the primary and standby EC2 instances are in different AWS Regions.
![\[Application connecting to the new database on the primary EC2 instance\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bdd49395-2f99-43e2-ad1d-a1d09d90fb58/images/37fcd4dc-5516-416b-a280-0c5f002880de.png)
**Data migration architecture**
After you have finished setting up the target architecture, you use Oracle Data Pump to migrate the on-premises data and schemas to the primary EC2 instance. During cutover, applications can’t access the on-premises database or the target database. You shut down these applications until they can be connected to the new target database on the primary EC2 instance.
The following image shows the architecture during the data migration. In this sample architecture, the primary and standby EC2 instances are in different AWS Regions.
![\[The source DB connects to the target DB. Applications are disconnected from source and target DBs\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bdd49395-2f99-43e2-ad1d-a1d09d90fb58/images/c58b669b-b11f-4d78-8911-c07b81b7c6a0.png)
## Tools
**AWS services**
+ [AWS Direct Connect](https://aws.amazon.com/directconnect/) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
**Other tools and services**
+ [Oracle Active Data Guard](https://docs.oracle.com/en/database/oracle/oracle-database/21/sbydb/introduction-to-oracle-data-guard-concepts.html#GUID-5E73667D-4A56-445E-911F-1E99092DD8D7) helps you create, maintain, manage, and monitor standby databases.
+ [Oracle Data Pump](https://www.oracle.com/technetwork/documentation/data-pump-overview-084963.html) helps you move data and metadata from one database to another at high speeds.
## Best practices
+ [Best Practices for Running Oracle Database on AWS](https://docs.aws.amazon.com/whitepapers/latest/oracle-database-aws-best-practices/architecting-for-security-and-performance.html)
+ [Importing data using NETWORK\$1LINK](https://docs.oracle.com/database/121/SUTIL/GUID-23E58D59-A477-4A87-BD0E-C82447581D0A.htm#SUTIL856)
## Epics
### Set up the EC2 instances on AWS
| Task | Description | Skills required |
| --- | --- | --- |
| Identify the source hardware configuration for the on-premises host and the kernel parameters. | Validate the on-premises configuration, including storage size, input/output operations per second (IOPS), and CPU. This is important for Oracle licensing, which is based on CPU cores. | DBA, SysAdmin |
| Create the infrastructure on AWS. | Create the virtual private clouds (VPCs), private subnets, security groups, network access control lists (ACLs), route tables, and internet gateway. For more information, see the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.html) | DBA, AWS systems administrator |
| Set up the EC2 instances by using Active Data Guard. | Configure AWS EC2 instances by using an Active Data Guard configuration, as described in the [AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html). The version of Oracle Database on the EC2 instance can be different from the on-premises version because this pattern uses logical backups. Note the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.html)For more information, see:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.html) | DBA, AWS systems administrator |
### Migrate the database to Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Create a dblink to the on-premises database from the EC2 instance. | Create a database link (dblink) between the Oracle database on the EC2 instance and the on-premises Oracle database. For more information, see [Using Network Link Import to Move Data](https://docs.oracle.com/database/121/SUTIL/GUID-3E1D4B46-E856-4ABE-ACC5-977A898BB0F1.htm#SUTIL806) (Oracle documentation). | DBA |
| Verify the connection between the EC2 instance and the on-premises host. | Use the dblink to confirm that the connection between the EC2 instance and the on-premises database is functioning. For instructions, see [CREATE DATABASE LINK](https://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_5005.htm) (Oracle documentation). | DBA |
| Stop all applications connected to the on-premises database. | After the database downtime is approved, shut down any applications and dependent jobs that connent to your on-premises database. You can do this either from the application directly or from the database by using cron. For more information, see [Use the Crontab Utility to Schedule Tasks on Oracle Linux](https://docs.oracle.com/en/learn/oracle-linux-crontab/index.html). | DBA, App developer |
| Schedule the data migration job. | On the target host, use the command `impdb` to schedule the Data Pump import. This connects the target database to the on-premises host and starts the data migration. For more information, see [Data Pump Import](https://docs.oracle.com/database/121/SUTIL/GUID-D11E340E-14C6-43B8-AB09-6335F0C1F71B.htm#SUTIL300) and [NETWORK\$1LINK](https://docs.oracle.com/database/121/SUTIL/GUID-0871E56B-07EB-43B3-91DA-D1F457CF6182.htm#SUTIL919) (Oracle documentation). | DBA |
| Validate the data migration. | Data validation is a crucial step. For data validation, you can use custom tools or Oracle tools, such as a combination of dblink and SQL queries. | DBA |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Put the source database in read-only mode. | Confirm that the application is shut down and no changes are being made to the source database. Open the source database in read-only mode. This helps you avoid any open transactions. For more information, see `ALTER DATABASE` in [SQL Statements](https://docs.oracle.com/database/121/SQLRF/statements_1006.htm#i2135540) (Oracle documentation). | DBA, DevOps engineer, App developer |
| Validate the object count and data. | To validate the data and object, use custom tools or Oracle tools, such as a combination of dblink and SQL queries. | DBA, App developer |
| Connect the applications to the database on the primary EC2 instance. | Change the application’s connection attribute to point to the new database you created on the primary EC2 instance. | DBA, App developer |
| Validate the application performance. | Start the application. Validate the functionality and performance of the application by using [Automated Workload Repository](https://docs.oracle.com/database/121/RACAD/GUID-C3CD2DCE-38BD-46BA-BC32-7A28CAC9A7FD.htm#RACAD951) (Oracle documentation). | App developer, DevOps engineer, DBA |
## Related resources
**AWS references**
+ [Migrating Oracle databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html)
+ [Amazon EC2 for Oracle](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/ec2-oracle.html)
+ [Migrating bulky Oracle databases to AWS for cross-platform environments](https://docs.aws.amazon.com/prescriptive-guidance/latest/migrate-bulky-oracle-databases/welcome.html)
+ [VPCs and subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html)
+ [Tutorial: Create a VPC for use with a database instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Tutorials.WebServerDB.CreateVPC.html)
**Oracle references**
+ [Oracle Data Guard Configurations](https://docs.oracle.com/en/database/oracle/oracle-database/21/sbydb/introduction-to-oracle-data-guard-concepts.html#GUID-AB9DF863-2C7E-4767-81F2-56AD0FA30B49)
+ [Data Pump Import](https://docs.oracle.com/database/121/SUTIL/GUID-D11E340E-14C6-43B8-AB09-6335F0C1F71B.htm#SUTIL300)
# Migrate RHEL BYOL systems to AWS License-Included instances by using AWS MGN
*Mike Kuznetsov, Amazon Web Services*
## Summary
When you migrate your workloads to AWS by using AWS Application Migration Service (AWS MGN), you might have to lift and shift (rehost) your Red Hat Enterprise Linux (RHEL) instances and change the license from the default Bring Your Own License (BYOL) model to an AWS License Included (LI) model during migration. AWS MGN supports a scalable approach that uses Amazon Machine Image (AMI) IDs. This pattern describes how to accomplish the license change on RHEL servers during the rehost migration at scale. It also explains how to change the license for a RHEL system that’s already running on Amazon Elastic Compute Cloud (Amazon EC2).
## Prerequisites and limitations
**Prerequisites **
+ Access to the target AWS account
+ AWS MGN initialized in the target AWS account and Region for the migration (not required if you have already migrated from your on-premises system to AWS)
+ A source RHEL server with a valid RHEL license
## Architecture
This pattern covers two scenarios:
+ Migrating a system from on premises directly into an AWS LI instance by using AWS MGN. For this scenario, follow the instructions in the first epic (*Migrate to LI instance - option 1*) and third epic.
+ Changing the licensing model from BYOL to LI for a previously migrated RHEL system that’s already running on Amazon EC2. For this scenario, follow the instructions in the second epic (*Migrate to LI instance* - *option 2*) and third epic.
**Note**
The third epic involves reconfiguring the new RHEL instance to use the Red Hat Update Infrastructure (RHUI) servers provided by AWS. This process is the same for both scenarios.
## Tools
**AWS services**
+ [AWS Application Migration Service (AWS MGN)](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) helps you rehost (lift and shift) applications to the AWS Cloud without change and with minimal downtime.
## Epics
### Migrate to LI instance - option 1 (for an on-premises RHEL system)
| Task | Description | Skills required |
| --- | --- | --- |
| Find the AMI ID of the RHEL AWS LI instance in the target Region. | Visit [AWS Marketplace](https://aws.amazon.com/marketplace) or use the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) to find the RHEL AMI ID that matches the version of the RHEL source system (for example, RHEL-7.7), and write down the AMI ID. On the Amazon EC2 console, you can filter the AMIs by using one of the following search terms:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html) | Cloud administrator |
| Configure AWS MGN launch settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html)AWS MGN will now use this version of the launch template to launch test or cutover instances. For more information, see the [AWS MGN documentation](https://docs.aws.amazon.com/mgn/latest/ug/ec2-launch.html). | Cloud administrator |
| Validate settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html) | Cloud administrator |
| Launch the new LI instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html) | Cloud administrator |
### Migrate to LI instance - option 2 (for a RHEL BYOL EC2 instance)
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate your RHEL BYOL EC2 instance to an AWS LI instance. | You can switch RHEL systems that you previously migrated to AWS as BYOL to AWS LI instances by moving their disks (Amazon Elastic Block Store volumes) and attaching them to a new LI instance. To make this switch, follow these steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html) | Cloud administrator |
### Reconfigure RHEL OS to use AWS-provided RHUI – both options
| Task | Description | Skills required |
| --- | --- | --- |
| Deregister the OS from the Red Hat subscription and license. | After migration and successful cutover, the RHEL system has to be removed from the Red Hat subscription to stop consuming the Red Hat license and avoid double billing.To remove RHEL OS from the Red Hat subscription, follow the process described in the [Red Hat Subscription Management (RHSM) documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/installation_guide/chap-subscription-management-unregistering). Use the CLI command:
subscription-manager unregister
You can also disable the subscription manager plugin to stop checking the status of the subscription on every **yum** call. To do this, edit the configuration file `/etc/yum/pluginconf.d/subscription-manager.conf` and change the parameter `enabled=1` to `enabled=0`. | Linux or system administrator |
| Replace the old update configuration (RHUI, Red Hat Satellite network, yum repositories) with the AWS-provided RHUI. | You must reconfigure the migrated RHEL system to use the AWS-provided RHUI servers. This gives you access to the RHUI servers within AWS Regions without requiring the external update infrastructure. The change involves the following process:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html)Here are the detailed steps and commands:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-rhel-byol-systems-to-aws-license-included-instances-by-using-aws-mgn.html) | Linux or system administrator |
| Validate the configuration. | On the target migrated instance, verify that the new configuration is correct:
sudo yum clean all sudo yum repolist
| Linux or system administrator |
## Related resources
+ [AWS Application Migration Service (AWS MGN) User Guide](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html)
+ [Get an AWS RHUI client package supporting IMDSv2](https://access.redhat.com/solutions/5009491) (Red Hat Knowledgebase article)
+ [Amazon EC2 launch templates](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html) (Amazon EC2 documentation)
# Migrate an on-premises Microsoft SQL Server database to Amazon EC2
*Senthil Ramasamy, Amazon Web Services*
## Summary
This pattern describes how to migrate an on-premises Microsoft SQL Server database to Microsoft SQL Server on an Amazon Elastic Compute Cloud (Amazon EC2) instance. It covers two options for migration: using AWS Database Migration Service (AWS DMS) or using native Microsoft SQL Server tools such as backup and restore, Copy Database Wizard, or copy and attach database.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An operating system supported by Amazon EC2 (for a complete list of supported operating system versions, see [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/))
+ A Microsoft SQL Server source database in an on-premises data center
**Product versions**
+ For on-premises and Amazon EC2 instance databases, AWS DMS supports:
+ SQL Server versions 2005, 2008, 2008R2, 2012, 2014, 2016, 2017, and 2019
+ Enterprise, Standard, Workgroup, Developer, and Web editions
+ For the latest list of supported versions, see [Using a Microsoft SQL Server Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.SQLServer.html).
## Architecture
**Source technology stack**
+ On-premises Microsoft SQL Server database
**Target technology stack**
+ Microsoft SQL Server database on an EC2 instance
**Target architecture**
![\[Primary and standby Microsoft SQL Server instances on EC2 instances in two Availability Zones.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f0a155b3-4977-4e1f-8332-89eab29c1e25/images/53e2c27d-ceb4-4d88-a022-93dd0b343eaf.png)
**Data migration architecture**
+ Using AWS DMS
![\[Migrating on-premises SQL Server data to an EC2 instance by using AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f0a155b3-4977-4e1f-8332-89eab29c1e25/images/1cbe32ea-e285-4cac-9153-4428bad9b229.png)
+ Using native SQL Server tools
![\[Migrating on-premises SQL Server data to an EC2 instance by using native SQL Server tools.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f0a155b3-4977-4e1f-8332-89eab29c1e25/images/ad2caf54-7399-4038-91a3-acba9fa7da29.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/) helps you migrate your data to and from widely used commercial and open-source databases, including Oracle, SQL Server, MySQL, and PostgreSQL. You can use AWS DMS to migrate your data into the AWS Cloud, between on-premises instances (through an AWS Cloud setup), or between combinations of cloud and on-premises setups.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
+ Native Microsoft SQL Server tools include backup and restore, Copy Database Wizard, and copy and attach database.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions. | | DBA |
| Identify the target operating system version. | | DBA, Systems administrator |
| Identify the hardware requirements for the target server instance based on the Microsoft SQL Server compatibility list and capacity requirements. | | DBA, Systems administrator |
| Identify the storage requirements for type and capacity. | | DBA, Systems administrator |
| Identify the network requirements, including latency and bandwidth. | | DBA, Systems administrator |
| Choose the EC2 instance type based on capacity, storage features, and network features. | | DBA, Systems administrator |
| Identify the network and host access security requirements for the source and target databases. | | DBA, Systems administrator |
| Identify a list of users required for the Microsoft SQL Server software installation. | | DBA, Systems administrator |
| Determine the backup strategy. | | DBA |
| Determine the availability requirements. | | DBA |
| Identify the application migration and cutover strategy. | | DBA, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC) and subnets. | | Systems administrator |
| Create security groups and network access control list (ACL). | | Systems administrator |
| Configure and start an EC2 instance. | | Systems administrator |
### Install the software
| Task | Description | Skills required |
| --- | --- | --- |
| Create the users and groups required for Microsoft SQL Server software. | | DBA, Systems administrator |
| Download the Microsoft SQL Server software. | | DBA, Systems administrator |
| Install Microsoft SQL Server software on the EC2 instance and configure the server. | | DBA, Systems administrator |
### Migrate the data - option 1
| Task | Description | Skills required |
| --- | --- | --- |
| Use native Microsoft SQL Server tools or third-party tools to migrate the database objects and data. | Tools include backup and restore, Copy Database Wizard, and copy and attach database. For more information, see the guide [Migrating Microsoft SQL Server databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-sql-server/). | DBA |
### Migrate the data - option 2
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the data by using AWS DMS. | For more information about using AWS DMS, see the links in the [Related resources](#migrate-an-on-premises-microsoft-sql-server-database-to-amazon-ec2-resources) section. | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | Use AWS Schema Conversion Tool (AWS SCT) to analyze and modify SQL code that’s embedded in application source code. | DBA, App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application switch-over strategy. | | DBA, App owner, Systems administrator |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down all temporary AWS resources. | Temporary resources include the AWS DMS replication instance and the EC2 instance for AWS SCT. | DBA, Systems administrator |
| Review and validate the project documents. | | DBA, App owner, Systems administrator |
| Gather metrics around time to migrate, percent of manual versus tool cost savings, and so on. | | DBA, App owner, Systems administrator |
| Close the project and provide feedback. | | DBA, App owner, Systems administrator |
## Related resources
**References**
+ [Migrating Microsoft SQL Server databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-sql-server/)
+ [Amazon EC2](https://aws.amazon.com/ec2/)
+ [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/)
+ [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/)
+ [AWS Database Migration Service](https://aws.amazon.com/dms/)
+ [Microsoft Products on AWS](https://aws.amazon.com/windows/products/)
+ [Microsoft Licensing on AWS](https://aws.amazon.com/windows/resources/licensing/)
+ [Microsoft SQL Server on AWS](https://aws.amazon.com/windows/products/sql/)
**Tutorials and videos**
+ [Getting Started with ](https://aws.amazon.com/ec2/getting-started/)Amazon EC2
+ [Getting Started with ](https://aws.amazon.com/dms/getting-started/)AWS Database Migration Service
+ [Join an Amazon EC2 instance to your Simple AD Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/simple_ad_join_instance.html)
+ [Join an Amazon EC2 instance to your AWS Managed Microsoft AD Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_join_instance.html)
+ [AWS Database Migration Service ](https://www.youtube.com/watch?v=zb4GcjEdl8U)(video)
+ [Introduction to Amazon EC2 – Elastic Cloud Server & Hosting with AWS](https://www.youtube.com/watch?v=TsRBftzZsQo) (video)
# Rehost on-premises workloads in the AWS Cloud: migration checklist
*Srikanth Rangavajhala, Amazon Web Services*
## Summary
Rehosting on-premises workloads in the Amazon Web Services (AWS) Cloud involves the following migration phases: planning, pre-discovery, discovery, build, test, and cutover. This pattern outlines the phases and their related tasks. The tasks are described at a high level and support about 75% of all application workloads. You can implement these tasks over two to three weeks in an agile sprint cycle.
You should review and vet these tasks with your migration team and consultants. After the review, you can gather the input, eliminate or re-evaluate tasks as necessary to meet your requirements, and modify other tasks to support at least 75% of the application workloads in your portfolio. You can then use an agile project management tool such as Atlassian Jira or Rally Software to import the tasks, assign them to resources, and track your migration activities.
The pattern assumes that you're using [AWS Cloud Migration Factory](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/solution-overview.html) to rehost your workloads, but you can use your migration tool of choice.
Amazon Macie can help identify sensitive data in your knowledge bases, stored as data sources, model invocation logs, and prompt stores in Amazon Simple Storage Service (Amazon S3) buckets. For more information, see the [Macie documentation](https://docs.aws.amazon.com/macie/latest/user/data-classification.html).
## Prerequisites and limitations
**Prerequisites**
+ Project management tool for tracking migration tasks (for example, Atlassian Jira or Rally Software)
+ Migration tool for rehosting your workloads on AWS (for example, [Cloud Migration Factory](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/solution-overview.html))
## Architecture
**Source platform **
+ On-premises source stack (including technologies, applications, databases, and infrastructure)
**Target platform**
+ AWS Cloud target stack (including technologies, applications, databases, and infrastructure)
**Architecture**
The following diagram illustrates rehosting (discovering and migrating servers from an on-premises source environment to AWS) by using Cloud Migration Factory and AWS Application Migration Service.
![\[Rehosting servers on AWS by using Cloud Migration Factory and Application Migration Service\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8e2d2d72-30cc-4e98-8abd-ac2ef95e599b/images/735ad65b-2646-4803-82c9-f7f93369b3a5.png)
## Tools
+ You can use a migration and project management tool of your choice.
## Epics
### Planning phase
| Task | Description | Skills required |
| --- | --- | --- |
| Groom the pre-discovery backlog. | Conduct the pre-discovery backlog grooming working session with department leads and application owners. | Project manager, Agile scrum leader |
| Conduct the sprint planning working session. | As a scoping exercise, distribute the applications that you want to migrate across sprints and waves. | Project manager, Agile scrum leader |
### Pre-discovery phase
| Task | Description | Skills required |
| --- | --- | --- |
| Confirm application knowledge. | Confirm and document the application owner and their knowledge of the application. Determine whether there's another point person for technical questions. | Migration specialist (interviewer) |
| Determine application compliance requirements. | Confirm with the application owner that the application doesn't have to comply with requirements for Payment Card Industry Data Security Standard (PCI DSS), Sarbanes-Oxley Act (SOX), personally identifiable information (PII), or other standards. If compliance requirements exist, teams must finish their compliance checks on the servers that will be migrated. | Migration specialist (interviewer) |
| Confirm production release requirements. | Confirm the requirements for releasing the migrated application to production (such as release date and downtime duration) with the application owner or technical contact. | Migration specialist (interviewer) |
| Get server list. | Get the list of servers that are associated with the targeted application. | Migration specialist (interviewer) |
| Get the logical diagram that shows the current state. | Obtain the current state diagram for the application from the enterprise architect or the application owner. | Migration specialist (interviewer) |
| Create a logical diagram that shows the target state. | Create a logical diagram of the application that shows the target architecture on AWS. This diagram should illustrate servers, connectivity, and mapping factors. | Enterprise architect, Business owner |
| Get server information. | Collect information about the servers that are associated with the application, including their configuration details. | Migration specialist (interviewer) |
| Add server information to the discovery template. | Add detailed server information to the application discovery template (see `mobilize-application-questionnaire.xlsx` in the attachment for this pattern). This template includes all the application-related security, infrastructure, operating system, and networking details. | Migration specialist (interviewer) |
| Publish the application discovery template. | Share the application discovery template with the application owner and migration team for common access and use. | Migration specialist (interviewer) |
### Discovery phase
| Task | Description | Skills required |
| --- | --- | --- |
| Confirm server list. | Confirm the list of servers and the purpose of each server with the application owner or technical lead. | Migration specialist |
| Identify and add server groups. | Identify server groups such as web servers or application servers, and add this information to the application discovery template. Select the tier of the application (web, application, database) that each server should belong to. | Migration specialist |
| Fill in the application discovery template. | Complete the details of the application discovery template with the help of the migration team, application team, and AWS. | Migration specialist |
| Add missing server details (middleware and OS teams). | Ask middleware and operating system (OS) teams to review the application discovery template and add any missing server details, including database information. | Migration specialist |
| Get inbound/outbound traffic rules (network team). | Ask the network team to get the inbound/outbound traffic rules for the source and destination servers. The network team should also add existing firewall rules, export these to a security group format, and add existing load balancers to the application discovery template. | Migration specialist |
| Identify required tagging. | Determine the tagging requirements for the application. | Migration specialist |
| Create firewall request details. | Capture and filter the firewall rules that are required to communicate with the application. | Migration specialist, Solutions architect, Network lead |
| Update the EC2 instance type. | Update the Amazon Elastic Compute Cloud (Amazon EC2) instance type to be used in the target environment, based on infrastructure and server requirements. | Migration specialist, Solutions architect, Network lead |
| Identify the current state diagram. | Identify or create the diagram that shows the current state of the application. This diagram will be used in the information security (InfoSec) request. | Migration specialist, Solutions architect |
| Finalize the future state diagram. | Finalize the diagram that shows the future (target) state for the application. This diagram will also be used in the InfoSec request. | Migration specialist, Solutions architect |
| Create firewall or security group service requests. | Create firewall or security group service requests (for development/QA, pre-production, and production). If you're using Cloud Migration Factory, include replication-specific ports if they're not already open. | Migration specialist, Solutions architect, Network lead |
| Review firewall or security group requests (InfoSec team). | In this step, the InfoSec team reviews and approves the firewall or security group requests that were created in the previous step. | InfoSec engineer, Migration specialist |
| Implement firewall security group requests (network team). | After the InfoSec team approves the firewall requests, the network team implements the required inbound/outbound firewall rules. | Migration specialist, Solutions architect, Network lead |
### Build phase (repeat for development/QA, pre-production, and production environments)
| Task | Description | Skills required |
| --- | --- | --- |
| Import the application and server data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/rehost-on-premises-workloads-in-the-aws-cloud-migration-checklist.html)If you aren't using Cloud Migration Factory, follow the instructions for setting up your migration tool. | Migration specialist, Cloud administrator |
| Check prerequisites for source servers. | Connect with the in-scope source servers to verify prerequisites such as TCP port 1500, TCP port 443, root volume free space, .NET Framework version, and other parameters. These are required for replication. For additional information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#prerequisites-2). | Migration specialist, Cloud administrator |
| Create a service request to install replication agents. | Create a service request to install replication agents on the in-scope servers for development/QA, pre-production, or production. | Migration specialist, Cloud administrator |
| Install the replication agents. | Install the replication agents on the in-scope source servers on the development/QA, pre-production, or production machines. For additional information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#install-the-replication-agents). | Migration specialist, Cloud administrator |
| Push the post-launch scripts. | Application Migration Service supports post-launch scripts to help you automate OS-level activities such as installing or uninstalling software after you launch target instances. This step pushes the post-launch scripts to Windows or Linux machines, depending on the servers identified for migration. For instructions, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#push-the-post-launch-scripts). | Migration specialist, Cloud administrator |
| Verify the replication status. | Confirm the replication status for the in-scope source servers automatically by using the provided script. The script repeats every five minutes until the status of all source servers in the given wave changes to **Healthy**. For instructions, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#verify-the-replication-status). | Migration specialist, Cloud administrator |
| Create the admin user. | A local admin or sudo user on source machines might be needed to troubleshoot any issues after migration cutover from the in-scope source servers to AWS. The migration team uses this user to log in to the target server when the authentication server (for example, the DC or LDAP server) is not reachable. For instructions for this step, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/step4.html#add-a-user-to-the-admin-group). | Migration specialist, Cloud administrator |
| Validate the launch template. | Validate the server metadata to make sure it works successfully and has no invalid data. This step validates both test and cutover metadata. For instructions, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#validate-launch-template-1). | Migration specialist, Cloud administrator |
### Test phase (repeat for development/QA, pre-production, and production environments)
| Task | Description | Skills required |
| --- | --- | --- |
| Create a service request. | Create a service request for the infrastructure team and other teams to perform application cutover to development/QA, pre-production, or production instances. | Migration specialist, Cloud administrator |
| Configure a load balancer (optional). | Configure required load balancers, such as an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html) or an [F5 load balancer](https://www.f5.com/resources/white-papers/load-balancing-101-nuts-and-bolts) with iRules. | Migration specialist, Cloud administrator |
| Launch instances for testing. | Launch all target machines for a given wave in Application Migration Service in test mode. For additional information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#launch-instances-for-testing). | Migration specialist, Cloud administrator |
| Verify the target instance status. | Verify the status of the target instance by checking the bootup process for all in-scope source servers in the same wave. It may take up to 30 minutes for the target instances to boot up. You can check the status manually by logging in to the Amazon EC2 console, searching for the source server name, and reviewing the **Status check** column. The status **2/2 checks passed** indicates that the instance is healthy from an infrastructure perspective. | Migration specialist, Cloud administrator |
| Modify DNS entries. | Modify Domain Name System (DNS) entries. (Use `resolv.conf` or `host.conf` for a Microsoft Windows environment.) Configure each EC2 instance to point to the new IP address of this host.Make sure that there are no DNS conflicts between on-premises and AWS Cloud servers. This step and the following steps are optional, depending on the environment where the server is hosted. | Migration specialist, Cloud administrator |
| Test connectivity to backend hosts from EC2 instances. | Check the logins by using the domain credentials for the migrated servers. | Migration specialist, Cloud administrator |
| Update the DNS A record. | Update the DNS A record for each host to point to the new Amazon EC2 private IP address. | Migration specialist, Cloud administrator |
| Update the DNS CNAME record. | Update the DNS CNAME record for virtual IPs (load balancer names) to point to the cluster for web and application servers. | Migration specialist, Cloud administrator |
| Test the application in applicable environments. | Log in to the new EC2 instance and test the application in the development/QA, pre-production, and production environments. | Migration specialist, Cloud administrator |
| Mark as ready for cutover. | When testing is complete, change the status of the source server to indicate that it's ready for cutover, so users can launch a cutover instance. For instructions, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#mark-as-ready-for-cutover). | Migration specialist, Cloud administrator |
### Cutover phase
| Task | Description | Skills required |
| --- | --- | --- |
| Create production deployment plan. | Create a production deployment plan (including a backout plan). | Migration specialist, Cloud administrator |
| Notify operations team of downtime. | Notify the operations team of the downtime schedule for the servers. Some teams might require a change request or service request (CR/SR) ticket for this notification. | Migration specialist, Cloud administrator |
| Replicate production machines. | Replicate production machines by using Application Migration Service or another migration tool. | Migration specialist, Cloud administrator |
| Shut down in-scope source servers. | After you verify the source servers’ replication status, you can shut down the source servers to stop transactions from client applications to the servers. You can shut down the source servers in the cutover window. For more information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#shut-down-the-in-scope-source-servers). | Cloud administrator |
| Launch instances for cutover. | Launch all target machines for a given wave in Application Migration Service in cutover mode. For more information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-factory-web-console.html#launch-instances-for-cutover). | Migration specialist, Cloud administrator |
| Retrieve target instance IPs. | Retrieve the IPs for target instances. If the DNS update is a manual process in your environment, you would need to get the new IP addresses for all target instances. For more information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-command-prompt.html#retrieve-the-target-instance-ip). | Migration specialist, Cloud administrator |
| Verify target server connections. | After you update the DNS records, connect to the target instances with the host name to verify the connections. For more information, see the [Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/list-of-automated-migration-activities-using-command-prompt.html#verify-the-target-server-connections). | Migration specialist, Cloud administrator |
## Related resources
+ [How to migrate](https://aws.amazon.com/migrate-modernize-build/cloud-migration/how-to-migrate/)
+ [AWS Cloud Migration Factory Implementation Guide](https://docs.aws.amazon.com/solutions/latest/cloud-migration-factory-on-aws/solution-overview.html)
+ [Automating large-scale server migrations with Cloud Migration Factory](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-factory-cloudendure/welcome.html)
+ [AWS Application Migration Service User Guide](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html)
+ [AWS Migration Acceleration Program](https://aws.amazon.com/migration-acceleration-program/)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/8e2d2d72-30cc-4e98-8abd-ac2ef95e599b/attachments/attachment.zip)
# Set up Multi-AZ infrastructure for a SQL Server Always On FCI by using Amazon FSx
*Manish Garg, T.V.R.L.Phani Kumar Dadi, Nishad Mankar, and RAJNEESH TYAGI, Amazon Web Services*
## Summary
If you need to migrate a large number of Microsoft SQL Server Always On Failover Cluster Instances (FCIs) quickly, this pattern can help you minimize provisioning time. By using automation and Amazon FSx for Windows File Server, it reduces manual efforts, human-made errors, and the time required to deploy a large number of clusters.
This pattern sets up the infrastructure for SQL Server FCIs in a Multi-Availability Zone (Multi-AZ) deployment on Amazon Web Services (AWS). The provisioning of the AWS services required for this infrastructure is automated by using [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) templates. SQL Server installation and cluster node creation on an [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) instance is performed by using PowerShell commands.
This solution uses a highly available Multi-AZ [Amazon FSx for Windows](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/what-is.html) file system as the shared witness for storing the SQL Server database files. The Amazon FSx file system and EC2 Windows instances that host SQL Server are joined to the same AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD) domain.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An AWS user with sufficient permissions to provision resources using AWS CloudFormation templates
+ AWS Directory Service for Microsoft Active Directory
+ Credentials in AWS Secrets Manager to authenticate to AWS Managed Microsoft AD in a key-value pair:
+ `ADDomainName`:
+ `ADDomainJoinUserName`:
+ `ADDomainJoinPassword`:
+ `TargetOU`:
**Note**
You will use the same key name in AWS Systems Manager automation for the AWS Managed Microsoft AD join activity.
+ SQL Server media files for SQL Server installation and Windows service or domain accounts created, which will be used during cluster creation
+ A virtual private cloud (VPC), with two public subnets in separate Availability Zones, two private subnets in the Availability Zones, an internet gateway, NAT gateways, route table associations, and a jump server
**Product versions**
+ Windows Server 2012 R2 and Microsoft SQL Server 2016
## Architecture
**Source technology stack**
+ On-premises SQL Server with FCIs using a shared drive
**Target technology stack**
+ AWS EC2 instances
+ Amazon FSx for Windows File Server
+ AWS Systems Manager Automation runbook
+ Network configurations (VPC, subnets, internet gateway, NAT gateways, jump server, security groups)
+ AWS Secrets Manager
+ AWS Managed Microsoft AD
+ Amazon EventBridge
+ AWS Identity and Access Management (IAM)
**Target architecture**
The following diagram shows an AWS account in a single AWS Region, with a VPC that includes two Availability Zones, two public subnets with NAT gateways, a jump server in the first public subnet, two private subnets, each with an EC2 instance for a SQL Server node in a node security group, and an Amazon FSx file system connecting to each of the SQL Server nodes. AWS Directory Service, Amazon EventBridge, AWS Secrets Manager, and AWS Systems Manager are also included.
![\[Multi-AZ architecture with resources in public and private subnets, with node security groups.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f09c0164-be2d-4665-a574-7ec29fd25082/images/543829a9-e130-4542-9c4e-7518c6cbe967.png)
**Automation and scale**
+ You can use AWS Systems Manager to join AWS Managed Microsoft AD and perform the SQL Server installation.
## Tools
**AWS services**
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.
+ [AWS Directory Service](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/what_is.html) provides multiple ways to use Microsoft Active Directory (AD) with other AWS services such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS) for SQL Server, and Amazon FSx for Windows File Server.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
+ [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) helps you manage your applications and infrastructure running in the AWS Cloud. It simplifies application and resource management, shortens the time to detect and resolve operational problems, and helps you manage your AWS resources securely at scale.
**Other tools**
+ [PowerShell](https://learn.microsoft.com/en-us/powershell/) is a Microsoft automation and configuration management program that runs on Windows, Linux, and macOS. This pattern uses PowerShell scripts.
**Code repository**
The code for this pattern is available in the GitHub [aws-windows-failover-cluster-automation](https://github.com/aws-samples/aws-windows-failover-cluster-automation) repository.
## Best practices
+ The IAM roles that are used to deploy this solution should adhere to the principle of least privilege. For more information, see the [IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
+ Follow the [AWS CloudFormation best practices](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html).
## Epics
### Deploy the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the Systems Manager CloudFormation stack. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) | AWS DevOps, DevOps engineer |
| Deploy the infrastructure stack. | After successful deployment of the Systems Manager stack, create the `infra` stack, which includes EC2 instance nodes, security groups, the Amazon FSx for Windows File Server file system, and the IAM role.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) | AWS DevOps, DevOps engineer |
### Set up the Windows SQL Server Always On FCI
| Task | Description | Skills required |
| --- | --- | --- |
| Install Windows tools. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) | AWS DevOps, DevOps engineer, DBA |
| Prestage the cluster computer objects in Active Directory Domain Services. | To prestage the cluster name object (CNO) in Active Directory Domain Services (AD DS) and prestage a virtual computer object (VCO) for a clustered role, follow the instructions in the [Windows Server documentation](https://learn.microsoft.com/en-us/windows-server/failover-clustering/prestage-cluster-adds). | AWS DevOps, DBA, DevOps engineer |
| Create the WSFC. | To create the Windows Server Failover Clustering (WSFC) cluster, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) | AWS DevOps, DBA, DevOps engineer |
| Install the SQL Server failover cluster. | After the WSFC cluster is set up, install the SQL Server cluster on the primary instance (node1).[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html)
| AWS DevOps, DBA, DevOps engineer |
| Add a secondary node to the cluster. | To add SQL Server to the secondary node (node 2), run the following PowerShell command.
| AWS DevOps, DBA, DevOps engineer |
| Test the SQL Server FCI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) | DBA, DevOps engineer |
### Clean up resources
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up resources. | To clean up the resources, use the AWS CloudFormation stack deletion process:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html)After the stack deletion is complete, the stacks will be in the `DELETE_COMPLETE` state. Stacks in the `DELETE_COMPLETE` state aren’t displayed in the CloudFormation console by default. To display deleted stacks, you must change the stack view filter as described in [Viewing deleted stacks on the AWS CloudFormation console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-view-deleted-stacks.html).If the deletion failed, a stack will be in the `DELETE_FAILED` state. For solutions, see [Delete stack fails](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-delete-stack-fails) in the CloudFormation documentation. | AWS DevOps, DBA, DevOps engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| AWS CloudFormation template failure | If the CloudFormation template fails during deployment, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) |
| AWS Managed Microsoft AD join failure | To troubleshoot the join issues, follow these steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.html) |
## Related resources
+ [Simplify your Microsoft SQL Server high availability deployments using Amazon FSx for Windows File Server](https://aws.amazon.com/blogs/storage/simplify-your-microsoft-sql-server-high-availability-deployments-using-amazon-fsx-for-windows-file-server/)
+ [Using FSx for Windows File Server with Microsoft SQL Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/sql-server.html)
# Use BMC Discovery queries to extract migration data for migration planning
*Ben Tailor-Hamblin, Emma Baldry, Simon Cunningham, and Shabnam Khan, Amazon Web Services*
## Summary
This guide provides query examples and steps to help you extract data from your on-premises infrastructure and applications by using BMC Discovery. The pattern shows you how to use BMC Discovery queries to scan your infrastructure and extract software, service, and dependency information. The extracted data is required for the assess and mobilize phases of a large-scale migration to the Amazon Web Services (AWS) Cloud. You can use this data to make critical decisions about which applications to migrate together as part of your migration plan.
## Prerequisites and limitations
**Prerequisites**
+ A license for BMC Discovery (formerly BMC ADDM) or the software as a service (SaaS) version of BMC Helix Discovery
+ On-premises or SaaS version of BMC Discovery, [installed](https://docs.bmc.com/docs/discovery/221/installing-1050933835.html)
**Note**
For on-premises versions of BMC Discovery, you must install the application on a client network with access to all networking and server devices that are in scope for a migration across multiple data centers. Access to the client network must be provided according to application installation instructions. If the scanning of Windows Server information is required, then you must set up a Windows proxy manager device in the network.
+ [Networking access](https://docs.bmc.com/docs/discovery/221/network-ports-used-for-discovery-communications-1050933821.html) to allow the application to scan devices across data centers, if you’re using BMC Helix Discovery
**Product versions**
+ BMC Discovery 22.2 (12.5)
+ BMC Discovery 22.1 (12.4)
+ BMC Discovery 21.3 (12.3)
+ BMC Discovery 21.05 (12.2)
+ BMC Discovery 20.08 (12.1)
+ BMC Discovery 20.02 (12.0)
+ BMC Discovery 11.3
+ BMC Discovery 11.2
+ BMC Discovery 11.1
+ BMC Discovery 11.0
+ BMC Atrium Discovery 10.2
+ BMC Atrium Discovery 10.1
+ BMC Atrium Discovery 10.0
## Architecture
The following diagram shows how asset managers can use BMC Discovery queries to scan BMC-modeled applications in both SaaS and on-premises environments.
![\[Architecture that uses BMC Discovery to extract software, service, and dependency information.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5e549882-8deb-4459-8891-e39bbf80e320/images/0ebb3e68-5828-45aa-86f4-c741c7b6cd94.jpeg)
The diagram shows the following workflow: An asset manager uses BMC Discovery or BMC Helix Discovery to scan database and software instances running on virtual servers hosted on multiple physical servers. The tool can model applications with components spanning multiple virtual and physical servers.
**Technology stack**
+ BMC Discovery
+ BMC Helix Discovery
## Tools
+ [BMC Discovery](https://docs.bmc.com/xwiki/bin/view/IT-Operations-Management/Discovery/BMC-Discovery/) is a data center discovery tool that helps you automatically discover your data center.
+ [BMC Helix Discovery](https://www.bmc.com/it-solutions/bmc-helix-discovery.html) is a SaaS-based discovery and dependency modeling system that helps you dynamically model your data assets and their dependencies.
## Best practices
It's a best practice to map application, dependency, and infrastructure data when you migrate to the cloud. Mapping helps you understand the complexity of your current environment and the dependencies among various components.
The asset information these queries provide is important for several reasons:
1. **Planning** – Understanding the dependencies between components helps you plan the migration process more effectively. For example, you might need to migrate certain components first in order to ensure that others can be migrated successfully.
1. **Risk assessment** – Mapping the dependencies between components can help you identify any potential risks or issues that can arise during the migration process. For example, you might discover that certain components rely on outdated or unsupported technologies that could cause issues in the cloud.
1. **Cloud architecture** – Mapping your application and infrastructure data can also help you to design a suitable cloud architecture that meets your organizational needs. For example, you might need to design a multi-tier architecture to support high availability or scalability requirements.
Overall, mapping application, dependency, and infrastructure data is a crucial step in the cloud migration process. The mapping exercise can help you better understand your current environment, identify any potential issues or risks, and design a suitable cloud architecture.
## Epics
### Identify and evaluate discovery tooling
| Task | Description | Skills required |
| --- | --- | --- |
| Identify ITSM owners. | Identify the IT Service Management (ITSM) owners (usually by reaching out to the operational support teams). | Migration lead |
| Check CMDB. | Identify the number of configuration management databases (CMDBs) that contain asset information, and then identify the sources of that information. | Migration lead |
| Identify discovery tools and check for use of BMC Discovery. | If your organization is using BMC Discovery to send data about your environment to the CMDB tool, check the scope and coverage of its scans. For example, check if BMC Discovery is scanning all data centers and if the access servers are located in perimeter zones. | Migration lead |
| Check the level of application modelling. | Check if applications are modelled in BMC Discovery. If not, recommend the use of the BMC Discovery tool to model which running software instances provide an application and business service. | Migration engineer, Migration lead |
### Extract infrastructure data
| Task | Description | Skills required |
| --- | --- | --- |
| Extract data on physical and virtual servers. | To extract data on the physical and virtual servers scanned by BMC Discovery, use [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html) to run the following query:
search Host show key as 'Serverid', virtual, name as 'HOSTNAME', os_type as 'osName', os_version as 'OS Version', num_logical_processors as 'Logical Processor Counts', cores_per_processor as 'Cores per Processor', logical_ram as 'Logical RAM', #Consumer:StorageUse:Provider:DiskDrive.size as 'Size'
You can use extracted data to determine the appropriate instance sizes for migration. | Migration engineer, Migration lead |
| Extract data on modeled applications. | If your applications are modeled in BMC Discovery, you can extract data about the servers that run the application software. To get the server names, use [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html) to run the following query:
search SoftwareInstance show key as 'ApplicationID', #RunningSoftware:HostedSoftware:Host:Host.key as 'ReferenceID', type, name
Applications are modeled in BMC Discovery by a collection of running software instances. The application is dependent on all the servers that run the application software. | BMC Discovery application owner |
| Extract data on databases. | To get a list of all scanned databases and the servers these databases are running on, use [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html) to run the following query:
search Database show key as 'Key', name, type as 'Source Engine Type', #Detail:Detail:ElementWithDetail:SoftwareInstance.name as 'Software Instance', #Detail:Detail:ElementWithDetail:SoftwareInstance.product_version as 'Product Version', #Detail:Detail:ElementWithDetail:SoftwareInstance.edition as 'Edition', #Detail:Detail:ElementWithDetail:SoftwareInstance.#RunningSoftware:HostedSoftware:Host:Host.key as 'ServerID'
| App owner |
| Extract data on server communication. | To get information on all the network communications between servers that’s collected by BMC Discovery from historic network communications logs, use [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html) to run the following query:
search Host TRAVERSE InferredElement:Inference:Associate:DiscoveryAccess TRAVERSE DiscoveryAccess:DiscoveryAccessResult:DiscoveryResult:NetworkConnectionList TRAVERSE List:List:Member:DiscoveredNetworkConnection PROCESS WITH networkConnectionInfo
| BMC Discovery application owner |
| Extract data on application discovery. | To get information on application dependencies, use [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html) to run the following query:
search SoftwareInstance show key as 'SRC App ID', #Dependant:Dependency:DependedUpon:SoftwareInstance.key as 'DEST App ID'
| BMC Discovery application owner |
| Extract data on business services. | To extract data on business services provided by hosts, use [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html) to run the following query:
search Host show name, #Host:HostedSoftware:AggregateSoftware:BusinessService.name as 'Name'
| BMC Discovery application owner |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| A query fails to run or contains unpopulated columns. | Review the asset records in BMC Discovery and determine which fields you require. Then, replace these fields in the query by using the [Query Builder](https://docs.bmc.com/docs/discovery/221/query-builder-1051985747.html). |
| The details of a dependent asset aren’t populated. | This is likely due to access permissions or network connectivity. The discovery tool might not have the necessary permissions to access certain assets, particularly if they are on different networks or in different environments.We recommend that you work closely with discovery subject matter experts to ensure that all relevant assets are identified. |
## Related resources
**References**
+ [BMC Discovery Licensing entitlement](https://docs.bmc.com/docs/discovery/bmc-discovery-licensing-entitlement-531336348.html) (BMC documentation)
+ [BMC Discovery features and components](https://docs.bmc.com/docs/discovery/221/bmc-discovery-features-and-components-1052418000.html) (BMC documentation)
+ [BMC Discovery User Guide](https://docs.bmc.com/xwiki/bin/view/IT-Operations-Management/Discovery/BMC-Discovery/) (BMC documentation)
+ [Searching for data (on BMC Discovery)](https://docs.bmc.com/docs/discovery/120/searching-for-data-911457232.html) (BMC documentation)
+ [Portfolio discovery and analysis for migration](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-portfolio-discovery/welcome.html) (AWS Prescriptive Guidance)
**Tutorials and videos**
+ [BMC Discovery: Webinar - Reporting Query Best Practices (Part 1)](https://www.youtube.com/watch?v=iwXy6x40kO8) (YouTube)
# Relocate
**Topics**
+ [Migrate an Amazon RDS for Oracle database to another AWS account and AWS Region using AWS DMS for ongoing replication](migrate-an-amazon-rds-for-oracle-database-to-another-aws-account-and-aws-region-using-aws-dms-for-ongoing-replication.md)
+ [Migrate an Amazon RDS DB instance to another VPC or account](migrate-an-amazon-rds-db-instance-to-another-vpc-or-account.md)
+ [Migrate an Amazon Redshift cluster to an AWS Region in China](migrate-an-amazon-redshift-cluster-to-an-aws-region-in-china.md)
+ [Transport PostgreSQL databases between two Amazon RDS DB instances using pg\$1transport](transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.md)
# Migrate an Amazon RDS for Oracle database to another AWS account and AWS Region using AWS DMS for ongoing replication
*Durga Prasad Cheepuri and Eduardo Valentim, Amazon Web Services*
## Summary
|
|
| Warning: IAM users have long-term credentials, which presents a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. |
| --- |
This pattern walks you through the steps for migrating an Amazon Relational Database Service (Amazon RDS) for Oracle source database to a different AWS account and AWS Region. The pattern uses a DB snapshot for a one-time full data load, and enables AWS Database Migration Service (AWS DMS) for ongoing replication.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account that contains the source Amazon RDS for Oracle database, which has been encrypted using a non-default AWS Key Management Service (AWS KMS) key
+ An active AWS account in a different AWS Region from the source database, to use for the target Amazon RDS for Oracle database
+ Virtual private cloud (VPC) peering between the source and target VPCs
+ Familiarity with [using an Oracle database as a source for AWS DMS](http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ Familiarity with [using an Oracle database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html)
**Product versions**
+ Oracle versions 11g (versions 11.2.0.3.v1 and later) and up to 12.2, and 18c. For the latest list of supported versions and editions, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and with [Using an Oracle database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html) in the AWS documentation. For Oracle versions supported by Amazon RDS, see [Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html).
## Architecture
**Source and target technology stacks**
+ Amazon RDS for Oracle DB instance
![\[Source AWS account connecting to target AWS account that contains source and target Regions\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5ecd5359-884e-455c-b5d0-ef08eda2ea1f/images/e17fa7fe-d924-4f35-9707-b93572fa1227.png)
**Ongoing replication architecture**
![\[DB on an EC2 instance connecting through VPC peering to a replication instance and Amazon RDS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5ecd5359-884e-455c-b5d0-ef08eda2ea1f/images/b60b3500-5d29-487a-bbab-0ae9f3f386aa.png)
## Tools
**Tools used for one-time full data load**
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) creates a storage volume snapshot of your DB instance, backing up the entire DB instance and not just individual databases. When you create a DB snapshot, you need to identify which DB instance you are going to back up, and then give your DB snapshot a name so you can restore from it later. The amount of time it takes to create a snapshot varies with the size of your databases. Because the snapshot includes the entire storage volume, the size of files, such as temporary files, also affects the amount of time it takes to create the snapshot. For more information about using DB snapshots, see [Creating a DB Snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html) in the Amazon RDS documentation.
+ [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) creates a** **key for** **Amazon RDS** **encryption. When you create an encrypted DB instance, you can also supply the [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) key identifier for your encryption key. If you don't specify an [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) key identifier, Amazon RDS uses your default encryption key for your new DB instance. [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) creates your default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS Region. For this pattern, the Amazon RDS DB instance should be encrypted using the non-default [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) key. For more information about using [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) keys for Amazon RDS encryption, see [Encrypting Amazon RDS resources](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html) in the Amazon RDS documentation.
**Tools used for ongoing replication**
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) is used to replicate ongoing changes and to keep the source and target databases in sync. For more information about using AWS DMS for ongoing replication, see [Working with an AWS DMS replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.html) in the AWS DMS documentation.
## Epics
### Configure your source AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the source Oracle DB instance. | Let the Amazon RDS for Oracle DB instance run in ARCHIVELOG mode, and set the retention period. For details, see [Working with an AWS managed Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Amazon-Managed). | DBA |
| Set supplemental logging for the source Oracle DB instance. | Set database-level and table-level supplemental logging for the Amazon RDS for Oracle DB instance. For details, see [Working with an AWS managed Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Amazon-Managed). | DBA |
| Update the AWS KMS key policy in the source account. | Update the AWS KMS key policy in the source AWS account to allow the target AWS account to use the encrypted Amazon RDS AWS KMS key. For details, see the [AWS KMS documentation](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html#key-policy-modifying-external-accounts). | SysAdmin |
| Create a manual Amazon RDS DB snapshot of the source DB instance. | | AWS IAM user |
| Share the manual, encrypted Amazon RDS snapshot with the target AWS account. | For details, see [Sharing a DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ShareSnapshot.html). | AWS IAM user |
### Configure your target AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Attach a policy. | In the target AWS account, attach an AWS Identity and Access Management (IAM) policy to the root IAM user, to allow the IAM user to copy an encrypted DB snapshot using the shared AWS KMS key. | SysAdmin |
| Switch to the source AWS Region. | | AWS IAM user |
| Copy the shared snapshot. | In the Amazon RDS console, in the **Snapshots** pane, choose **Shared with Me**, and select the shared snapshot. Copy the snapshot to the same AWS Region as the source database by using the Amazon Resource Name (ARN) for the AWS KMS key used by the source database. For details, see [Copying a DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CopySnapshot.html). | AWS IAM user |
| Switch to the target AWS Region, and create a new AWS KMS key. | | AWS IAM user |
| Copy the snapshot. | Switch to the source AWS Region. On the Amazon RDS console, in the **Snapshots** pane, choose **Owned by Me**, and select the copied snapshot. Copy the snapshot to the target AWS Region by using the AWS KMS key for the new target AWS Region. | AWS IAM user |
| Restore the snapshot. | Switch to the target AWS Region. On the Amazon RDS console, in the **Snapshots** pane, choose **Owned by Me**. Select the copied snapshot and restore it to an Amazon RDS for Oracle DB instance. For details, see [Restoring from a DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html). | AWS IAM user |
### Prepare your source database for ongoing replication
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Oracle user with the appropriate permissions. | Create an Oracle user with the required privileges for Oracle as a source for AWS DMS. For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). | DBA |
| Configure the source database for Oracle LogMiner or Oracle Binary Reader. | | DBA |
### Prepare your target database for ongoing replication
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Oracle user with the appropriate permissions. | Create an Oracle user with the required privileges for Oracle as a target for AWS DMS. For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html#CHAP_Target.Oracle.Privileges). | DBA |
### Create AWS DMS components
| Task | Description | Skills required |
| --- | --- | --- |
| Create a replication instance in the target AWS Region. | Create a replication instance in the VPC of the target AWS Region. For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html#CHAP_GettingStarted.ReplicationInstance). | AWS IAM user |
| Create source and target endpoints with required encryption, and test connections. | For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html#CHAP_GettingStarted.Endpoints). | DBA |
| Create replication tasks. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-amazon-rds-for-oracle-database-to-another-aws-account-and-aws-region-using-aws-dms-for-ongoing-replication.html)For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html#CHAP_GettingStarted.Tasks). | IAM user |
| Start the tasks and monitor them. | For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html). | AWS IAM user |
| Enable validation on the task if needed. | Note that enabling validation does have a performance impact on the replication. For details, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html). | AWS IAM user |
## Related resources
+ [Changing a key policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html#key-policy-modifying-external-accounts)
+ [Creating a manual Amazon RDS DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html)
+ [Sharing a manual Amazon RDS DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ShareSnapshot.html)
+ [Copying a snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CopySnapshot.html)
+ [Restoring from an Amazon RDS DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html)
+ [Getting started with AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html)
+ [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html)
+ [Using an Oracle database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html)
+ [AWS DMS setup using VPC peering](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.VPC.html#CHAP_ReplicationInstance.VPC.Configurations.ScenarioVPCPeer)
+ [How do I share manual Amazon RDS DB snapshots or DB cluster snapshots with another AWS account?](https://aws.amazon.com/premiumsupport/knowledge-center/rds-snapshots-share-account/) (AWS Knowledge Center article)
# Migrate an Amazon RDS DB instance to another VPC or account
*Dhrubajyoti Mukherjee, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an Amazon Relational Database Service (Amazon RDS) DB instance from one virtual private cloud (VPC) to another in the same AWS account, or from one AWS account to another AWS account.
This pattern is useful if you want to migrate your Amazon RDS DB instances to another VPC or account for separation or security reasons (for example, when you want to place your application stack and database in different VPCs).
Migrating a DB instance to another AWS account involves steps such as taking a manual snapshot, sharing it, and restoring the snapshot in the target account. This process can be time-consuming, depending on database changes and transaction rates. It also causes database downtime, so plan ahead for the migration. Consider a blue/green deployment strategy to minimize downtime. Alternatively, you can evaluate AWS Data Migration Service (AWS DMS) to minimize downtime for the change. However, this pattern doesn’t cover this option. To learn more, see the [AWS DMS documentation.](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ AWS Identity and Access Management (IAM) permissions required for the VPC, subnets, and Amazon RDS console
**Limitations**
+ Changes to a VPC cause a database reboot, resulting in application outages. We recommend that you migrate during low peak times.
+ Limitations when migrating Amazon RDS to another VPC:
+ The DB instance you’re migrating must be a single instance with no standby. It must not be a member of a cluster.
+ Amazon RDS must not be in multiple Availability Zones.
+ Amazon RDS must not have any read replicas.
+ The subnet group created in the target VPC must have subnets from the Availability Zone where the source database is running.
+ Limitations when migrating Amazon RDS to another AWS account:
+ Sharing snapshots encrypted with the default service key for Amazon RDS isn‘t currently supported.
## Architecture
**Migrating to a VPC in the same AWS account**
The following diagram shows the workflow for migrating an Amazon RDS DB instance to a different VPC in the same AWS account.
![\[Workflow for migrating an Amazon RDS DB instance to a different VPC in the same AWS account\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/dabcee69-9cc6-47f9-9964-635e349caaaf/images/73e16544-6276-4f03-9ae2-42b8c7c20315.png)
The steps consist of the following. See the [Epics](#migrate-an-amazon-rds-db-instance-to-another-vpc-or-account-epics) section for detailed instructions.
1. Create a DB subnet group in the target VPC. A DB subnet group is a collection of subnets that you can use to specify a specific VPC when you create DB instances.
1. Configure the Amazon RDS DB instance in the source VPC to use the new DB subnet group.
1. Apply the changes to migrate the Amazon RDS DB to the target VPC.
**Migrating to a different AWS account**
The following diagram shows the workflow for migrating an Amazon RDS DB instance to a different AWS account.
![\[Workflow for migrating an Amazon RDS DB instance to a different AWS account\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/dabcee69-9cc6-47f9-9964-635e349caaaf/images/5536e69e-3965-4ca2-8a0b-2573659b5f8f.png)
The steps consist of the following. See the [Epics](#migrate-an-amazon-rds-db-instance-to-another-vpc-or-account-epics) section for detailed instructions.
1. Access the Amazon RDS DB instance in the source AWS account.
1. Create an Amazon RDS snapshot in the source AWS account.
1. Share the Amazon RDS snapshot with the target AWS account.
1. Access the Amazon RDS snapshot in the target AWS account.
1. Create an Amazon RDS DB instance in the target AWS account.
## Tools
**AWS services**
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
## Best practices
+ If database downtime is a concern when migrating an Amazon RDS DB instance to another account, we recommend that you use [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html). This service provides data replication, which causes less than five minutes of outage time.
## Epics
### Migrate to a different VPC in the same AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Create a new VPC. | On the [Amazon VPC console](https://console.aws.amazon.com/vpc/), create a new VPC and subnets with the desired properties and IP address ranges. For detailed instructions, see the [Amazon VPC documentation](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html). | Administrator |
| Create a DB subnet group. | On the [Amazon RDS console](https://console.aws.amazon.com/rds/):[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-amazon-rds-db-instance-to-another-vpc-or-account.html)For additional information, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.CreateDBSubnetGroup). | Administrator |
| Modify the Amazon RDS DB instance to use the new subnet group. | On the Amazon RDS console:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-amazon-rds-db-instance-to-another-vpc-or-account.html)When the migration to the target VPC is complete, the target VPC's default security group is assigned to the Amazon RDS DB instance. You can configure a new security group for that VPC with the required inbound and outbound rules to your DB instance.Alternatively, use the AWS Command Line Interface (AWS CLI) to perform the migration to the target VPC by explicitly providing the new VPC security group ID. For example:
| Administrator |
### Migrate to a different AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Create a new VPC and subnet group in the target AWS account. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-amazon-rds-db-instance-to-another-vpc-or-account.html) | Administrator |
| Share a manual snapshot of the database and share it with the target account. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-amazon-rds-db-instance-to-another-vpc-or-account.html) | Administrator |
| Launch a new Amazon RDS DB instance. | Launch a new Amazon RDS DB instance from the shared snapshot in the target AWS account. For instructions, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html). | Administrator |
## Related resources
+ [Amazon VPC documentation](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html)
+ [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html)
+ [How do I change the VPC for an RDS DB instance?](https://aws.amazon.com/premiumsupport/knowledge-center/change-vpc-rds-db-instance/) (AWS re:Post article)
+ [How do I transfer ownership of Amazon RDS resources to a different AWS account?](https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-rds/) (AWS re:Post article)
+ [How do I share manual Amazon RDS DB snapshots or Aurora DB cluster snapshots with another AWS account?](https://aws.amazon.com/premiumsupport/knowledge-center/rds-snapshots-share-account/) (AWS re:Post article)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
# Migrate an Amazon Redshift cluster to an AWS Region in China
*Jing Yan, Amazon Web Services*
## Summary
This pattern provides a step-by-step approach to migrate an Amazon Redshift cluster to an AWS Region in China from another AWS Region.
This pattern uses SQL commands to recreate all the database objects, and uses the UNLOAD command to move this data from Amazon Redshift to an Amazon Simple Storage Service (Amazon S3) bucket in the source Region. The data is then migrated to an S3 bucket in the AWS Region in China. The COPY command is used to load data from the S3 bucket and transfer it to the target Amazon Redshift cluster.
Amazon Redshift doesn't currently support cross-Region features such as snapshot copying to AWS Regions in China. This pattern provides a way to work around that limitation. You can also reverse the steps in this pattern to migrate data from an AWS Region in China to another AWS Region.
## Prerequisites and limitations
*Prerequisites*
+ Active AWS accounts in both a China Region and an AWS Region outside China
+ Existing Amazon Redshift clusters in both a China Region and an AWS Region outside China
*Limitations*
+ This is an offline migration, which means the source Amazon Redshift cluster cannot perform write operations during the migration.
## Architecture
**Source technology stack **
+ Amazon Redshift cluster in an AWS Region outside China
**Target technology stack **
+ Amazon Redshift cluster in an AWS Region in China
**Target architecture **
![\[Migration of Amazon Redshift cluster data in S3 bucket in an AWS Region to bucket in a China Region.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f7d241d9-b700-406b-95a0-3e47e7f0fa60/images/b6016e3d-76db-4176-8f99-f804da94d3f2.png)
## Tools
**Tools**
+ [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html) – Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. You can use Amazon S3 to store data from Amazon Redshift, and you can copy data from an S3 bucket to Amazon Redshift.
+ [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html) – Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud.
+ [psql](https://www.postgresql.org/docs/8.4/app-psql.html) – psql is a terminal-based front-end to PostgreSQL.
## Epics
### Prepare for migration in the source Region
| Task | Description | Skills required |
| --- | --- | --- |
| Launch and configure an EC2 instance in the source Region. | Sign in to the AWS Management Console and open the Amazon Elastic Compute Cloud (Amazon EC2) console. Your current Region is displayed in the navigation bar at the top of the screen. This Region cannot be an AWS Region in China. From the Amazon EC2 console dashboard, choose "Launch instance," and create and configure an EC2 instance. Important: Ensure your EC2 security groups for inbound rules allow unrestricted access to TCP port 22 from your source machine. For instructions on how to launch and configure an EC2 instance, see the "Related resources" section. | DBA, Developer |
| Install the psql tool. | Download and install PostgreSQL. Amazon Redshift does not provide the psql tool, it is installed with PostgreSQL. For more information about using psql and installing PostgreSQL tools, see the "Related resources" section. | DBA |
| Record the Amazon Redshift cluster details. | Open the Amazon Redshift console, and choose "Clusters" in the navigation pane. Then choose the Amazon Redshift cluster name from the list. On the "Properties" tab, in the "Database configurations" section, record the "Database name" and "Port." Open the "Connection details" section and record the "Endpoint," which is in the "endpoint:/" format. Important: Ensure your Amazon Redshift security groups for inbound rules allow unrestricted access to TCP port 5439 from your EC2 instance. | DBA |
| Connect psql to the Amazon Redshift cluster. | At a command prompt, specify the connection information by running the "psql -h -U -d -p " command. At the psql password prompt, enter the password for the "" user. You are then connected to the Amazon Redshift cluster and can interactively enter commands. | DBA |
| Create an S3 bucket. | Open the Amazon S3 console, and create an S3 bucket to hold the files exported from Amazon Redshift. For instructions on how to create an S3 bucket, see the "Related resources" section. | DBA, AWS General |
| Create an IAM policy that supports unloading data. | Open the AWS Identity and Access Management (IAM) console and choose "Policies." Choose "Create policy," and choose the "JSON" tab. Copy and paste the IAM policy for unloading data from the "Additional information" section. Important: Replace "s3\$1bucket\$1name" with your S3 bucket’s name. Choose "Review policy," and enter a name and description for the policy. Choose "Create policy." | DBA |
| Create an IAM role to allow UNLOAD operation for Amazon Redshift. | Open the IAM console and choose "Roles." Choose "Create role," and choose "AWS service" in "Select type of trusted entity." Choose "Redshift" for the service, choose "Redshift – Customizable," and then choose "Next." Choose the "Unload" policy you created earlier, and choose "Next." Enter a "Role name," and choose "Create role." | DBA |
| Associate IAM role with the Amazon Redshift cluster. | Open the Amazon Redshift console, and choose "Manage IAM roles." Choose "Available roles" from the dropdown menu and choose the role you created earlier. Choose "Apply changes." When the "Status" for the IAM role on the "Manage IAM roles" shows as "In-sync", you can run the UNLOAD command. | DBA |
| Stop write operations to the Amazon Redshift cluster. | You must remember to stop all write operations to the source Amazon Redshift cluster until the migration is complete. | DBA |
### Prepare for migration in the target Region
| Task | Description | Skills required |
| --- | --- | --- |
| Launch and configure an EC2 instance in the target Region. | Sign in to the AWS Management Console for a Region in China, either Beijing or Ningxia. From the Amazon EC2 console, choose "Launch instance," and create and configure an EC2 instance. Important: Make sure your Amazon EC2 security groups for inbound rules allow unrestricted access to TCP port 22 from your source machine. For further instructions on how to launch and configure an EC2 instance, see the "Related resources" section. | DBA |
| Record the Amazon Redshift cluster details. | Open the Amazon Redshift console, and choose "Clusters" in the navigation pane. Then choose the Amazon Redshift cluster name from the list. On the "Properties" tab, in the "Database configurations" section, record the "Database name" and "Port." Open the "Connection details" section and record the "Endpoint," which is in the "endpoint:/" format. Important: Make sure your Amazon Redshift security groups for inbound rules allow unrestricted access to TCP port 5439 from your EC2 instance. | DBA |
| Connect psql to the Amazon Redshift cluster. | At a command prompt, specify the connection information by running the "psql -h -U -d -p " command. At the psql password prompt, enter the password for the "" user. You are then connected to the Amazon Redshift cluster and can interactively enter commands. | DBA |
| Create an S3 bucket. | Open the Amazon S3 console, and create an S3 bucket to hold the exported files from Amazon Redshift. For help with this and other stories, see the "Related resources" section. | DBA |
| Create an IAM policy that supports copying data. | Open the IAM console and choose "Policies." Choose "Create policy," and choose the "JSON" tab. Copy and paste the IAM policy for copying data from the "Additional information" section. Important: Replace "s3\$1bucket\$1name" with your S3 bucket’s name. Choose "Review policy," enter a name and description for the policy. Choose "Create policy." | DBA |
| Create an IAM role to allow COPY operation for Amazon Redshift. | Open the IAM console and choose "Roles." Choose "Create role," and choose "AWS service" in "Select type of trusted entity." Choose "Redshift" for the service, choose "Redshift – Customizable," and then choose "Next." Choose the "Copy" policy you created earlier, and choose "Next." Enter a "Role name," and choose "Create role." | DBA |
| Associate IAM role with the Amazon Redshift cluster. | Open the Amazon Redshift console, and choose "Manage IAM roles." Choose "Available roles" from the dropdown menu and choose the role you created earlier. Choose "Apply changes." When the "Status" for the IAM role on the "Manage IAM roles" shows as "In-sync", you can run the "COPY" command. | DBA |
### Verify source data and object information before beginning the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Verify the rows in the source Amazon Redshift tables. | Use the scripts in the "Additional information" section to verify and record the number of rows in the source Amazon Redshift tables. Remember to split the data evenly for the UNLOAD and COPY scripts. This will improve the data unloading and loading efficiency, because the data quantity covered by each script will be balanced. | DBA |
| Verify the number of database objects in the source Amazon Redshift cluster. | Use the scripts in the "Additional information" section to verify and record the number of databases, users, schemas, tables, views, and user-defined functions (UDFs) in your source Amazon Redshift cluster. | DBA |
| Verify SQL statement results before migration. | Some SQL statements for data validation should be sorted according to actual business and data situations. This is to verify the imported data to ensure it is consistent and displayed correctly. | DBA |
### Migrate data and objects to the target Region
| Task | Description | Skills required |
| --- | --- | --- |
| Generate Amazon Redshift DDL scripts. | Generate Data Definition Language (DDL) scripts by using the links from the "SQL statements to query Amazon Redshift" section in the "Additional information" section. These DDL scripts should include the "create user," "create schema," "privileges on schema to user," "create table/view," "privileges on objects to user," and "create function" queries. | DBA |
| Create objects in the Amazon Redshift cluster for the target Region. | Run the DDL scripts by using the AWS Command Line Interface (AWS CLI) in the AWS Region in China. These scripts will create objects in the Amazon Redshift cluster for the target Region. | DBA |
| Unload source Amazon Redshift cluster data to the S3 bucket. | Run the UNLOAD command to unload data from the Amazon Redshift cluster in the source Region to the S3 bucket. | DBA, Developer |
| Transfer source Region S3 bucket data to target Region S3 bucket. | Transfer the data from your source Region S3 bucket to the target S3 bucket. Because the "\$1 aws s3 sync" command cannot be used, make sure you use the process outlined in the "Transferring Amazon S3 data from AWS Regions to AWS Regions in China" article in the "Related resources" section. | Developer |
| Load data into the target Amazon Redshift cluster. | In the psql tool for your target Region, run the COPY command to load data from the S3 bucket to the target Amazon Redshift cluster. | DBA |
### Verify the data in the source and target Regions after the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Verify and compare the number of rows in the source and target tables. | Verify and compare the number of table rows in the source and target Regions to ensure all are migrated. | DBA |
| Verify and compare the number of source and target database objects. | Verify and compare all database objects in the source and target Regions to ensure all are migrated. | DBA |
| Verify and compare SQL script results in the source and target Regions. | Run the SQL scripts prepared before the migration. Verify and compare the data to ensure that the SQL results are correct. | DBA |
| Reset the passwords of all users in the target Amazon Redshift cluster. | After the migration is complete and all data is verified, you should reset all user passwords for the Amazon Redshift cluster in the AWS Region in China. | DBA |
## Related resources
+ [Transferring Amazon S3 data from AWS Regions to AWS Regions in China](https://aws.amazon.com/cn/blogs/storage/transferring-amazon-s3-data-from-aws-regions-to-aws-regions-in-china/)
+ [Creating an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html)
+ [Resetting an Amazon Redshift user password](https://docs.aws.amazon.com/redshift/latest/dg/r_ALTER_USER.html)
+ [psql documentation](https://www.postgresql.org/docs/8.4/static/app-psql.html)
## Additional information
*IAM policy for unloading data*
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::s3_bucket_name"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::s3_bucket_name/*"]
}
]
}
```
*IAM policy for copying data*
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::s3_bucket_name"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::s3_bucket_name/*"]
}
]
}
```
*SQL statements to query Amazon Redshift*
```
##Database
select * from pg_database where datdba>1;
##User
select * from pg_user where usesysid>1;
##Schema
SELECT n.nspname AS "Name",
pg_catalog.pg_get_userbyid(n.nspowner) AS "Owner"
FROM pg_catalog.pg_namespace n
WHERE n.nspname !~ '^pg_' AND n.nspname <> 'information_schema'
ORDER BY 1;
##Table
select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema');
select schemaname,count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema') group by schemaname order by 1;
##View
SELECT
n.nspname AS schemaname,c.relname AS viewname,pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
FROM
pg_catalog.pg_class AS c
INNER JOIN
pg_catalog.pg_namespace AS n
ON c.relnamespace = n.oid
WHERE relkind = 'v' and n.nspname not in ('information_schema','pg_catalog');
##UDF
SELECT
n.nspname AS schemaname,
p.proname AS proname,
pg_catalog.pg_get_userbyid(p.proowner) as "Owner"
FROM pg_proc p
LEFT JOIN pg_namespace n on n.oid = p.pronamespace
WHERE p.proowner != 1;
```
*SQL scripts to generate DDL statements*
+ [Get\$1schema\$1priv\$1by\$1user script](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_get_schema_priv_by_user.sql)
+ [Generate\$1tbl\$1ddl script](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql)
+ [Generate\$1view\$1ddl](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_view_ddl.sql)
+ [Generate\$1user\$1grant\$1revoke\$1ddl](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_user_grant_revoke_ddl.sql)
+ [Generate\$1udf\$1ddl](https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_udf_ddl.sql)
# Transport PostgreSQL databases between two Amazon RDS DB instances using pg\$1transport
*Raunak Rishabh and Jitender Kumar, Amazon Web Services*
## Summary
This pattern describes the steps for migrating extremely large databases between two Amazon Relational Database Service (Amazon RDS) for PostgreSQL DB instances by using the **pg\$1transport** extension. This extension provides a physical transport mechanism to move each database. By streaming the database files with minimal processing, it provides an extremely fast method for migrating large databases between DB instances with minimal downtime. This extension uses a pull model where the target DB instance imports the database from the source DB instance.
## Prerequisites and limitations
**Prerequisites **
+ Both DB instances must run the same major version of PostgreSQL.
+ The database must not exist on the target. Otherwise, the transport fails.
+ No extension other than **pg\$1transport** must be enabled in the source database.
+ All source database objects must be in the default **pg\$1default** tablespace.
+ The security group of the source DB instance should allow traffic from the target DB instance.
+ Install a PostgreSQL client like [psql](https://www.postgresql.org/docs/11/app-psql.html) or [PgAdmin](https://www.pgadmin.org/) to work with the Amazon RDS PostgreSQL DB instance. You can install the client either in your local system or use an Amazon Elastic Compute Cloud (Amazon EC2) instance. In this pattern, we use psql on an EC2 instance.
**Limitations **
+ You can't transport databases between different major versions of Amazon RDS for PostgreSQL.
+ The access privileges and ownership from the source database are not transferred to the target database.
+ You can't transport databases on read replicas or on parent instances of read replicas.
+ You can't use **reg** data types in any database tables that you plan to transport with this method.
+ You can run up to 32 total transports (including both imports and exports) at the same time on a DB instance.
+ You cannot rename or include/exclude tables. Everything is migrated as is.
**Caution**
+ Make backups before removing the extension, because removing the extension also removes dependent objects and some data that's critical to the operation of the database.
+ Consider the instance class and processes running on other databases on the source instance when you determine the number of workers and `work_mem` values for **pg\$1transport**.
+ When the transport starts, all connections on the source database are ended and the database is put into read-only mode.
**Note**
When the transport is running on one database, it doesn’t affect other databases on the same server.** **
**Product versions**
+ Amazon RDS for PostgreSQL 10.10 and later, and Amazon RDS for PostgreSQL 11.5 and later. For the latest version information, see [Transporting PostgreSQL Databases Between DB Instances](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.TransportableDB.html) in the Amazon RDS documentation.
## Architecture
![\[Transporting PostgreSQL databases between Amazon RDS DB instances\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/d5fb7ea3-32b7-4602-b382-3cf5c075c7c9/images/aec4d8d2-37a8-4136-9042-f9667ac4aebb.png)
## Tools
+ **pg\$1transport** provides a physical transport mechanism to move each database. By streaming the database files with minimal processing, physical transport moves data much faster than traditional dump and load processes and requires minimal downtime. PostgreSQL transportable databases use a pull model where the destination DB instance imports the database from the source DB instance. You install this extension on your DB instances when you prepare the source and target environments, as explained in this pattern.
+ [psql](https://www.postgresql.org/docs/11/app-psql.html) enables you to connect to, and work with, your PostgreSQL DB instances. To install **psql** on your system, see the [PostgreSQL Downloads](https://www.postgresql.org/download/) page.
## Epics
### Create the target parameter group
| Task | Description | Skills required |
| --- | --- | --- |
| Create a parameter group for the target system. | Specify a group name that identifies it as a target parameter group; for example, `pgtarget-param-group`. For instructions, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html#USER_WorkingWithParamGroups.Creating). | DBA |
| Modify the parameters for the parameter group. | Set the following parameters:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.html)For more information about these parameters, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.TransportableDB.html). | DBA |
### Create the source parameter group
| Task | Description | Skills required |
| --- | --- | --- |
| Create a parameter group for the source system. | Specify a group name that identifies it as a source parameter group; for example, `pgsource-param-group`. For instructions, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html#USER_WorkingWithParamGroups.Creating). | DBA |
| Modify the parameters for the parameter group. | Set the following parameters:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.html)For more information about these parameters, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.TransportableDB.html). | DBA |
### Prepare the target environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create a new Amazon RDS for PostgreSQL DB instance to transport your source database to. | Determine the instance class and PostgreSQL version based on your business requirements. | DBA, Systems administrator, Database architect |
| Modify the security group of the target to allow connections on the DB instance port from the EC2 instance. | By default, the port for the PostgreSQL instance is 5432. If you're using another port, connections to that port must be open for the EC2 instance. | DBA, Systems administrator |
| Modify the instance, and assign the new target parameter group. | For example, `pgtarget-param-group`. | DBA |
| Restart the target Amazon RDS DB instance. | The parameters `shared_preload_libraries` and `max_worker_processes` are static parameters and require a reboot of the instance. | DBA, Systems administrator |
| Connect to the database from the EC2 instance using psql. | Use the command:
psql -h -p PORT -U username -d database -W
| DBA |
| Create the pg\$1transport extension. | Run the following query as a user with the `rds_superuser` role:
create extension pg_transport;
| DBA |
### Prepare the source environment
| Task | Description | Skills required |
| --- | --- | --- |
| Modify the security group of the source to allow connections on the DB instance port from the Amazon EC2 instance and target DB instance | By default, the port for PostgreSQL instance is 5432. If you're using another port, connections to that port must be open for the EC2 instance. | DBA, Systems administrator |
| Modify the instance and assign the new source parameter group. | For example, `pgsource-param-group`. | DBA |
| Restart the source Amazon RDS DB instance. | The parameters `shared_preload_libraries` and `max_worker_processes` are static parameters and require a reboot of the instance. | DBA |
| Connect to the database from the EC2 instance using psql. | Use the command:
psql -h -p PORT -U username -d database -W
| DBA |
| Create the pg\$1transport extension and remove all other extensions from the databases to be transported. | The transport will fail if there are any extensions other than **pg\$1transport** installed on the source database. This command must by run by a user with the `rds_superuser` role. | DBA |
### Perform the transport
| Task | Description | Skills required |
| --- | --- | --- |
| Perform a dry run. | Use the `transport.import_from_server` function to perform a dry run first:
The last parameter of this function (set to `true`) defines the dry run. This function displays any errors that you would see when you run the main transport. Resolve the errors before you run the main transport. | DBA |
| If the dry run is successful, initiate the database transport. | Run the `transport.import_from_server` function to perform the transport. It connects to the source and imports the data.
The last parameter of this function (set to `false`) indicates that this isn’t a dry run. | DBA |
| Perform post-transport steps. | After the database transport is complete:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.html) | DBA |
## Related resources
+ [Amazon RDS documentation](https://docs.aws.amazon.com/rds/)
+ [pg\$1transport documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Procedural.Importing.html#PostgreSQL.TransportableDB.Setup)
+ [Migrating databases using RDS PostgreSQL Transportable Databases](https://aws.amazon.com/blogs/database/migrating-databases-using-rds-postgresql-transportable-databases/) (blog post)
+ [PostgreSQL downloads](https://www.postgresql.org/download/linux/redhat/)
+ [psql utility](https://www.postgresql.org/docs/11/app-psql.html)
+ [Creating a DB Parameter Group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html#USER_WorkingWithParamGroups.Creating)
+ [Modify Parameters in a DB Parameter Group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html#USER_WorkingWithParamGroups.Modifying)
+ [PostgreSQL downloads](https://www.postgresql.org/download/)
# Replatform
**Topics**
+ [Export a Microsoft SQL Server database to Amazon S3 by using AWS DMS](export-a-microsoft-sql-server-database-to-amazon-s3-by-using-aws-dms.md)
+ [Migrate Oracle CLOB values to individual rows in PostgreSQL on AWS](migrate-oracle-clob-values-to-individual-rows-in-postgresql-on-aws.md)
+ [Migrate Oracle PeopleSoft to Amazon RDS Custom](migrate-oracle-peoplesoft-to-amazon-rds-custom.md)
+ [Migrate Oracle ROWID functionality to PostgreSQL on AWS](migrate-oracle-rowid-functionality-to-postgresql-on-aws.md)
+ [Migrate Oracle Database error codes to an Amazon Aurora PostgreSQL-Compatible database](migrate-oracle-database-error-codes-to-an-amazon-aurora-postgresql-compatible-database.md)
+ [Migrate SAP ASE on Amazon EC2 to Amazon Aurora PostgreSQL-Compatible using AWS SCT and AWS DMS](migrate-sap-ase-on-amazon-ec2-to-amazon-aurora-postgresql-compatible-using-aws-sct-and-aws-dms.md)
+ [Migrate Windows SSL certificates to an Application Load Balancer using ACM](migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.md)
+ [Migrate a messaging queue from Microsoft Azure Service Bus to Amazon SQS](migrate-a-messaging-queue-from-microsoft-azure-service-bus-to-amazon-sqs.md)
+ [Migrate an Oracle JD Edwards EnterpriseOne database to AWS by using Oracle Data Pump and AWS DMS](migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.md)
+ [Migrate an Oracle PeopleSoft database to AWS by using AWS DMS](migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.md)
+ [Migrate an on-premises MySQL database to Amazon RDS for MySQL](migrate-an-on-premises-mysql-database-to-amazon-rds-for-mysql.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server.md)
+ [Migrate data from Microsoft Azure Blob to Amazon S3 by using Rclone](migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone.md)
+ [Migrate from Couchbase Server to Couchbase Capella on AWS](migrate-from-couchbase-server-to-couchbase-capella-on-aws.md)
+ [Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2](migrate-from-ibm-websphere-application-server-to-apache-tomcat-on-amazon-ec2.md)
+ [Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2 with Auto Scaling](migrate-from-ibm-websphere-application-server-to-apache-tomcat-on-amazon-ec2-with-auto-scaling.md)
+ [Migrate a .NET application from Microsoft Azure App Service to AWS Elastic Beanstalk](migrate-a-net-application-from-microsoft-azure-app-service-to-aws-elastic-beanstalk.md)
+ [Migrate from Oracle WebLogic to Apache Tomcat (TomEE) on Amazon ECS](migrate-from-oracle-weblogic-to-apache-tomcat-tomee-on-amazon-ecs.md)
+ [Migrate an Oracle database from Amazon EC2 to Amazon RDS for Oracle using AWS DMS](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-oracle-using-aws-dms.md)
+ [Migrate an on-premises Oracle database to Amazon OpenSearch Service using Logstash](migrate-an-on-premises-oracle-database-to-amazon-opensearch-service-using-logstash.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for Oracle](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-using-oracle-data-pump.md)
+ [Migrate from PostgreSQL on Amazon EC2 to Amazon RDS for PostgreSQL using pglogical](migrate-from-postgresql-on-amazon-ec2-to-amazon-rds-for-postgresql-using-pglogical.md)
+ [Migrate an on-premises PostgreSQL database to Aurora PostgreSQL](migrate-an-on-premises-postgresql-database-to-aurora-postgresql.md)
+ [Migrate an on-premises Microsoft SQL Server database to Microsoft SQL Server on Amazon EC2 running Linux](migrate-an-on-premises-microsoft-sql-server-database-to-microsoft-sql-server-on-amazon-ec2-running-linux.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using linked servers](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-linked-servers.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using native backup and restore methods](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-native-backup-and-restore-methods.md)
+ [Migrate a Microsoft SQL Server database to Aurora MySQL by using AWS DMS and AWS SCT](migrate-a-microsoft-sql-server-database-to-aurora-mysql-by-using-aws-dms-and-aws-sct.md)
+ [Migrate an on-premises MariaDB database to Amazon RDS for MariaDB using native tools](migrate-an-on-premises-mariadb-database-to-amazon-rds-for-mariadb-using-native-tools.md)
+ [Migrate an on-premises MySQL database to Aurora MySQL](migrate-an-on-premises-mysql-database-to-aurora-mysql.md)
+ [Migrate on-premises MySQL databases to Aurora MySQL using Percona XtraBackup, Amazon EFS, and Amazon S3](migrate-on-premises-mysql-databases-to-aurora-mysql-using-percona-xtrabackup-amazon-efs-and-amazon-s3.md)
+ [Migrate on-premises Java applications to AWS using AWS App2Container](migrate-on-premises-java-applications-to-aws-using-aws-app2container.md)
+ [Migrate shared file systems in an AWS large migration](migrate-shared-file-systems-in-an-aws-large-migration.md)
+ [Migrate an Oracle database to Amazon RDS for Oracle by using Oracle GoldenGate flat file adapters](migrate-an-oracle-database-to-amazon-rds-for-oracle-by-using-oracle-goldengate-flat-file-adapters.md)
+ [Change Python and Perl applications to support database migration from Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible Edition](change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.md)
# Export a Microsoft SQL Server database to Amazon S3 by using AWS DMS
*Sweta Krishna, Amazon Web Services*
## Summary
Organizations often need to copy databases to Amazon Simple Storage Service (Amazon S3) for database migration, backup and restore, data archiving, and data analytics. This pattern describes how you can export a Microsoft SQL Server database to Amazon S3. The source database can be hosted on premises or on Amazon Elastic Compute Cloud (Amazon EC2) or Amazon Relational Database Service (Amazon RDS) for Microsoft SQL Server on the Amazon Web Services (AWS) Cloud.
The data is exported by using AWS Database Migration Service (AWS DMS). By default, AWS DMS writes full load and change data capture (CDC) data in comma-separated value (.csv) format. For more compact storage and faster query options, this pattern uses the Apache Parquet (.parquet) format option.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An AWS Identity and Access Management (IAM) role for the account with write, delete, and tag access to the target S3 bucket, and AWS DMS (`dms.amazonaws.com`) added as a trusted entity to this IAM role
+ An on-premises Microsoft SQL Server database (or Microsoft SQL Server on an EC2 instance or an Amazon RDS for SQL Server database)
+ Network connectivity between the virtual private cloud (VPC) on AWS and the on-premises network provided by AWS Direct Connect or a virtual private network (VPN)
**Limitations**
+ A VPC-enabled (gateway VPC) S3 bucket isn't currently supported in AWS DMS versions earlier than 3.4.7.
+ Changes to the source table structure during full load are not supported.
+ AWS DMS full large binary object (LOB) mode is not supported.
**Product versions**
+ Microsoft SQL Server versions 2005 or later for the Enterprise, Standard, Workgroup, and Developer editions.
+ Support for Microsoft SQL Server version 2019 as a source is available in AWS DMS versions 3.3.2 and later.
## Architecture
**Source technology stack **
+ An on-premises Microsoft SQL Server database (or Microsoft SQL Server on an EC2 instance or an Amazon RDS for SQL Server database)** **
**Target technology stack **
+ AWS Direct Connect
+ AWS DMS
+ Amazon S3
**Target architecture **
![\[Data migrates from SQL Server database through Direct Connect into AWS DMS and then to S3 bucket.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/75b8b20f-a1a8-4633-9816-1b370cc7e92c/images/85bd433c-4a0a-4825-8661-e53f53265191.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the database version. | Validate the source database version and make sure that it’s supported by AWS DMS. For information about supported SQL Server database versions, see [Using a Microsoft SQL Server database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html). | DBA |
| Create a VPC and security group. | In your AWS account, create a VPC and security group. For more information, see the [Amazon VPC documentation](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html). | System Administrator |
| Create a user for the AWS DMS task. | Create an AWS DMS user in the source database and grant it READ permissions. This user will be used by AWS DMS. | DBA |
| Test the DB connectivity. | Test the connectivity to the SQL Server DB instance from the AWS DMS user. | DBA |
| Create an S3 bucket. | Create the target S3 bucket. This bucket will hold the migrated table data. | Systems administrator |
| Create an IAM policy and role. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/export-a-microsoft-sql-server-database-to-amazon-s3-by-using-aws-dms.html) | Systems administrator |
### Migrate data by using AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS DMS replication instance. | Sign in to the AWS Management Console, and open the AWS DMS console. In the navigation pane, choose **Replication instances**, **Create replication instance**. For instructions, see [step 1](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Replication.html#CHAP_GettingStarted.Replication.ReplicationInstance) in the AWS DMS documentation. | DBA |
| Create source and target endpoints. | Create source and target endpoints. Test the connection from the replication instance to both source and target endpoints. For instructions, see [step 2](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Replication.html#CHAP_GettingStarted.Replication.Endpoints) in the AWS DMS documentation. | DBA |
| Create a replication task. | Create a replication task, and select full load or full load with change data capture (CDC) to migrate data from SQL Server to the S3 bucket. For instructions, see [step 3](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Replication.html#CHAP_GettingStarted.Replication.Tasks) in the AWS DMS documentation. | DBA |
| Start the data replication. | Start the replication task, and monitor the logs for any errors. | DBA |
### Validate the data
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the migrated data. | On the console, navigate to your target S3 bucket. Open the subfolder that has the same name as the source database. Confirm that the folder contains all the tables that were migrated from the source database. | DBA |
### Clean up resources
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down and delete temporary AWS resources. | Shut down temporary AWS resources that you created for the data migration, such as the AWS DMS replication instance, and delete them after you validate the export. | DBA |
## Related resources
+ [AWS Database Migration Service User Guide](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [Using a Microsoft SQL Server database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html)
+ [Using Amazon S3 as a target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html)
+ [Using an S3 bucket as an AWS DMS target](https://repost.aws/knowledge-center/s3-bucket-dms-target) (AWS re:Post)
## Additional information
Use the following code to add an IAM policy with S3 bucket permissions for the AWS DMS role. Replace `bucketname` with the name of your bucket.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucketname*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucketname*"
]
}
]
}
```
# Migrate Oracle CLOB values to individual rows in PostgreSQL on AWS
*Sai Krishna Namburu and Sindhusha Paturu, Amazon Web Services*
## Summary
This pattern describes how to split Oracle character large object (CLOB) values into individual rows in Amazon Aurora PostgreSQL-Compatible Edition and Amazon Relational Database Service (Amazon RDS) for PostgreSQL. PostgreSQL doesn’t support the CLOB data type.
Tables with interval partitions are identified in the source Oracle database, and the table name, the type of partition, the interval of the partition, and other metadata are captured and loaded into the target database. You can load CLOB data that is less than 1 GB in size into target tables as text by using AWS Database Migration Service (AWS DMS), or you can export the data in CSV format, load it into an Amazon Simple Storage Service (Amazon S3) bucket, and migrate it to your target PostgreSQL database.
After migration, you can use the custom PostgreSQL code that is provided with this pattern to split the CLOB data into individual rows based on the new line character identifier (`CHR(10)`) and populate the target** **table.
## Prerequisites and limitations
**Prerequisites **
+ An Oracle database table that has interval partitions and records with a CLOB data type.
+ An Aurora PostgreSQL-Compatible or Amazon RDS for PostgreSQL database that has a table structure that’s similar to the source table (same columns and data types).
**Limitations **
+ The CLOB value cannot exceed 1 GB.
+ Each row in the target table must have a new line character identifier.
**Product versions**
+ Oracle 12c
+ Aurora Postgres 11.6
## Architecture
The following diagram shows a source Oracle table with CLOB data, and the equivalent PostgreSQL table in Aurora PostgreSQL-Compatible version 11.6.
![\[Source CLOB table and equivalent target PostgreSQL table.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/55806ee7-6a9f-4058-9a47-a07de68223ca/images/79b9d4b9-6f20-4db5-8ca8-2a599769a498.png)
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [Amazon Relational Database Service (Amazon RDS) for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Other tools**
You can use the following client tools to connect to, access, and manage your Aurora PostgreSQL-Compatible and Amazon RDS for PostgreSQL databases. (These tools aren’t used within this pattern.)
+ [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
+ [DBeaver](https://dbeaver.io/) is an open-source database tool for developers and database administrators. You can use the tool to manipulate, monitor, analyze, administer, and migrate your data.
## Best practices
For best practices for migrating your database from Oracle to PostgreSQL, see the AWS blog post [Best practices for migrating an Oracle database to Amazon RDS PostgreSQL or Amazon Aurora PostgreSQL: Migration process and infrastructure considerations](https://aws.amazon.com/blogs/database/best-practices-for-migrating-an-oracle-database-to-amazon-rds-postgresql-or-amazon-aurora-postgresql-migration-process-and-infrastructure-considerations/).
For best practices for configuring the AWS DMS task for migrating large binary objects, see [Migrating large binary objects (LOBs)](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html#CHAP_BestPractices.LOBS) in the AWS DMS documentation.
## Epics
### Identify the CLOB data
| Task | Description | Skills required |
| --- | --- | --- |
| Analyze the CLOB data. | In the source Oracle database, analyze the CLOB data to see whether it contains column headers, so you can determine the method for loading the data into the target table. To analyze the input data, use the following query.`SELECT * FROM clobdata_or; ` | Developer |
| Load the CLOB data to the target database. | Migrate the table that has CLOB data to an interim (staging) table in the Aurora or Amazon RDS target database. You can use AWS DMS or upload the data as a CSV file to an Amazon S3 bucket.For information about using AWS DMS for this task, see [Using an Oracle database as a source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and [Using a PostgreSQL database as a target](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html) in the AWS DMS documentation.For information about using Amazon S3 for this task, see [Using Amazon S3 as a target](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html) in the AWS DMS documentation. | Migration engineer, DBA |
| Validate the target PostgreSQL table. | Validate the target data, including headers, against the source data by using the following queries in the target database.
SELECT * FROM clobdata_pg; SELECT * FROM clobdatatarget;
Compare the results against the query results from the source database (from the first step). | Developer |
| Split the CLOB data into separate rows. | Run the custom PostgreSQL code provided in the [Additional information](#migrate-oracle-clob-values-to-individual-rows-in-postgresql-on-aws-additional) section to split the CLOB data and insert it into separate rows in the target PostgreSQL table. | Developer |
### Validate the data.
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the data in the target table. | Validate the data inserted into the target table by using the following queries.
SELECT * FROM clobdata_pg; SELECT * FROM clobdatatarget;
| Developer |
## Related resources
+ [CLOB data type](https://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF0021) (Oracle documentation)
+ [Data types](https://www.postgresql.org/docs/11/datatype.html) (PostgreSQL documentation)
## Additional information
**PostgreSQL function for splitting CLOB data**
```
do
$$
declare
totalstr varchar;
str1 varchar;
str2 varchar;
pos1 integer := 1;
pos2 integer ;
len integer;
begin
select rawdata||chr(10) into totalstr from clobdata_pg;
len := length(totalstr) ;
raise notice 'Total length : %',len;
raise notice 'totalstr : %',totalstr;
raise notice 'Before while loop';
while pos1 < len loop
select position (chr(10) in totalstr) into pos2;
raise notice '1st position of new line : %',pos2;
str1 := substring (totalstr,pos1,pos2-1);
raise notice 'str1 : %',str1;
insert into clobdatatarget(data) values (str1);
totalstr := substring(totalstr,pos2+1,len);
raise notice 'new totalstr :%',totalstr;
len := length(totalstr) ;
end loop;
end
$$
LANGUAGE 'plpgsql' ;
```
**Input and output examples**
You can use the following examples to try out the PostgreSQL code before you migrate your data.
Create an Oracle database with three input lines.
```
CREATE TABLE clobdata_or (
id INTEGER GENERATED ALWAYS AS IDENTITY,
rawdata clob );
insert into clobdata_or(rawdata) values (to_clob('test line 1') || chr(10) || to_clob('test line 2') || chr(10) || to_clob('test line 3') || chr(10));
COMMIT;
SELECT * FROM clobdata_or;
```
This displays the following output.
| | |
| --- |--- |
| id | rawdata |
| 1 | test line 1 test line 2 test line 3 |
Load the source data into a PostgreSQL staging table (`clobdata_pg`) for processing.
```
SELECT * FROM clobdata_pg;
CREATE TEMP TABLE clobdatatarget (id1 SERIAL,data VARCHAR );
SELECT * FROM clobdatatarget;
```
This displays the following output.
| | |
| --- |--- |
| id1 | data |
| 1 | test line 1 |
| 2 | test line 2 |
| 3 | test line 3 |
# Migrate Oracle PeopleSoft to Amazon RDS Custom
*Gaurav Gupta, Amazon Web Services*
## Summary
[Oracle PeopleSoft](https://www.oracle.com/applications/peoplesoft/) is an enterprise resource planning (ERP) solution for enterprise-wide processes. PeopleSoft has a three-tier architecture: client, application, and database. PeopleSoft can be run on [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html). Now, you can also run PeopleSoft on [Amazon RDS Custom](https://aws.amazon.com/rds/custom/), which provides access to the underlying operating system.
[Amazon RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/working-with-custom-oracle.html) is a managed database service for legacy, custom, and packaged applications that require access to the underlying operating system and database environment. When you migrate your Oracle database to Amazon RDS Custom, Amazon Web Services (AWS) can manage backup tasks and high availability, while you can focus on maintaining your PeopleSoft application and functionality. For key factors to consider for a migration, see [Oracle database migration strategies](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/strategies.html) in AWS Prescriptive Guidance.
This pattern focuses on the steps to migrate a PeopleSoft database on Amazon Elastic Compute Cloud (Amazon EC2) to Amazon RDS Custom by using an Oracle Recovery Manager (RMAN) backup. It uses an [Amazon Elastic File System (Amazon EFS)](https://aws.amazon.com/efs/) shared file system between the EC2 instance and Amazon RDS Custom, although you can also use Amazon FSx or any shared drive. The pattern uses an RMAN full backup (sometimes referred to as a level 0 backup).
## Prerequisites and limitations
**Prerequisites**
+ An Oracle version 19C source database that is running on Amazon EC2 with Oracle Linux 7, Oracle Linux 8, Red Hat Enterprise Linux (RHEL) 7, or RHEL 8. In the examples for this pattern, the source database name is `FSDMO92`, but this isn’t a requirement.
**Note**
You can also use this pattern with on-premises Oracle source databases. You must have the appropriate network connectivity between the on-premises network and a virtual private cloud (VPC).
+ A PeopleSoft 9.2 demo instance.
+ A single PeopleSoft application tier. However, you can adapt this pattern to work with multiple application tiers.
+ Amazon RDS Custom configured with at least 8 GB of swap space.
**Limitations **
This pattern doesn’t support the following configurations:
+ Setting the database `ARCHIVE_LAG_TARGET` parameter to a value outside the 60–7200 range
+ Disabling the DB instance log mode (`NOARCHIVELOG`)
+ Turning off the Amazon Elastic Block Store (Amazon EBS) optimized attribute of the EC2 instance
+ Modifying the original EBS volumes attached to the EC2 instance
+ Adding new EBS volumes or changing the volume type from gp2 to gp3
+ Changing the extension format for the `LOG_ARCHIVE_FORMAT` parameter (requires `*.arc`)
+ Multiplexing or changing the control file location and name (it has to be `/rdsdbdata/db/*DBNAME*/controlfile/control-01.ctl`)
For additional information about these and other unsupported configurations, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html#custom-troubleshooting.fix-unsupported).
**Product versions**
For Oracle Database versions and instance classes supported by Amazon RDS Custom, see [Requirements and limitations for Amazon RDS Custom for Oracle.](https://docs.amazonaws.cn/en_us/AmazonRDS/latest/UserGuide/custom-reqs-limits.html)
## Architecture
**Target technology stack**
+ Application Load Balancer
+ Amazon EFS
+ Amazon RDS Custom for Oracle
+ AWS Secrets Manager
+ Amazon Simple Storage Service (Amazon S3)
**Target architecture**
The following architecture diagram represents a PeopleSoft system running in a single [Availability Zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) on AWS. The application tier is accessed through an [Application Load Balancer](https://aws.amazon.com/elasticloadbalancing/application-load-balancer/). Both the application and the databases are in private subnets, and the Amazon RDS Custom and Amazon EC2 database instance use an Amazon EFS shared file system to store and access the RMAN backup files. Amazon S3 is used for creating the custom RDS Oracle engine and for storing the redo logs metadata.
![\[Webservers, app servers, Amazon RDS Custom, an EC2 DB instance, and Amazon EFS in private subnets.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bd423dfe-f3c8-42d9-ac84-bf3d093c52bc/images/0e9a6431-e6c7-4047-ae6c-85311938041f.jpeg)
## Tools
**Tools**
*AWS services*
+ [Amazon RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/working-with-custom-oracle.html) is a managed database service for legacy, custom, and packaged applications that require access to the underlying operating system and database environment. It automates database administration tasks, such as backups and high availability.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud. This pattern uses an Amazon EFS shared file system to store and access the RMAN backup files.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically. In this pattern, you retrieve the database user passwords from Secrets Manager to create the `RDSADMIN` and `ADMIN` users and to change the `sys` and `system` passwords.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon Elastic Compute Cloud (Amazon EC2) instances, containers, and IP addresses in one or more Availability Zones. This pattern uses an Application Load Balancer.
*Other tools*
+ Oracle Recovery Manager (RMAN) provides backup and recovery support for Oracle databases. This pattern uses RMAN to perform a hot backup of the source Oracle database on Amazon EC2 that is restored on Amazon RDS Custom.
## Best practices
+ For database initialization parameters, customize the standard pfile that’s provided by the Amazon RDS Custom DB instance for PeopleSoft instead of using the spfile from the Oracle source database. This is because white spaces and comments cause issues when creating read replicas in Amazon RDS Custom. For more information about database initialization parameters, see Oracle Support Note 1100831.1 (requires an [Oracle Support](https://support.oracle.com/portal/) account).
+ Amazon RDS Custom uses Oracle automatic memory management by default. If you want to use the Hugemem kernel, you can configure Amazon RDS Custom to use automatic shared memory management instead.
+ Leave the `memory_max_target` parameter enabled by default. The framework uses this in the background to create read replicas.
+ Enable Oracle Flashback Database. This feature is useful when reinstating the standby in failover (not switchover) testing scenarios.
## Epics
### Set up the DB instance and file system
| Task | Description | Skills required |
| --- | --- | --- |
| Create the DB instance. | In the Amazon RDS console, create an Amazon RDS Custom for Oracle DB instance with a DB name called FSDMO92 (or your source database name).For instructions, see [Working with Amazon RDS Custom](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-custom.html) in the AWS documentation and the [Amazon RDS Custom for Oracle – New Control Capabilities in Database Environment](https://aws.amazon.com/blogs/aws/amazon-rds-custom-for-oracle-new-control-capabilities-in-database-environment/) blog post. This ensures that the database name is set to the same name as the source database. (If kept blank, the EC2 instance and database name will be set to `ORCL`.) | DBA |
### Perform an RMAN full backup of the source Amazon EC2 database
| Task | Description | Skills required |
| --- | --- | --- |
| Create a backup script. | Create an RMAN backup script to back up the database to the Amazon EFS file system that you mounted (`/efs` in the following example). You can use the example code or run one of your existing RMAN scripts.
#!/bin/bash Dt=`date +'%Y%m%d-%H%M'` BACKUP_LOG="rman-${ORACLE_SID}-$Dt" export TAGDATE=`date +%Y%m%d%H%M`; LOGPATH=/u01/scripts/logs rman target / >> $LOGPATH/rman-${ORACLE_SID}-$Dt << EOF SQL "ALTER SYSTEM SWITCH LOGFILE"; SQL "ALTER SESSION SET NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS""; RUN { ALLOCATE CHANNEL ch11 TYPE DISK MAXPIECESIZE 5G; ALLOCATE CHANNEL ch12 TYPE DISK MAXPIECESIZE 5G; BACKUP AS COMPRESSED BACKUPSET FULL DATABASE FORMAT '/efs/rman_backup/FSCM/%d_%T_%s_%p_FULL' ; SQL "ALTER SYSTEM ARCHIVE LOG CURRENT"; BACKUP FORMAT '/efs/rman_backup/FSCM/%d_%T_%s_%p_ARCHIVE' ARCHIVELOG ALL DELETE ALL INPUT ; BACKUP CURRENT CONTROLFILE FORMAT '/efs/rman_backup/FSCM/%d_%T_%s_%p_CONTROL'; } EXIT; EOF
| DBA |
| Run the backup script. | To run the RMAN backup script, log in as the Oracle Home User, and run the script.
$ chmod a+x rman_backup.sh $ ./rman_backup.sh &
| DBA |
| Check for errors and note the name of the backup file. | Check the RMAN log file for errors. If everything looks fine, list the backup of the control file by running the following command.
RMAN> list backup of controlfile;
using target database control file instead of recovery catalog
Note the name of the output file.
List of Backup Sets ===================
BS Key Type LV Size Device Type Elapsed Time Completion Time ------- ---- -- ---------- ----------- ------------ --------------- 12 Full 21.58M DISK 00:00:01 13-JUL-22 BP Key: 12 Status: AVAILABLE Compressed: NO Tag: TAG20220713T150155 Piece Name: /efs/rman_backup/FSCM/FSDMO92_20220713_12_1_CONTROL Control File Included: Ckp SCN: 16559159985898 Ckp time: 13-JUL-22
You will use the backup control file `/efs/rman_backup/FSCM/FSDMO92_20220713_12_1_CONTROL` when you restore the database on Amazon RDS Custom. | DBA |
### Shut down the source application tier
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the application. | To shut down the source application tier, use the `psadmin` utility or the `psadmin` command line utility.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA, PeopleSoft Administrator |
### Configure the target Amazon RDS Custom database
| Task | Description | Skills required |
| --- | --- | --- |
| Install the nfs-utils rpm package. | To install the `nfs-utils rpm` package, run the following command.
$ yum install -y nfs-utils
| DBA |
| Mount the EFS storage. | Get the Amazon EFS mount command from the Amazon EFS console page. Mount the EFS file system on the Amazon RDS instance by using a Network File System (NFS) client.
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-xxxxxxxxxx.efs.eu-west-1.amazonaws.com:/ /efs sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-xxxxxxxxxx.efs.eu-west-1.amazonaws.com:/ /efs
| DBA |
### Drop the starter database and create the directories to store the database files
| Task | Description | Skills required |
| --- | --- | --- |
| Pause automation mode. | You have to pause [automation mode](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-managing.html#custom-managing.pausing) on your Amazon RDS Custom DB instance before you proceed with the next steps, to make sure that automation doesn’t interfere with the RMAN restore activity.You can pause the automation by using the AWS console or the AWS Command Line Interface (AWS CLI) command (make sure that you have [configured the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) first).
When you specify the duration of the pause, make sure that you leave enough time for the RMAN restore. This depends on the size of the source database, so modify the 360 value accordingly.Also, make sure that the total time of the paused automation does not overlap with the backup or maintenance window of the database. | DBA |
| Create and modify the parameter file for PeopleSoft | To create and modify the pfile for PeopleSoft, use the standard pfile created with the Amazon RDS Custom DB instance. Add the parameters you need for PeopleSoft.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA |
| Drop the starter database. | To drop the existing Amazon RDS Custom database, use the following code.
$ sqlplus / as sysdba SQL> shutdown immediate; SQL> startup mount exclusive restrict; SQL> drop database; SQL> exit
| |
| Restore the Amazon RDS Custom database from the backup. | Restore the database by using the following script. The script will first restore the control file and then restore the entire database from the backup pieces stored on the EFS mount.
#!/bin/bash Dt=`date +'%Y%m%d-%H%M'` BACKUP_LOG="rman-${ORACLE_SID}-$Dt" export TAGDATE=`date +%Y%m%d%H%M`; LOGPATH=/rdsdbdata/scripts/logs rman target / >> $LOGPATH/rman-${ORACLE_SID}-$Dt << EOF restore controlfile from "/efs/rman_backup/FSCM/FSDMO92_20220713_12_1_CONTROL"; alter database mount; run { set newname for database to '/rdsdbdata/db/FSDMO92_A/datafile/%f_%b'; SET NEWNAME FOR TEMPFILE 1 TO '/rdsdbdata/db/FSDMO92_A/datafile/%f_%b'; RESTORE DATABASE; SWITCH DATAFILE ALL; SWITCH TEMPFILE ALL; RECOVER DATABASE; } EOF sqlplus / as sysdba >> $LOGPATH/rman-${ORACLE_SID}-$Dt<<-EOF ALTER DATABASE RENAME FILE '/u01/psoft/db/oradata/FSDMO92/redo01.log' TO '/rdsdbdata/db/FSDMO92_A/onlinelog/redo01.log'; ALTER DATABASE RENAME FILE '/u01/psoft/db/oradata/FSDMO92/redo02.log' TO '/rdsdbdata/db/FSDMO92_A/onlinelog/redo02.log'; ALTER DATABASE RENAME FILE '/u01/psoft/db/oradata/FSDMO92/redo03.log' TO '/rdsdbdata/db/FSDMO92_A/onlinelog/redo03.log'; alter database clear unarchived logfile group 1; alter database clear unarchived logfile group 2; alter database clear unarchived logfile group 3; alter database open resetlogs; EXIT EOF
| DBA |
### Retrieve passwords from Secrets Manager, create users, and change passwords
| Task | Description | Skills required |
| --- | --- | --- |
| Retrieve the password from Secrets Manager. | You can perform this step by using the AWS console or the AWS CLI. The following steps show instructions for the console.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA |
| Create the RDSADMIN user. | `RDSADMIN` is the database user for monitoring and orchestrating the Amazon RDS Custom DB instance. Because the starter database was dropped and the target database was restored from the source using RMAN, you must recreate this user after the restore operation to make sure that Amazon RDS Custom monitoring works as expected. You also must create a separate profile and tablespace for the `RDSADMIN` user.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA |
| Create the master user. | Because the starter database was dropped and the target database was restored from the source by using RMAN, you must recreate the master user. In this example, the master user name is `admin`.
SQL> create user admin identified by ; SQL> grant dba to admin
| DBA |
| Change the system passwords. | Change the system passwords by using the password you retrieved from Secrets Manager.
SQL> alter user sys identified by xxxxxxxxxxx; SQL> alter user system identified by xxxxxxxxxx;
If you don’t change these passwords, Amazon RDS Custom displays the error message, "The database monitoring user or user credentials have changed." | DBA |
### Configure the TNS entries for Amazon RDS Custom and PeopleSoft
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the tnsnames file. | To connect to the database from the application tier, configure the `tnsnames.ora` file so you can connect to the database from the application tier. In the following example, you can see that there is a soft link to the `tnsnames.ora` file, but the file is empty by default.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA |
### Create the spfile softlink
| Task | Description | Skills required |
| --- | --- | --- |
| Create the spfile softlink. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA |
### Perform post-migration steps
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the schema, connections, and maintenance tasks. | To finalize the migration, perform the following tasks.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-peoplesoft-to-amazon-rds-custom.html) | DBA |
## Related resources
+ [Working with Amazon RDS Custom](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-custom.html)
+ [Amazon RDS Custom for Oracle – New Control Capabilities in Database Environment](https://aws.amazon.com/blogs/aws/amazon-rds-custom-for-oracle-new-control-capabilities-in-database-environment/) (blog post)
+ [Integrate Amazon RDS Custom for Oracle with Amazon EFS](https://aws.amazon.com/blogs/database/integrate-amazon-rds-custom-for-oracle-with-amazon-efs/) (blog post)
+ [Configuring Amazon RDS as an Oracle PeopleSoft Database](https://d1.awsstatic.com/whitepapers/configuring-amazon-rds-as-peoplesoft-database.pdf) (AWS whitepaper)
# Migrate Oracle ROWID functionality to PostgreSQL on AWS
*Rakesh Raghav and Ramesh Pathuri, Amazon Web Services*
## Summary
This pattern describes options for migrating the `ROWID` pseudocolumn functionality in Oracle Database to a PostgreSQL database in Amazon Relational Database Service (Amazon RDS) for PostgreSQL, Amazon Aurora PostgreSQL-Compatible Edition, or Amazon Elastic Compute Cloud (Amazon EC2).
In an Oracle database, the `ROWID` pseudocolumn is a physical address of a row in a table. This pseudocolumn is used to uniquely identify a row even if the primary key isn’t present on a table. PostgreSQL has a similar pseudocolumn called `ctid`, but it cannot be used as a `ROWID`. As explained in the [PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-system-columns.html), `ctid` might change if it’s updated or after every `VACUUM` process.
There are three ways you can create the `ROWID` pseudocolumn functionality in PostgreSQL:
+ Use a primary key column instead of `ROWID` to identify a row in a table.
+ Use a logical primary/unique key (which might be a composite key) in the table.
+ Add a column with auto-generated values and make it a primary/unique key to mimic `ROWID`.
This pattern walks you through all three implementations and describes the advantages and disadvantages of each option.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Procedural Language/PostgreSQL (PL/pgSQL) coding expertise
+ Source Oracle Database
+ An Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible cluster, or an EC2 instance to host the PostgreSQL database
**Limitations **
+ This pattern provides workarounds for the `ROWID` functionality. PostgreSQL doesn’t provide an equivalent to `ROWID` in Oracle Database.
**Product versions**
+ PostgreSQL 11.9 or later
## Architecture
**Source technology stack **
+ Oracle Database
**Target technology stack **
+ Aurora PostgreSQL-Compatible, Amazon RDS for PostgreSQL, or an EC2 instance with a PostgreSQL database
![\[Converting an Oracle Database to PostgreSQL on AWS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9a2ce994-4f68-4975-aab2-796cc20a3c82/images/6e7c2ef6-f440-476a-9003-f1f166718e15.png)
**Implementation options**
There are three options to work around the lack of `ROWID` support in PostgreSQL, depending on whether your table has a primary key or unique index, a logical primary key, or an identity attribute. Your choice depends on your project timelines, your current migration phase, and dependencies on application and database code.
|
|
| Option | Description | Advantages | Disadvantages |
| --- |--- |--- |--- |
| **Primary key or unique index** | If your Oracle table has a primary key, you can use the attributes of this key to uniquely identify a row. | No dependency on proprietary database features.Minimal impact on performance, because primary key fields are indexed. | Requires changes to application and database code that relies on `ROWID` to switch to primary key fields. |
| **Logical primary/unique key** | If your Oracle table has a logical primary key, you can use the attributes of this key to uniquely identify a row. A logical primary key consists of an attribute or a set of attributes that can uniquely identify a row, but it isn’t enforced on the database through a constraint. | No dependency on proprietary database features. | Requires changes to application and database code that relies on `ROWID` to switch to primary key fields.Significant impact on performance if the attributes of the logical primary key aren’t indexed. However, you can add a unique index to prevent performance issues. |
| **Identity attribute** | if your Oracle table doesn't have a primary key, you can create an additional field as `GENERATED ALWAYS AS IDENTITY`. This attribute generates a unique value whenever data is inserted into the table, so it can be used to uniquely identify a row for Data Manipulation Language (DML) operations. | No dependency on proprietary database features.PostgreSQL database populates the attribute and maintains its uniqueness. | Requires changes to application and database code that relies on `ROWID` to switch to identity attribute.Significant impact on performance if the additional field isn’t indexed. However, you can add an index to prevent performance issues. |
## Tools
+ [Amazon Relational Database Service (Amazon RDS) for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell. In this pattern, you can use the AWS CLI to run SQL commands through **pgAdmin**.
+ [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
## Epics
### Identify the source tables
| Task | Description | Skills required |
| --- | --- | --- |
| Identify Oracle tables that use the `ROWID` attribute. | Use the AWS Schema Conversion Tool (AWS SCT) to identify Oracle tables that have `ROWID` functionality. For more information, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.ToPostgreSQL.html#CHAP_Source.Oracle.ToPostgreSQL.ConvertRowID).—or—In Oracle, use the `DBA_TAB_COLUMNS` view to identify tables that have a `ROWID` attribute. These fields might be used to store alphanumeric 10-byte characters. Determine the usage and convert these to a `VARCHAR` field if appropriate. | DBA or developer |
| Identify code that references these tables. | Use AWS SCT to generate a migration assessment report to identify procedures affected by `ROWID`. For more information, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_AssessmentReport.html).—or—In the source Oracle database, use the text field of the `dba_source` table to identify objects that use `ROWID` functionality. | DBA or developer |
### Determine primary key usage
| Task | Description | Skills required |
| --- | --- | --- |
| Identify tables that don’t have primary keys. | In the source Oracle database, use `DBA_CONSTRAINTS` to identify tables that don’t have primary keys. This information will help you determine the strategy for each table. For example:
select dt.* from dba_tables dt where not exists (select 1 from all_constraints ct where ct.owner = Dt.owner and ct.table_name = Dt.table_name and ct.constraint_type = 'P' ) and dt.owner = '{schema}'
| DBA or developer |
### Identify and apply the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Apply changes for tables that have a defined or logical primary key. | Make the application and database code changes shown in the [Additional information](#migrate-oracle-rowid-functionality-to-postgresql-on-aws-additional) section to use a unique primary key or a logical primary key to identify a row in your table. | DBA or developer |
| Add an additional field to tables that don’t have a defined or logical primary key. | Add an attribute of type `GENERATED ALWAYS AS IDENTITY`. Make the application and database code changes shown in the [Additional information](#migrate-oracle-rowid-functionality-to-postgresql-on-aws-additional) section. | DBA or developer |
| Add an index if necessary. | Add an index to the additional field or logical primary key to improve SQL performance. | DBA or developer |
## Related resources
+ [PostgreSQL CTID](https://www.postgresql.org/docs/current/ddl-system-columns.html) (PostgreSQL documentation)
+ [Generated Columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html) (PostgreSQL documentation)
+ [ROWID Pseudocolumn](https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/ROWID-Pseudocolumn.html#GUID-F6E0FBD2-983C-495D-9856-5E113A17FAF1) (Oracle documentation)
## Additional information
The following sections provide Oracle and PostgreSQL code examples to illustrate the three approaches.
**Scenario 1: Using a primary unique key**
In the following examples, you create the table `testrowid_s1` with `emp_id` as the primary key.
*Oracle code:*
```
create table testrowid_s1 (emp_id integer, name varchar2(10), CONSTRAINT testrowid_pk PRIMARY KEY (emp_id));
INSERT INTO testrowid_s1(emp_id,name) values (1,'empname1');
INSERT INTO testrowid_s1(emp_id,name) values (2,'empname2');
INSERT INTO testrowid_s1(emp_id,name) values (3,'empname3');
INSERT INTO testrowid_s1(emp_id,name) values (4,'empname4');
commit;
SELECT rowid,emp_id,name FROM testrowid_s1;
ROWID EMP_ID NAME
------------------ ---------- ----------
AAAF3pAAAAAAAMOAAA 1 empname1
AAAF3pAAAAAAAMOAAB 2 empname2
AAAF3pAAAAAAAMOAAC 3 empname3
AAAF3pAAAAAAAMOAAD 4 empname4
UPDATE testrowid_s1 SET name = 'Ramesh' WHERE rowid = 'AAAF3pAAAAAAAMOAAB' ;
commit;
SELECT rowid,emp_id,name FROM testrowid_s1;
ROWID EMP_ID NAME
------------------ ---------- ----------
AAAF3pAAAAAAAMOAAA 1 empname1
AAAF3pAAAAAAAMOAAB 2 Ramesh
AAAF3pAAAAAAAMOAAC 3 empname3
AAAF3pAAAAAAAMOAAD 4 empname4
```
*PostgreSQL code:*
```
CREATE TABLE public.testrowid_s1
(
emp_id integer,
name character varying,
primary key (emp_id)
);
insert into public.testrowid_s1 (emp_id,name) values
(1,'empname1'),(2,'empname2'),(3,'empname3'),(4,'empname4');
select emp_id,name from testrowid_s1;
emp_id | name
--------+----------
1 | empname1
2 | empname2
3 | empname3
4 | empname4
update testrowid_s1 set name = 'Ramesh' where emp_id = 2 ;
select emp_id,name from testrowid_s1;
emp_id | name
--------+----------
1 | empname1
3 | empname3
4 | empname4
2 | Ramesh
```
**Scenario 2: Using a logical primary key**
In the following examples, you create the table `testrowid_s2` with `emp_id` as the logical primary key.
*Oracle code:*
```
create table testrowid_s2 (emp_id integer, name varchar2(10) );
INSERT INTO testrowid_s2(emp_id,name) values (1,'empname1');
INSERT INTO testrowid_s2(emp_id,name) values (2,'empname2');
INSERT INTO testrowid_s2(emp_id,name) values (3,'empname3');
INSERT INTO testrowid_s2(emp_id,name) values (4,'empname4');
commit;
SELECT rowid,emp_id,name FROM testrowid_s2;
ROWID EMP_ID NAME
------------------ ---------- ----------
AAAF3rAAAAAAAMeAAA 1 empname1
AAAF3rAAAAAAAMeAAB 2 empname2
AAAF3rAAAAAAAMeAAC 3 empname3
AAAF3rAAAAAAAMeAAD 4 empname4
UPDATE testrowid_s2 SET name = 'Ramesh' WHERE rowid = 'AAAF3rAAAAAAAMeAAB' ;
commit;
SELECT rowid,emp_id,name FROM testrowid_s2;
ROWID EMP_ID NAME
------------------ ---------- ----------
AAAF3rAAAAAAAMeAAA 1 empname1
AAAF3rAAAAAAAMeAAB 2 Ramesh
AAAF3rAAAAAAAMeAAC 3 empname3
AAAF3rAAAAAAAMeAAD 4 empname4
```
*PostgreSQL code:*
```
CREATE TABLE public.testrowid_s2
(
emp_id integer,
name character varying
);
insert into public.testrowid_s2 (emp_id,name) values
(1,'empname1'),(2,'empname2'),(3,'empname3'),(4,'empname4');
select emp_id,name from testrowid_s2;
emp_id | name
--------+----------
1 | empname1
2 | empname2
3 | empname3
4 | empname4
update testrowid_s2 set name = 'Ramesh' where emp_id = 2 ;
select emp_id,name from testrowid_s2;
emp_id | name
--------+----------
1 | empname1
3 | empname3
4 | empname4
2 | Ramesh
```
**Scenario 3: Using an identity attribute**
In the following examples, you create the table `testrowid_s3` with no primary key and by using an identity attribute.
*Oracle code:*
```
create table testrowid_s3 (name varchar2(10));
INSERT INTO testrowid_s3(name) values ('empname1');
INSERT INTO testrowid_s3(name) values ('empname2');
INSERT INTO testrowid_s3(name) values ('empname3');
INSERT INTO testrowid_s3(name) values ('empname4');
commit;
SELECT rowid,name FROM testrowid_s3;
ROWID NAME
------------------ ----------
AAAF3sAAAAAAAMmAAA empname1
AAAF3sAAAAAAAMmAAB empname2
AAAF3sAAAAAAAMmAAC empname3
AAAF3sAAAAAAAMmAAD empname4
UPDATE testrowid_s3 SET name = 'Ramesh' WHERE rowid = 'AAAF3sAAAAAAAMmAAB' ;
commit;
SELECT rowid,name FROM testrowid_s3;
ROWID NAME
------------------ ----------
AAAF3sAAAAAAAMmAAA empname1
AAAF3sAAAAAAAMmAAB Ramesh
AAAF3sAAAAAAAMmAAC empname3
AAAF3sAAAAAAAMmAAD empname4
```
*PostgreSQL code:*
```
CREATE TABLE public.testrowid_s3
(
rowid_seq bigint generated always as identity,
name character varying
);
insert into public.testrowid_s3 (name) values
('empname1'),('empname2'),('empname3'),('empname4');
select rowid_seq,name from testrowid_s3;
rowid_seq | name
-----------+----------
1 | empname1
2 | empname2
3 | empname3
4 | empname4
update testrowid_s3 set name = 'Ramesh' where rowid_seq = 2 ;
select rowid_seq,name from testrowid_s3;
rowid_seq | name
-----------+----------
1 | empname1
3 | empname3
4 | empname4
2 | Ramesh
```
# Migrate Oracle Database error codes to an Amazon Aurora PostgreSQL-Compatible database
*Sai Parthasaradhi and Veeranjaneyulu Grandhi, Amazon Web Services*
## Summary
This pattern shows how to migrate Oracle Database error codes to an [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) database by using a predefined metadata table.
Oracle Database error codes don’t always have a corresponding PostgreSQL error code. This difference in error codes can make it difficult to configure the processing logic of the procedures or functions in the target PostgreSQL architecture.
You can simplify the process by storing the source and target database error codes that are meaningful to your PL/pgSQL program in a metadata table. Then, configure the table to flag valid Oracle Database error codes and map them to their PostgreSQL equivalents before continuing with the remaining process logic. If the Oracle Database error code isn’t in the metadata table, the process exits with the exception. Then, you can manually review the error details and add the new error code to the table if your program requires it.
By using this configuration, your Amazon Aurora PostgreSQL-Compatible database can handle errors in the same way that your source Oracle database does.
**Note**
Configuring a PostgreSQL database to handle Oracle Database error codes correctly usually requires changes to the database and application code.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Oracle Database with instance and listener services up and running
+ An Amazon Aurora PostgreSQL-Compatible cluster that’s up and running
+ Familiarity with Oracle Database
+ Familiarity with PostgreSQL databases
## Architecture
The following diagram shows an example Amazon Aurora PostgreSQL-Compatible database workflow for data error code validation and handling:
![\[Data error code validation and handling for an Aurora PostgreSQL-Compatible database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/82751f40-2fd9-4ce7-ab61-0874552d857b/images/b7ab627e-8f34-4635-8660-93c5c80ce38d.png)
The diagram shows the following workflow:
1. A table holds Oracle Database error codes and classifications and their equivalent PostgreSQL error codes and classifications. The table includes a **valid\$1error** column that classifies if specific, predefined error codes are valid or not.
1. When a PL/pgSQL function (**func\$1processdata**) throws an exception, it invokes a second PL/pgSQL function (**error\$1validation**).
1. The **error\$1validation** function accepts the Oracle Database error code as an input argument. Then, the function checks the incoming error code against the table to see if the error is included in the table.
1. If the Oracle Database error code is included in the table, then the **error\$1validation** function returns a **TRUE** value and the process logic continues. If the error code isn’t included in the table, then the function returns a **FALSE **value, and the process logic exits with an exception.
1. When the function returns a **FALSE** value, then the error details are manually reviewed by the application’s functional lead to determine its validity.
1. The new error code is then either manually added to the table or not. If the error code is valid and added to the table, then the **error\$1validation** function returns a **TRUE** value the next time the exception occurs. If the error code isn’t valid, and the process must fail when the exception occurs, then the error code isn’t added to the table.
**Technology stack**
+ Amazon Aurora PostgreSQL
+ pgAdmin
+ Oracle SQL Developer
## Tools
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [pgAdmin](https://www.pgadmin.org/) is an open-source administration and development tool for PostgreSQL. It provides a graphical interface that simplifies the creation, maintenance, and use of database objects.
+ [Oracle SQL Developer](https://www.oracle.com/in/database/technologies/appdev/sqldeveloper-landing.html) is a free, integrated development environment that simplifies the development and management of Oracle Database in both traditional and cloud deployments.
## Epics
### Migrate Oracle Database error codes to your Amazon Aurora PostgreSQL-Compatible database
| Task | Description | Skills required |
| --- | --- | --- |
| Create a table in the Amazon Aurora PostgreSQL-Compatible database. | Run the following PostgreSQL [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) command:
(
source_error_code numeric NOT NULL,
target_error_code character varying NOT NULL,
valid_error character varying(1) NOT NULL
);
| PostgreSQL Developer, Oracle, RDS/Aurora for PostgreSQL |
| Add PostgreSQL error codes and their corresponding Oracle Database error codes to the table. | Run the PostgreSQL [INSERT](https://www.postgresql.org/docs/current/sql-insert.html) command to add the required error code values to the **error\$1codes** table.The PostgreSQL error codes must use the character varying data type (**SQLSTATE **value). The Oracle error codes must use the numeric data type (**SQLCODE** value).**Example Insert statements:**
insert into error_codes values (-1817,'22007','Y'); insert into error_codes values (-1816,'22007','Y'); insert into error_codes values (-3114,'08006','N');
If you’re catching Oracle-specific Java database connectivity (JDBC) exceptions, you must replace those exceptions with either generic cross-database exceptions or switch to PostgreSQL-specific exceptions. | PostgreSQL Developer, Oracle, RDS/Aurora for PostgreSQL |
| Create a PL/pgSQL function to validate error codes. | Create a PL/pgSQL function by running the PostgreSQL [CREATE FUNCTION](https://www.postgresql.org/docs/current/sql-createfunction.html) command. Make sure that the function does the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-database-error-codes-to-an-amazon-aurora-postgresql-compatible-database.html) | PostgreSQL Developer, Oracle, RDS/Aurora for PostgreSQL |
| Manually review new error codes as they’re recorded by the PL/pgSQL function. | Manually review the new error codes.If a new error code is valid for your use case, add it to the **error\$1codes** table by running the PostgreSQL **INSERT** command.-or-If a new error code isn’t valid for your use case, don’t add it to the table. The process logic will continue to fail and exit with exception when the error occurs. | PostgreSQL Developer, Oracle, RDS/Aurora for PostgreSQL |
## Related resources
[Appendix A. PostgreSQL Error Codes](https://www.postgresql.org/docs/11/errcodes-appendix.html) (PostgreSQL documentation)
[Database error messages](https://docs.oracle.com/cd/E11882_01/server.112/e17766/toc.htm) (Oracle Database documentation)
# Migrate SAP ASE on Amazon EC2 to Amazon Aurora PostgreSQL-Compatible using AWS SCT and AWS DMS
*Amit Kumar and Ankit Gupta, Amazon Web Services*
## Summary
This pattern describes how to migrate an SAP Adaptive Server Enterprise (SAP ASE) database that is hosted on an Amazon Elastic Compute Cloud (Amazon EC2) instance to Amazon Aurora PostgreSQL-Compatible Edition by using AWS Schema Conversion Tool (AWS SCT) and AWS Database Migration Service (AWS DMS). The pattern focuses on both data definition language (DDL) conversions for stored objects and data migration.
Aurora PostgreSQL-Compatible supports online transaction processing (OLTP) workloads. This managed service provides configurations that automatically scale on demand. It can automatically start up, shut down, scale up, or scale down your database based on your application's needs. You can run your database in the cloud without managing any database instances. Aurora PostgreSQL-Compatible provides a cost-effective option for infrequent, intermittent, or unpredictable workloads.
The migration process consists of two main phases:
+ Converting the database schema by using AWS SCT
+ Migrating the data by using AWS DMS
Detailed instructions for both phases are provided in the *Epics* section. For information about troubleshooting issues that are specific to using AWS DMS with SAP ASE databases, see [Troubleshooting issues with SAP ASE](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting.html#CHAP_Troubleshooting.SAP) in the AWS DMS documentation.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source SAP ASE database on an EC2 instance with server, database, and listener services up and running
+ A target Aurora PostgreSQL-Compatible database
**Limitations**
+ The port number for connections must be 5432.
+ The [huge\$1pages](https://www.postgresql.org/docs/9.6/static/runtime-config-resource.html) feature is on by default but can be modified.
+ Point-in-time recovery (PITR) granularity is 5 minutes.
+ Cross-Region replication is currently not available.
+ The maximum storage size for an Aurora database is 128 TiB.
+ You can create up to 15 read replicas.
+ The table size limit is constrained only by the size of the Aurora cluster volume, so the maximum table size for an Aurora PostgreSQL-Compatible DB cluster is 32 TiB. We recommend that you follow best practices for table design, such as partitioning large tables.
**Product versions**
+ Source database: AWS DMS currently supports SAP ASE 15, 15.5, 15.7, and 16.x. See the [AWS DMS User Guide](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SAP.html) for the latest information about SAP ASE version support.
+ Target database: PostgreSQL 9.4 and later (for version 9.x), 10.x, 11.x, 12.x, 13.x, and 14.x. See the [AWS DMS User Guide](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html) for the latest supported PostgreSQL versions.
+ Amazon Aurora 1.x or later. For the latest information, see [Aurora PostgreSQL-Compatible releases and engine versions](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Updates.20180305.html) in the Aurora documentation.
## Architecture
**Source technology stack **
+ SAP ASE database running on Amazon EC2
**Target technology stack **
+ Aurora PostgreSQL-Compatible database
**Migration architecture **
![\[Migrating an SAP ASE database to Aurora PostgreSQL-Compatible by using AWS SCT and AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/07fbdea1-0242-40ae-8e5f-2ce4a620a047/images/a3b018f3-2e7b-4c37-a218-870c56132acb.png)
## Tools
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and most of the custom code to a format that’s compatible with the target database.
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) supports several different source and target databases. For more information, see [Sources for Data Migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html) and [Targets for Data Migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html) in the AWS DMS documentation. For the most comprehensive version and feature support, we recommend that you use the latest version of AWS DMS.
## Epics
### Set up the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Configure network access in the source EC2 instance. | Set up security groups in the EC2 instance that hosts your source SAP ASE database.For instructions, see [Amazon EC2 security groups for Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html?icmpid=docs_ec2_console) in the Amazon EC2 documentation. | Systems administrator |
| Create your target Aurora PostgreSQL-Compatible DB cluster. | Install, configure, and launch an Aurora PostgreSQL-Compatible cluster for your target database.For more information, see [Creating an Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html) in the Aurora documentation. | DBA |
| Set up authorization for the target DB cluster. | Set up security groups and firewalls for the target database.For instructions, see [Creating an Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html) in the Aurora documentation. | DBA, Systems administrator |
### Convert your database schema with AWS SCT
| Task | Description | Skills required |
| --- | --- | --- |
| Launch AWS SCT. | Launch AWS SCT by following the instructions in the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_GettingStarted.html).AWS SCT provides a project-based user interface to automatically convert the database schema of your SAP ASE source database into a format that’s compatible with your target Aurora PostgreSQL-Compatible DB instance. | DBA |
| Create AWS SCT endpoints. | Create endpoints for the source SAP ASE and target PostgreSQL databases.For instructions, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.AddServers). | DBA |
| Create an assessment report. | Create a database migration assessment report to evaluate the migration and detect any incompatible objects and functions.For instructions, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_UserInterface.html#CHAP_UserInterface.AssessmentReport). | DBA |
| Convert the schema. | Convert the database schema by following the instructions in the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Converting.html). | DBA |
| Validate database objects. | If AWS SCT cannot convert a database object, it will identify its name and other details. You must convert these objects manually.To identify these mismatches, follow the instructions in the AWS blog post [Validate database objects after migrating from SAP ASE to Amazon RDS for PostgreSQL or Amazon Aurora PostgreSQL](https://aws.amazon.com/blogs/database/validate-database-objects-after-migrating-from-sap-ase-to-amazon-rds-for-postgresql-or-amazon-aurora-postgresql/). | DBA |
### Analyze the AWS DMS migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions. | Check the SAP ASE database versions for compatibility with AWS DMS. For more information, see [Sources for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html#CHAP_Introduction.Sources.title) and [Targets for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html) in the AWS DMS documentation. | DBA |
| Identify the requirements for the storage type and capacity. | Choose the appropriate storage capacity for the target database based on the size of your source database. | DBA, Systems administrator |
| Choose the instance type, capacity, and other features of the replication instance. | Choose the instance type, capacity, storage features, and network features that meet your requirements.For guidance, see [Choosing the right AWS DMS replication instance for your migration](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Types.html) in the AWS DMS documentation. | DBA, Systems administrator |
| Identify network access security requirements. | Identify the network access security requirements for the source and target databases.Follow the guidance in [Setting up a network for a replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.VPC.html) in the AWS DMS documentation. | DBA, Systems administrator |
### Migrate the data
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the data by creating a migration task in AWS DMS. | To migrate data, create a task and follow the instructions in the [AWS DMS documentation.](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html) We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support. | DBA |
| Validate the data. | To validate that your data was migrated accurately from the source database to the target database, follow the [data validation guidelines](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html) provided in the AWS DMS documentation. | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Identify the application migration strategy. | Choose one of the [seven strategies (7Rs)](https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-database-migration/planning-phase.html) for migrating applications to the cloud. | DBA, App owner, Systems administrator |
| Follow the application migration strategy. | Complete the database tasks identified by the application team, including updating DNS connection details for the target database and updating dynamic queries. | DBA, App owner, Systems administrator |
### Cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | Switch the connection from the source database to the target database. For more information, see the [Cut over](https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-database-migration/cut-over.html) section of the *Migration strategy for relational databases*. | DBA, App owner, Systems administrator |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary AWS resources. | Terminate all migration tasks, replication instances, endpoints, and other AWS SCT and AWS DMS resources. For more information, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.Replication.html#CHAP_GettingStarted.Replication.Deleting). | DBA, Systems administrator |
| Review and validate the project documents. | Validate all the steps in the project documentation to ensure that all tasks have been completed successfully. | DBA, App owner, Systems administrator |
| Close the project. | Close the migration project and provide any feedback. | DBA, App owner, Systems administrator |
## Related resources
**References**
+ [Enable encrypted connections for PostgreSQL DB instances in Amazon RDS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html) (AWS Prescriptive Guidance)
+ [Transport PostgreSQL databases between two Amazon RDS DB instances using pg\$1transport](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg_transport.html) (AWS Prescriptive Guidance)
+ [Amazon Aurora pricing](https://aws.amazon.com/rds/aurora/pricing/)
+ [Best practices with Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/AuroraPostgreSQL.BestPractices.html) (Amazon Aurora documentation)
+ [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [Using an SAP ASE Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SAP.html)
**Tutorials and videos**
+ [Getting Started with AWS Database Migration Service](https://aws.amazon.com/dms/getting-started/)
+ [AWS Database Migration Service](https://www.youtube.com/watch?v=zb4GcjEdl8U) (video)
# Migrate Windows SSL certificates to an Application Load Balancer using ACM
*Chandra Sekhar Yaratha and Igor Kovalchuk, Amazon Web Services*
## Summary
The pattern provides guidance for using AWS Certificate Manager (ACM) to migrate existing Secure Sockets Layer (SSL) certificates from websites that are hosted on on-premises servers or Amazon Elastic Compute Cloud (Amazon EC2) instances on Microsoft Internet Information Services (IIS). The SSL certificates can then be used with Elastic Load Balancing on AWS.
SSL protects your data, affirms your identity, provides better search engine rankings, helps meet Payment Card Industry Data Security Standard (PCI DSS) requirements, and improves customer trust. Developers and IT teams that manage these workloads want their web applications and infrastructure, including the IIS server and Windows Server, to remain compliant with their baseline policies.
This pattern covers manually exporting existing SSL certificates from Microsoft IIS, converting them from Personal Information Exchange (PFX) format to the Private Enhanced Mail (PEM) format that ACM supports, and then importing them into ACM in your AWS account. It also describes how to create an Application Load Balancer for your application and configure the Application Load Balancer to use your imported certificates. HTTPS connections are then terminated on the Application Load Balancer, and you don’t need further configuration overhead on the web server. For more information, see [Create an HTTPS listener for your Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html).
Windows servers use .pfx or .p12 files to contain the public key file (SSL certificate) and its unique private key file. The Certificate Authority (CA) provides you with your public key file. You use your server to generate the associated private key file where the certificate signing request (CSR) was created.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A virtual private cloud (VPC) on AWS with at least one private and one public subnet in each Availability Zone used by your targets
+ IIS version 8.0 or later running on Windows Server 2012 or later
+ A web application running on IIS
+ Administrator access to the IIS server
## Architecture
**Source technology stack**
+ IIS web server implementation with SSL to ensure that data is transmitted securely in an encrypted connection (HTTPS)
**Source architecture**
![\[Source architecture for migrating Windows SSL certificates to Application Load Balancer using ACM\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/cad6e465-da39-4819-970e-10e1c30e0a1f/images/e63efb6f-205b-4e20-a043-6bc954470191.png)
**Target technology stack**
+ ACM certificates in your AWS account
+ An Application Load Balancer configured to use imported certificates
+ Windows Server instances in the private subnets
**Target architecture**
![\[Target architecture for migrating Windows SSL certificates to Application Load Balancer using ACM\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/cad6e465-da39-4819-970e-10e1c30e0a1f/images/45ac7fba-fbad-4c74-9b1f-80ca212dae08.png)
## Tools
+ [AWS Certificate Manager (ACM)](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html) helps you create, store, and renew public and private SSL/TLS X.509 certificates and keys that protect your AWS websites and applications.
+ [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across EC2 instances, containers, and IP addresses in one or more Availability Zones.
## Best practices
+ Enforce traffic redirects from HTTP to HTTPS.
+ Configure security groups for your Application Load Balancer properly to allow inbound traffic only to specific ports.
+ Launch your EC2 instances in different Availability Zones to ensure high availability.
+ Configure your application’s domain to point to the Application Load Balancer’s DNS name instead of its IP address.
+ Make sure that the Application Load Balancer has application-layer [health checks](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html) configured.
+ Configure the threshold for health checks.
+ Use [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) to monitor the Application Load Balancer.
## Epics
### Export a .pfx file
| Task | Description | Skills required |
| --- | --- | --- |
| Export the .pfx file from Windows Server. | To export the SSL certificate as a .pfx file from the on-premises IIS manager in Windows Server:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.html)Your .pfx file should now be saved to the location and path you specified. | Systems administrator |
### Convert the PFX-encoded certificate to PEM format
| Task | Description | Skills required |
| --- | --- | --- |
| Download and install the OpenSSL toolkit. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.html) | Systems administrator |
| Convert the PFX-encoded certificate to PEM format. | The following steps convert the PFX-encoded, signed certificate file into three files in PEM format:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.html)To convert the PFX encoded certificate:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.html) | Systems administrator |
### Import a certificate into ACM
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare to import the certificate. | On the [ACM console](https://console.aws.amazon.com/acm/home), choose **Import a certificate**. | Cloud administrator |
| Provide the certificate body. | For **Certificate body**, paste the PEM-encoded certificate that you want to import. For more information about the commands and steps described in this and other tasks in this epic, see [Importing a certificate](https://docs.aws.amazon.com/acm/latest/userguide/import-certificate-api-cli.html) in the ACM documentation. | Cloud administrator |
| Provide the certificate private key. | For **Certificate private key**, paste the PEM-encoded, unencrypted private key that matches the certificate's public key. | Cloud administrator |
| Provide the certificate chain. | For **Certificate chain**, paste the PEM-encoded certificate chain, which is stored in the `CertificateChain.pem` file. | Cloud administrator |
| Import the certificate. | Choose **Review and import**. Confirm that the information about your certificate is correct, and then choose **Import**. | Cloud administrator |
### Create an Application Load Balancer
| Task | Description | Skills required |
| --- | --- | --- |
| Create and configure the load balancer and listeners. | Follow the instructions in the [Elastic Load Balancing documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html) to configure a target group, register targets, and create an Application Load Balancer and listener. Add a second listener (HTTPS) for port 443. | Cloud administrator |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Windows PowerShell doesn't recognize the OpenSSL command even after you add it to the system path. | Check `$env:path` to make sure that it includes the location of the OpenSSL binaries.If it doesn’t, run the following command in PowerShell:
$env:path = $env:path + ";C:\OpenSSL-Win64\bin"
|
## Related resources
**Importing a certificate into ACM**
+ [ACM console](https://console.aws.amazon.com/acm/home)
+ [Certificate and key format for importing](https://docs.aws.amazon.com/acm/latest/userguide/import-certificate-format.html)
+ [Importing a certificate](https://aws.amazon.com/blogs/security/how-to-import-pfx-formatted-certificates-into-aws-certificate-manager-using-openssl/)
+ [AWS Certificate Manager User Guide](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html)
**Creating an Application Load Balancer**
+ [Create an Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html)
+ [Application Load Balancer User Guide](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)
# Migrate a messaging queue from Microsoft Azure Service Bus to Amazon SQS
*Nisha Gambhir, Amazon Web Services*
## Summary
This pattern describes how to migrate a .NET Framework or .NET Core web or console application from using the Microsoft Azure Service Bus queue messaging platform to Amazon Simple Queue Service (Amazon SQS).
Applications use messaging services to send data to, and receive data from, other applications. These services help build decoupled, highly scalable microservices, distributed systems, and serverless applications in the cloud.
Azure Service Bus queues are part of a broader Azure messaging infrastructure that supports queuing and publish/subscribe messaging.
Amazon SQS is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Amazon SQS eliminates the complexity and overhead associated with managing and operating message-oriented middleware, and enables developers to focus on differentiating work. Using Amazon SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A .NET Framework or .NET Core web or console application that uses Azure Service Bus queues (sample code attached)
**Product versions**
+ .NET Framework 3.5 or later, or .NET Core 1.0.1, 2.0.0, or later
## Architecture
**Source technology stack **
+ A .NET (Core or Framework) web or console application that uses an Azure Service Bus queue to send messages
**Target technology stack **
+ Amazon SQS
## Tools
**Tools**
+ Microsoft Visual Studio
**Code**
To create an AWS Identity and Access management (IAM) policy for Amazon SQS:
1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
2. In the navigation pane on the left, choose **Policies**, and then choose **Create policy**.
3. Choose the **JSON** tab, and paste the following code:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"sqs:GetQueueUrl",
"sqs:ChangeMessageVisibility",
"sqs:SendMessageBatch",
"sqs:ReceiveMessage",
"sqs:SendMessage",
"sqs:GetQueueAttributes",
"sqs:ListQueueTags",
"sqs:ListDeadLetterSourceQueues",
"sqs:DeleteMessageBatch",
"sqs:PurgeQueue",
"sqs:DeleteQueue",
"sqs:CreateQueue",
"sqs:ChangeMessageVisibilityBatch",
"sqs:SetQueueAttributes"
],
"Resource": "arn:aws:sqs:*::*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "sqs:ListQueues",
"Resource": "*"
}
]
}
```
4. Choose **Review policy**, type a name, and then choose **Create policy**.
5. Attach the newly created policy to your existing IAM role or create a new role.
## Epics
### Set up Amazon SQS in AWS
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM policy for Amazon SQS. | Create the IAM policy that will provide access to Amazon SQS. See the Code section for a sample policy. | System engineer |
| Create an AWS profile. | Create a new profile by running the AWS Tools for PowerShell command Set-AWSCredential. This command stores your access key and secret key in your default credentials file under the profile name you specify. Link the Amazon SQS policy you created earlier with this account. Keep the AWS access key ID and secret access key. These will be required in the next steps. | System engineer |
| Create an SQS queue. | You can create a standard queue or a first in, first out (FIFO) queue. For instructions, see the link in the References section. | System engineer |
### Revise your .NET application code
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS Toolkit for Visual Studio. | This toolkit is an extension for Microsoft Visual Studio and makes it easier for you to build and deploy .NET applications in AWS. For installation and usage instructions, see the link in the References section. | Application developer |
| Install the AWSSDK.SQS NuGet package. | You can install AWSSDK.SQS by choosing "Manage NuGet Package" in Visual Studio or by running the command "Install-Package AWSSDK.SQS". | Application developer |
| Create an AWSCredentials object in your .NET application. | The sample application in the attachment shows how to create a BasicAWSCredentials object, which inherits from AWSCredentials. You can use the access key ID and secret access key from earlier, or let the object pick these from the .aws folder as part of the user profile at run time. | Application developer |
| Create an SQS client object. | Create an SQS client object (AmazonSQSClient) for .NET Framework. This is part of the Amazon.SQS namespace. This object is required instead of IQueueClient, which is part of the Microsoft.Azure.ServiceBus namespace. | Application developer |
| Call the SendMessageAsync method to send messages to the SQS queue. | Change the code that sends the message to the queue to use the amazonSqsClient.SendMessageAsync method. For details, see the attached code sample. | Application developer |
| Call the ReceiveMessageAsync method to receive messages from the SQS queue. | Change the code that receives the message to use the amazonSqsClient.ReceiveMessageAsync method. For details, see the attached code sample. | Application developer |
| Call the DeleteMessageAsync method to delete messages from the SQS queue. | To delete messages, change the code from the queueClient.CompleteAsync method to the amazonSqsClient.DeleteMessageAsync method. For details, see the attached code sample. | Application developer |
## Related resources
+ [AWS SDK for .NET Developer Guide](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/welcome.html)
+ [Messaging Using Amazon SQS](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/sqs-apis-intro.html)
+ [Creating and Using an Amazon SQS Queue with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/how-to-sqs.html)
+ [Send an Amazon SQS Message](https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/SendMessage.html)
+ [Receive a Message from an Amazon SQS Queue](https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/ReceiveMessage.html)
+ [Delete a Message from an Amazon SQS Queue](https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/DeleteMessage.html)
+ [AWS Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/welcome.html)
## Additional information
This pattern includes two sample applications (see the attachments section):
+ **AzureSbTestApp** includes code that uses the Azure Service Bus queue.
+ **AmazonSqsTestApp** uses Amazon SQS. This is a console application that uses .NET Core 2.2 and includes examples for sending and receiving messages.
Notes:
+ queueClient is an object of IQueueClient, which is part of the Microsoft.Azure.ServiceBus namespace (included in the Microsoft.Azure.ServiceBus NuGet package).
+ amazonSqsClient is an object of AmazonSQSClient, which is part of the Amazon.SQS namespace (included in the AWSSDK.SQS NuGet package).
+ Depending upon where the code is running, say if its running on EC2, the role needs to have permission to write into the SQS Queue.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/25334709-7000-4f60-87ed-ea41acb41a99/attachments/attachment.zip)
# Migrate an Oracle JD Edwards EnterpriseOne database to AWS by using Oracle Data Pump and AWS DMS
*Thanigaivel Thirumalai, Amazon Web Services*
## Summary
You can migrate and run your JD Edwards EnterpriseOne database on [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html). When you migrate your database to Amazon RDS, AWS can take care of backup tasks and high availability setup, so you can concentrate on maintaining your EnterpriseOne application and its functionality. For a comprehensive list of key factors to consider during the migration process, see [Oracle database migration strategies](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/strategies.html) in AWS Prescriptive Guidance.
There are multiple ways to migrate an EnterpriseOne database, including:
+ Using Oracle Universal Batch Engine (UBE) R98403 for schema and table creation, and using AWS Database Migration Service (AWS DMS) for migration
+ Using DB native tools for schema and table creation and using AWS DMS for migration
+ Using DB native tools for the migration of existing data (full load) and using AWS DMS for change data capture (CDC) tasks
This pattern covers the third option. It explains how to migrate your on-premises EnterpriseOne databases to Amazon RDS for Oracle by using Oracle Data Pump with [AWS DMS](https://aws.amazon.com/dms) and its CDC feature.
[Oracle JD Edwards EnterpriseOne](https://www.oracle.com/applications/jd-edwards-enterpriseone/) is an enterprise resource planning (ERP) solution for organizations that manufacture, construct, distribute, service, or manage products or physical assets. JD Edwards EnterpriseOne supports various hardware, operating systems, and database platforms.
When you migrate critical ERP applications such as JD Edwards EnterpriseOne, minimizing downtime is key. AWS DMS minimizes downtime by supporting both full load and continuous replication from the source database to the target database. AWS DMS also provides real-time monitoring and logging for the migration, which can help you identify and resolve any issues that could cause downtime.
When you replicate changes with AWS DMS, you must specify a time or system change number (SCN) as the starting point for reading changes from the database logs. It's crucial to keep these logs accessible on the server for a designated amount of time (we recommend 15 days) to ensure that AWS DMS has access to these changes.
## Prerequisites and limitations
**Prerequisites**
+ An Amazon RDS for Oracle database provisioned in your AWS Cloud environment as the target database. For instructions, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.Oracle.html).
+ An EnterpriseOne database that’s running on premises or on an Amazon Elastic Compute Cloud (Amazon EC2) instance on AWS.
**Note**
This pattern is designed for migrating from on premises to AWS, but it was tested by using an EnterpriseOne database on an EC2 instance. If you plan to migrate from your on-premises environment, you must configure the appropriate network connectivity.
+ Schema details. Identify which Oracle database schema (for example, DV920) you plan to migrate for EnterpriseOne. Before you start the migration process, gather the following details about the schema:
+ Schema size
+ The number of objects per object type
+ The number of invalid objects
**Limitations**
+ You have to create any schemas you want on the target Amazon RDS for Oracle database―AWS DMS doesn't create these for you. (The [Epics](#migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms-epics) section describes how to use Data Pump to export and import schemas.) The schema name must already exist for the target Oracle database. Tables from the source schema are imported to the user or the schema, and AWS DMS uses the administrator or system account to connect to the target instance. To migrate multiple schemas, you can create multiple replication tasks. You can also migrate data to different schemas on a target instance. To do this, use schema transformation rules on the AWS DMS table mappings.
+ This pattern has been tested with a demo dataset. We recommend that you validate compatibility for your dataset and customization.
+ This pattern uses an EnterpriseOne database that’s running on Microsoft Windows. However, you can use the same process with other operating systems that are supported by AWS DMS.
## Architecture
The following diagram shows a system that’s running EnterpriseOne on an Oracle database as the source database, and an Amazon RDS for Oracle database as the target database. The data is exported from the source Oracle database and imported into the target Amazon RDS for Oracle database by using Oracle Data Pump, and replicated for CDC updates by using AWS DMS.
![\[Diagram showing data replication from on-premises Oracle to Amazon RDS using AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/c8ec3789-f80e-4f3a-a3f4-72a4541316b0/images/4e3e3477-2fe0-4a5d-b95e-05a8aafe8b68.png)
1. Oracle Data Pump extracts data from the source database, and the data is sent to the Amazon RDS for Oracle database target.
1. CDC data is sent from the source database to a source endpoint in AWS DMS.
1. From the source endpoint, the data is sent to the AWS DMS replication instance, where the replication task is performed.
1. After the replication task is complete, the data is sent to the target endpoint in AWS DMS.
1. From the target endpoint, the data is sent to the Amazon RDS for Oracle database instance.
## Tools
**AWS services**
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Relational Database Service (Amazon RDS) for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) helps you set up, operate, and scale an Oracle relational database in the AWS Cloud.
**Other services**
+ [Oracle Data Pump](https://docs.oracle.com/cd/B19306_01/server.102/b14215/dp_overview.htm) helps you move data and metadata from one database to another at high speed.
## Best practices
**Migrating LOBs**
If your source database contains large binary objects (LOBs) that need to be migrated to the target database, AWS DMS provides the following options:
+ **Full LOB mode** – AWS DMS migrates all the LOBs from the source to the target database regardless of their size. Although the migration is slower than the other modes, the advantage is that data isn’t truncated. For better performance, you can create a separate task on the new replication instance to migrate the tables that have LOBs that are larger than a few megabytes.
+ **Limited LOB mode** – You specify the maximum size of LOB column data, which allows AWS DMS to pre-allocate resources and apply the LOBs in bulk. If the size of the LOB columns exceeds the size that is specified in the task, AWS DMS truncates the data and sends warnings to the AWS DMS log file. You can improve performance by using limited LOB mode if your LOB data size is within the limited LOB size.
+ **Inline LOB mode** – You can migrate LOBs without truncating the data or slowing the performance of your task by replicating both small and large LOBs. First, specify a value for the `InlineLobMaxSize` parameter, which is available only when full LOB mode is set to `true`. The AWS DMS task transfers the small LOBs inline, which is more efficient. Then, AWS DMS migrates the large LOBs by performing a lookup from the source table. However, inline LOB mode works only during the full load phase.
**Generating sequence values**
During the AWS DMS CDC process, incremental sequence numbers aren’t replicated from the source database. To avoid discrepancies in sequence values, you must generate the most recent sequence value from the source for all sequences, and apply it to the target Amazon RDS for Oracle database.
**AWS Secrets Manager**
To help manage your credentials, we recommend that you follow the instructions in the blog post [Manage your AWS DMS endpoint credentials with AWS Secrets Manager](https://aws.amazon.com/blogs/database/manage-your-aws-dms-endpoint-credentials-with-aws-secrets-manager/).
**Performance**
+ **Replication instances** ‒ For guidance on choosing the best instance size, see [Selecting the best size for a replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.SizingReplicationInstance.html) in the AWS DMS documentation.
+ **Connectivity options** ‒ To avoid latency issues, we recommend that you choose the right connectivity option. AWS Direct Connect provides the shortest path to AWS resources, because it is a dedicated connection between your corporate data centers and AWS. While in transit, your network traffic remains on the AWS global network and never goes over the internet. This reduces the chance of hitting bottlenecks or unexpected increases in latency when compared with using VPN or the public internet.
+ **Network bandwidth** ‒ To optimize performance, verify that your network throughput is fast. If you are using a VPN tunnel between your on-premises source database and AWS DMS, ensure that the bandwidth is sufficient for your workload.
+ **Task parallelism** ‒ You can speed up data replication by loading multiple tables in parallel during full load. This pattern uses of RDBMS endpoints, so this option applies only to the full load process. Task parallelism is controlled by the `MaxFullLoadSubTasks` parameter, which determines how many full load sub-tasks are run in parallel. By default, this parameter is set to 8, which means that eight tables (if selected in table mapping) are loaded together during full mode. You can adjust this parameter in the full-load task settings section of the JSON script for the task.
+ **Table parallelism** ‒ AWS DMS also enables you to load a single large table by using multiple parallel threads. This is particularly useful for Oracle source tables that have billions of records as well as multiple partitions and subpartitions. If the source table isn’t partitioned, you can use column boundaries for parallel loads.
+ **Split loads** ‒ When you split loads across multiple tasks or AWS DMS instances, remember transaction boundaries when you capture changes.
## Epics
### Use Oracle Data Pump to export the EnterpriseOne schema
| Task | Description | Skills required |
| --- | --- | --- |
| Generate the SCN. | When the source database is active and in use by the EnterpriseOne application, initiate the data export with Oracle Data Pump. You must first generate a system change number (SCN) from the source database for both data consistency during the export with Oracle Data Pump and as a starting point for CDC in AWS DMS.To generate the current SCN from your source database, use the following SQL statement:
SQL> select current_scn from v$database;
CURRENT_SCN ----------- 30009727
Save the generated SCN. You will use the SCN when you export the data and to create the AWS DMS replication task. | DBA |
| Create the parameter file. | To create a parameter file for exporting the schema, you can use the following code.
Export: Release 19.0.0.0.0 - Production on *** *** ** **:**:** **** Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production Starting "SYSTEM"."SYS_EXPORT_SCHEMA_02": system/********@PARFILE='E:\exp_dms_datapump.par' Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS Processing object type SCHEMA_EXPORT/STATISTICS/MARKER Processing object type SCHEMA_EXPORT/USER Processing object type SCHEMA_EXPORT/ROLE_GRANT Processing object type SCHEMA_EXPORT/DEFAULT_ROLE Processing object type SCHEMA_EXPORT/TABLESPACE_QUOTA Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA Processing object type SCHEMA_EXPORT/TABLE/TABLE Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT . . exported ""."
" 228.9 MB 496397 rows
Master table "SYSTEM"."SYS_EXPORT_SCHEMA_02" successfully loaded/unloaded ****************************************************************************** Dump file set for SYSTEM.SYS_EXPORT_SCHEMA_02 is: E:\DMSDUMP\EXPORT_DMS_DATA.DMP Job "SYSTEM"."SYS_EXPORT_SCHEMA_02" successfully completed at *** *** ** **:**:** **** elapsed 0 00:01:57
| DBA |
### Use Oracle Data Pump to import the EnterpriseOne schema
| Task | Description | Skills required |
| --- | --- | --- |
| Transfer the dump file to the target instance. | To transfer your files by using the `DBMS_FILE_TRANSFER` utility, you need to create a database link from the source database to the Amazon RDS for Oracle instance. After the link is established, you can use the utility to transfer the Data Pump files directly to the Amazon RDS instance.Alternatively, you can transfer the Data Pump files to [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) and then import them into the Amazon RDS for Oracle instance. For more information about this option, see the [Additional information](#migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms-additional) section.To create a database link `ORARDSDB` that connects to the Amazon RDS master user at the target DB instance, run the following commands on the source database:
sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 on *** *** ** **:**:** **** Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to: Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 Version 19.3.0.0.0
SQL> create database link orardsdb connect to admin identified by "******" using '(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = orcl.******.us-east-1.rds.amazonaws.com)(PORT = 1521))(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl)))';
Database link created.
SQL>
| DBA |
| Test the database link. | Test the database link to make sure that you can connect to the Amazon RDS for Oracle target database by using `sqlplus`.
SQL> select name from v$database@orardsdb;
NAME --------- ORCL
| DBA |
| Transfer the dump file to the target database. | To copy the dump file over to the Amazon RDS for Oracle database, you can either use the default `DATA_PUMP_DIR` directory or you can create your own directory by using the following code, which has to run on the target Amazon RDS instance:
The following script copies a dump file named `EXPORT_DMS_DATA.DMP` from the source instance to a target Amazon RDS for Oracle database by using the database link named `orardsdb`. You must run the script on the source database instance.
| DBA |
| List the dump file in the target database. | After the PL/SQL procedure is complete, you can list the data dump file in the Amazon RDS for Oracle database by using the following code:
select * from table (rdsadmin.rds_file_util.listdir(p_directory => 'DMS_TARGET_PUMP_DIR'));
| DBA |
| Create JDE-specific users in the target Instance. | Create a JD Edwards profile and role by using these commands in the target instance:
SQL> CREATE ROLE "JDEADMIN"; CREATE ROLE "JDEUSER"; Role created. Role created.
Grant the required permissions to the role:
SQL> GRANT CREATE ANY SEQUENCE TO JDE_ROLE; GRANT DROP ANY SEQUENCE TO JDE_ROLE; GRANT CREATE ANY TRIGGER TO JDE_ROLE; GRANT DROP ANY TRIGGER TO JDE_ROLE;
| DBA, JDE CNC |
| Create tablespaces in the target instance. | Create the required tablespaces in the target instance by using the following commands for the schemas that are involved in this migration:
SQL> CREATE TABLESPACE ; Tablespace created.
SQL> CREATE TABLESPACE ; Tablespace created.
| DBA, JDE CNC |
| Initiate the import on the target database. | Before you start the import process, set up the roles, schemas, and tablespaces on the target Amazon RDS for Oracle database by using the data dump file.To perform the import, access the target database with the Amazon RDS primary user account, and use the connection string name in the `tnsnames.ora` file, which includes the Amazon RDS for Oracle Database `tns-entry`. If necessary, you can include a remap option to import the data dump file into a different tablespace or under a different schema name.To start the import, use the following code:
To ensure a successful import, check the import log file for any errors, and review details such as object count, row count, and invalid objects. If there are any invalid objects, recompile them. Additionally, compare the source and target database objects to confirm that they match. | DBA |
### Provision an AWS DMS replication instance with the source and target endpoints
| Task | Description | Skills required |
| --- | --- | --- |
| Download the template. | Download the AWS CloudFormation [DMS\$1instance.yaml](https://aws-database-blog.s3.amazonaws.com/artifacts/Migrating_oracle_using_DMS/DMS_Instance.yaml) template to provision the AWS DMS replication instance and its source and target endpoints. | Cloud administrator, DBA |
| Start the stack creation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html) | Cloud administrator, DBA |
| Specify the parameters. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html) | Cloud administrator, DBA |
| Create the stack. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html)The provisioning should complete in approximately 5–10 minutes. It is complete when the AWS CloudFormation Stacks page shows **CREATE\$1COMPLETE**. | Cloud administrator, DBA |
| Set up the endpoints. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html) | Cloud administrator, DBA |
| Test connectivity. | After the source and target endpoints shows the status as **Active**, test the connectivity. Choose **Run test** for each endpoint (source and target) to make sure that the status shows as successful. | Cloud administrator, DBA |
### Create an AWS DMS replication task for live replication
| Task | Description | Skills required |
| --- | --- | --- |
| Create the replication task. | Create the AWS DMS replication task by using the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html)After you create the task, AWS DMS migrates ongoing changes to the Amazon RDS for Oracle database instance from the SCN that you provided under CDC start mode. You can also verify the migration by reviewing the CloudWatch logs. | Cloud administrator, DBA |
| Repeat the replication task. | Repeat the previous steps to create replication tasks for other JD Edwards schemas that are part of the migration. | Cloud administrator, DBA, JDE CNC administrator |
### Validate the database schema on the target Amazon RDS for Oracle database
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the data transfer. | After the AWS DMS task starts, you can check the **Table statistics** tab on the **Tasks** page to see the changes made to the data.You can monitor the status of ongoing replication in the console on the **Database migration tasks** page.For more information, see [AWS DMS data validation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html). | Cloud administrator, DBA |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Stop replication. | Discontinue the replication procedure and stop the source application services. | Cloud administrator, DBA |
| Launch the JD Edwards application. | Launch the target JD Edwards presentation and logic tier application on AWS, and direct it to the Amazon RDS for Oracle database.When you access the application, you should notice that all connections are now established with the Amazon RDS for Oracle database. | DBA, JDE CNC administrator |
| Turn off the source database. | After you confirm that there are no more connections, you can turn the source database off. | DBA |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| You receive a warning message to enable [supplemental logging](https://docs.oracle.com/database/121/SUTIL/GUID-D2DDD67C-E1CC-45A6-A2A7-198E4C142FA3.htm#SUTIL1583) in the source database for on-going replication | Enter these commands to enable supplemental logging:
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS; SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (UNIQUE) COLUMNS; SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (FOREIGN KEY) COLUMNS; SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS; SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (UNIQUE) COLUMNS;
|
| AWS DMS has supplemental logging turned off. | Supplemental logging is turned off by default in AWS DMS. To turn it on for a source Oracle endpoint:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html) |
| Supplemental logging isn’t enabled at the CDB level. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.html) |
| You receive the error message: "Test Endpoint failed: Application-Status: 1020912, Application-Message: LogMiner is not supported in Oracle PDB environment Endpoint initialization failed." | If you encounter this error message, you can use Binary Reader instead of LogMiner.Under **Endpoint settings**, add this line to the extra connection attributes for your source database:
useLogMinerReader=N;useBfile=Y;
|
## Related resources
+ [Getting started with AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html)
+ [Best Practices for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html)
+ [Migrating Oracle Databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html)
+ [AWS Database Migration Service resource type reference for AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_DMS.html)
+ [Manage your AWS DMS endpoint credentials with AWS Secrets Manager](https://aws.amazon.com/blogs/database/manage-your-aws-dms-endpoint-credentials-with-aws-secrets-manager/)
+ [Troubleshooting migration tasks in AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting.html#CHAP_Troubleshooting.Oracle.RecordsMissing)
+ [Best practices for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html)
## Additional information
**Transfer files using Amazon S3**
To transfer the files to Amazon S3, you can use the AWS CLI or the Amazon S3 console. After you transfer the files to Amazon S3, you can use the Amazon RDS for Oracle instance to import the Data Pump files from Amazon S3.
If you choose to transfer the dump file using Amazon S3 integration as an alternate method, perform the follow steps:
1. Create an S3 bucket.
1. Export the data from the source database using Oracle Data Pump.
1. Upload the Data Pump files to the S3 bucket.
1. Download the Data Pump files from the S3 bucket to the target Amazon RDS for Oracle database.
1. Perform the import using the Data Pump files.
**Note**
To transfer large data files between S3 and RDS instances, we recommend that you use the [Amazon S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html) feature.
# Migrate an Oracle PeopleSoft database to AWS by using AWS DMS
*sampath kathirvel, Amazon Web Services*
## Summary
[Oracle PeopleSoft](https://www.oracle.com/applications/peoplesoft/) is an enterprise resource planning (ERP) solution for enterprise-wide processes. PeopleSoft has a three-tier architecture: client, application, and database. PeopleSoft can be run on [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html).
If you migrate your Oracle database to Amazon RDS, Amazon Web Services (AWS) can take care of backup tasks and high availability, leaving you free to concentrate on maintaining your PeopleSoft application and its functionality. For a comprehensive list of key factors to consider during the migration process, see [Oracle database migration strategies](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/strategies.html) in AWS Prescriptive Guidance.
This pattern provides a solution for migrating your on-premises Oracle databases to Amazon RDS for Oracle using Oracle Data Pump with [AWS Database Migration Service (AWS DMS)](http://aws.amazon.com/dms) and its change data capture (CDC) feature.
When migrating critical ERP applications such as Oracle PeopleSoft, minimizing the downtime is key. AWS DMS minimizes downtime by supporting both full load and continuous replication. from the source database to the target database. AWS DMS also provides real-time monitoring and logging of the migration, which can help you to identify and resolve any issues that could cause downtime.
When replicating changes with AWS DMS, you must specify a time or system change number (SCN) as the starting point for AWS DMS to read changes from the database logs. It's crucial to keep these logs accessible on the server for a designated amount of time to ensure that AWS DMS has access to these changes.
## Prerequisites and limitations
**Prerequisites**
+ Provisioned Amazon RDS for Oracle database in your AWS Cloud environment as the target database.
+ An Oracle PeopleSoft database running on premises or on Amazon Elastic Compute Cloud (Amazon EC2) in the AWS Cloud.
**Note**
This pattern is designed for migrating from on premises to AWS, but it was tested by using Oracle Database on an Amazon EC2 instance. For migrating from on premises, you will need to configure the appropriate network connectivity.
+ Schema details. When migrating an Oracle PeopleSoft application to Amazon RDS for Oracle, it is necessary to identify which Oracle database schema (for example, `SYSADM`) to migrate. Before starting the migration process, gather the following details about the schema:
+ Size
+ The number of objects per object type
+ The number of invalid objects.
This information will aid in the migration process.
**Limitations**
+ This scenario has been tested only with the PeopleSoft DEMO database. It hasn’t been tested with a large dataset.
## Architecture
The following diagram shows an instance running an Oracle database as the source database and an Amazon RDS for Oracle database as the target database. The data is exported and imported from the source Oracle database to the target Amazon RDS for Oracle database using Oracle Data Pump and replicated for CDC changes using AWS DMS.
![\[Five-step process from on-premises DB instance to Amazon RDS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/c8ec3789-f80e-4f3a-a3f4-72a4541316b0/images/4e3e3477-2fe0-4a5d-b95e-05a8aafe8b68.png)
1. The initial step involves extracting data from the source database by using Oracle Data Pump, followed by sending it to the Amazon RDS for Oracle database target.
1. Data is sent from the source database to a source endpoint in AWS DMS.
1. From the source endpoint, the data is sent to the AWS DMS replication instance, where the replication task is performed.
1. After the replication task is completed, the data is sent to the target endpoint in AWS DMS.
1. From the target endpoint, the data is sent to the Amazon RDS for Oracle database instance.
## Tools
**AWS services**
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ [Amazon Relational Database Service (Amazon RDS) for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) helps you set up, operate, and scale an Oracle relational database in the AWS Cloud.
**Other services**
+ [Oracle Data Pump](https://docs.oracle.com/cd/B19306_01/server.102/b14215/dp_overview.htm) helps you move data and metadata from one database to another at high speeds.
## Best practices
**Migrating LOBs**
If your source database contains large binary objects (LOBs) that need to be migrated to the target database, AWS DMS provides the following options:
+ **Full LOB mode** – AWS DMS migrates all the LOBs from the source to the target database regardless of their size. Although the migration is slower, the advantage is that data isn’t truncated. For better performance, you can create a separate task on the new replication instance to migrate the tables that have LOBs larger than a few megabytes.
+ **Limited LOB mode** – You specify the maximum size of LOB column data, which allows AWS DMS to pre-allocate resources and apply the LOBs in bulk. If the size of the LOB columns exceeds the size that is specified in the task, AWS DMS truncates the data and sends warnings to the AWS DMS log file. You can improve performance by using Limited LOB mode if your LOB data size is within the Limited LOB size.
+ **Inline LOB mode** – You can migrate LOBs without truncating the data or slowing the performance of your task by replicating both small and large LOBs. First, specify a value for the InlineLobMaxSize parameter, which is available only when Full LOB mode is set to true. The AWS DMS task transfers the small LOBs inline, which is more efficient. Then, AWS DMS migrates the large LOBs by performing a lookup from the source table. However, Inline LOB mode works only during the full load phase.
**Generating sequence values**
Keep in mind that during the change data capture process with AWS DMS, incremental sequence numbers are not replicated from the source database. To avoid discrepancies in sequence values, you must generate the most recent sequence value from the source for all sequences, and apply it to the target Amazon RDS for Oracle database.
**Credential management**
To help secure your AWS resources, we recommend following the [best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) for AWS Identity and Access Management (IAM).
## Epics
### Provision an AWS DMS replication instance with the source and target endpoints
| Task | Description | Skills required |
| --- | --- | --- |
| Download the template. | Download the [DMS\$1instance.yaml](https://aws-database-blog.s3.amazonaws.com/artifacts/Migrating_oracle_using_DMS/DMS_Instance.yaml) AWS CloudFormation template to provision the AWS DMS replication instance and its source and target endpoints. | Cloud administrator, DBA |
| Start the stack creation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.html) | Cloud administrator, DBA |
| Specify the parameters. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.html) | Cloud administrator, DBA |
| Create the stack. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.html)The provisioning should complete in approximately 5–10 minutes. It is complete when the AWS CloudFormation Stacks page shows **CREATE\$1COMPLETE**. | Cloud administrator, DBA |
| Set up the endpoints. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.html) | Cloud administrator, DBA |
| Test connectivity. | After the source and target endpoints shows status as Active, test the connectivity. Choose **Run test** for each endpoint (source and target) to make sure that the status shows as successful. | Cloud administrator, DBA |
### Export the PeopleSoft schema from the on-premises Oracle database by using Oracle Data Pump
| Task | Description | Skills required |
| --- | --- | --- |
| Generate the SCN. | When the source database is active and in use by the application, initiate the data export with Oracle Data Pump. You must first generate a system change number (SCN) from the source database for both data consistency during the export with Oracle Data Pump and as a starting point for change data capture in AWS DMS.To generate the current SCN from your source database, enter the following SQL statement.
SQL> select name from v$database; SQL> select name from v$database; NAME --------- PSFTDMO SQL> SELECT current_scn FROM v$database; CURRENT_SCN ----------- 23792008
Save the generated SCN to use when you export the data and for creating the AWS DMS replication task. | DBA |
| Create the parameter file. | To create a parameter file for exporting the schema, you can use the following code.
You can also define your own `DATA_PUMP_DIR` by using the following commands, based on your requirements.
SQL> CREATE OR REPLACE DIRECTORY DATA_PUMP_DIR AS '/opt/oracle/product/19c/dbhome_1/dmsdump/'; Directory created. SQL> GRANT READ, WRITE ON DIRECTORY DATA_PUMP_DIR TO system; Grant succeeded. SQL> SQL> SELECT owner, directory_name, directory_path FROM dba_directories WHERE directory_name='DATA_PUMP_DIR'; OWNER DIRECTORY_NAME DIRECTORY_PATH ------------------------------------------------------------------------------------------------------------------ SYS DATA_PUMP_DIR /opt/oracle/product/19c/dbhome_1/dmsdump/
| DBA |
| Export the schema. | To perform the export, use the `expdp` utility.
| DBA |
### Import the PeopleSoft schema into the Amazon RDS for Oracle database by using Oracle Data Pump
| Task | Description | Skills required |
| --- | --- | --- |
| Transfer the dump file to the target instance. | To transfer your files using `DBMS_FILE_TRANSFER`, you need to create a database link from the source database to the Amazon RDS for Oracle instance. After the link is established, you can use the utility to transfer the Data Pump files directly to the RDS instance.Alternatively, you can transfer the Data Pump files to [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) and then import them into the Amazon RDS for Oracle instance. For more information about this option, see the Additional information section.To create a database link `ORARDSDB` that connects to the Amazon RDS master user at the target DB instance, run the following commands on the source database.
$sqlplus / as sysdba $ SQL> create database link orardsdb connect to admin identified by "*****" using '(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = testpsft.*******.us-west-2.rds.amazonaws.com)(PORT = 1521))(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl)))'; Database link created.
| DBA |
| Test the database link. | Test the database link to make sure that you can connect using sqlplus to the Amazon RDS for Oracle target database.
SQL> SQL> select name from v$database@orardsdb; NAME --------- ORCL SQL>
| DBA |
| Transfer the dump file to the target database. | To copy the dump file over to Amazon RDS for Oracle database, you can either use the default `DATA_PUMP_DIR` directory or you can create your own directory using the following code.
The following script copies a dump file named `export_dms_sample_data_01.dmp` from the source instance to a target Amazon RDS for Oracle database using the database link named `orardsdb`.
| DBA |
| List the dump file in the target database. | After the PL/SQL procedure is completed, you can list the data dump file in the Amazon RDS for Oracle database by using the following code.
SQL> select * from table (rdsadmin.rds_file_util.listdir(p_directory => ‘TARGET_PUMP_DIR’));
| DBA |
| Initiate the import on the target database. | Before you start the import process, set up the roles, schemas, and tablespaces on the target Amazon RDS for Oracle database by using the data dump file.To perform the import, access the target database with the Amazon RDS master user account, and use the connection string name in the `tnsnames.ora` file, which includes the Amazon RDS for Oracle Database `tns-entry`. If necessary, you can include a remap option to import the data dump file into a different tablespace or under a different schema name.To start the import, use the following code.
To ensure a successful import, check the import log file for any errors, and review details such as object count, row count, and invalid objects. If there are any invalid objects, recompile them. Additionally, compare the source and target database objects to confirm that they match. | DBA |
### Create an AWS DMS replication task using CDC to perform live replication
| Task | Description | Skills required |
| --- | --- | --- |
| Create the replication task. | Create the AWS DMS replication task by using the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.html)After you create the task, it migrates the CDC to the Amazon RDS for Oracle database instance from the SCN that you provided under CDC start mode. You can also verify by reviewing the CloudWatch logs. | Cloud administrator, DBA |
### Validate the database schema on the target Amazon RDS for Oracle database
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the data transfer. | After the AWS DMS task starts, you can check the **Table statistics** tab on the **Tasks** page to see the changes made to the data.You can monitor the status of ongoing replication in the console on the **Database migration tasks** page.For more information, see [AWS DMS data validation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html). | Cloud administrator, DBA |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Stop replication. | Discontinue the replication procedure and halt the source application services. | Cloud administrator, DBA |
| Launch the PeopleSoft middle tier. | Launch the target PeopleSoft middle tier application in AWS, and direct it to the recently migrated Amazon RDS for Oracle database.When you access the application, you should notice that all app connections are now established with the Amazon RDS for Oracle database. | DBA, PeopleSoft administrator |
| Turn off the source database. | After you confirm that there are no more connections to the source database, it can be turned off. | DBA |
## Related resources
+ [Getting started with AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_GettingStarted.html)
+ [Best Practices for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html)
+ [Migrating Oracle Databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html)
## Additional information
**Transfer files using Amazon S3**
To transfer the files to Amazon S3, you can use the AWS CLI or the Amazon S3 console. After you transfer the files to Amazon S3, you can use the Amazon RDS for Oracle instance to import the Data Pump files from Amazon S3.
If you choose to transfer the dump file using Amazon S3 integration as an alternate method, perform the follow steps:
1. Create an S3 bucket.
1. Export the data from the source database using Oracle Data Pump.
1. Upload the Data Pump files to the S3 bucket.
1. Download the Data Pump files from the S3 bucket to the target Amazon RDS for Oracle database.
1. Perform the import using the Data Pump files.
**Note**
To transfer large data files between S3 and RDS instances, it is recommended to use the Amazon S3 Transfer Acceleration feature.
**Activate supplemental logging**
If you receive a warning message to enable [supplemental logging](https://docs.oracle.com/database/121/SUTIL/GUID-D2DDD67C-E1CC-45A6-A2A7-198E4C142FA3.htm#SUTIL1583) in the source database for on-going replication, use the following steps.
```
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (UNIQUE) COLUMNS;
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (FOREIGN KEY) COLUMNS;
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (UNIQUE) COLUMNS;
```
# Migrate an on-premises MySQL database to Amazon RDS for MySQL
*Lorenzo Mota, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an on-premises MySQL database to Amazon Relational Database Service (Amazon RDS) for MySQL. The pattern discusses the use of AWS Database Migration Service (AWS DMS) or native MySQL tools such as **mysqldump** for a full database migration. This pattern is primarily for DBAs and solution architects. It can be used in small or large projects as a testing procedure (we recommend at least one testing cycle) or as a final migration procedure.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A MySQL source database in an on-premises data center
**Limitations**
+ Database size limit: [64 TB](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html)
**Product versions**
+ MySQL versions 5.5, 5.6, 5.7, 8.0. For the latest list of supported versions, see [MySQL on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html) in the AWS documentation. If you're using AWS DMS, see also [Using a MySQL-Compatible Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html) for MySQL versions currently supported by AWS DMS.
## Architecture
**Source technology stack**
+ An on-premises MySQL database
**Target technology stack**
+ An Amazon RDS DB instance running MySQL
**Target architecture**
The following diagram shows the target Amazon RDS for MySQL implementation after migration.
![\[Target Amazon RDS for MySQL implementation after cutover.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/808809dd-030f-42af-a5a7-c4ba40456193/images/2e10114e-e389-4d24-9b6a-fa56beee5369.png)
**AWS data migration architecture**
**Using AWS DMS:**
The following diagram shows the data migration architecture when you use AWS DMS to send full and incremental changes until cutover. The network connection from on premises to AWS depends on your requirements and is out of scope for this pattern.
![\[Data migration architecture to AWS when you use AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/808809dd-030f-42af-a5a7-c4ba40456193/images/ecc9b282-1897-4971-99ed-83223b17000d.png)
** Using native MySQL tools:**
The following diagram shows the data migration architecture when you use native MySQL tools. The export dump files are copied to Amazon Simple Storage Service (Amazon S3) and imported into the Amazon RDS for MySQL database in AWS before the cutover. The network connection from on premises to AWS depends on your requirements and is out of scope for this pattern.
![\[Data migration to AWS architecture when you use native MySQL tools.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/808809dd-030f-42af-a5a7-c4ba40456193/images/3bbec989-c3eb-473e-ba4a-032d6a4271c5.png)
**Notes:**
+ Depending on downtime requirements and the size of the database, using AWS DMS or a change data capture (CDC) tool minimizes cutover time. AWS DMS can help reduce cutover time to the new target to a minimum (typically minutes). An offline strategy with **mysqldump **can suffice if the size of the database and network latency allow for a short window. (We recommend testing to get an approximate time.)
+ Usually a CDC strategy such as AWS DMS requires more monitoring and complexity than offline options.
## Tools
+ **AWS services**: [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores to the AWS Cloud or between combinations of cloud and on-premises setups. For information about MySQL source and target databases supported by AWS DMS, see [Migrating MySQL-Compatible Databases to AWS](https://docs.aws.amazon.com/dms/latest/sbs/CHAP_MySQL.html). If your source database isn't supported by AWS DMS, you must choose another method to migrate your data.
+ **Native MySQL tools**: [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html)
+ **Third-party tools**: [Percona XtraBackup](https://www.percona.com/software/mysql-database/percona-xtrabackup)
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate database versions. | Validate the source and target database versions. | DBA |
| Identify hardware requirements. | Identify the hardware requirements for the target server. | DBA, Systems administrator |
| Identify storage requirements. | Identify storage requirements (such as storage type and capacity) for the target database. | DBA, Systems administrator |
| Choose the instance type. | Choose the target instance type based on capacity, storage features, and networking features. | DBA, Systems administrator |
| Identify network access requirements. | Identify the security requirements for network access for the source and target databases. | DBA, Systems administrator |
| Identify unsupported objects. | Identify unsupported objects (if any) and determine the migration effort. | DBA |
| Identify dependencies. | Identify any dependencies on remote databases. | DBA |
| Determine the application migration strategy. | Determine the strategy for migrating client applications. | DBA, App owner, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | Configure route tables, internet gateway, NAT gateways, and subnets. For more information, see [VPCs and Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.html) in the Amazon RDS documentation. | Systems administrator |
| Create security groups. | Configure ports and CIDR ranges or specific IPs depending on your requirements. The default port for MySQL is 3306. For more information, see [Controlling access with security groups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) in the Amazon RDS documentation. | Systems administrator |
| Configure and start an Amazon RDS for MySQL DB instance. | For instructions, see [Creating an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateInstance.html) in the Amazon RDS documentation. Check for supported versions. | Systems administrator |
### Migrate data ‒ option 1 (using native tools)
| Task | Description | Skills required |
| --- | --- | --- |
| Use native MySQL tools or third-party tools to migrate database objects and data. | For instructions, see the documentation for MySQL tools such as [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) and [Percona XtraBackup](https://www.percona.com/software/mysql-database/percona-xtrabackup) (for physical migration). For more information about options, see the blog post [Migration options for MySQL to Amazon RDS for MySQL or Amazon Aurora MySQL](https://aws.amazon.com/blogs/database/migration-options-for-mysql-to-amazon-rds-for-mysql-or-amazon-aurora-mysql/). | DBA |
### Migrate data ‒ option 2 (using AWS DMS)
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate data with AWS DMS. | For instructions, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html). | DBA |
### Perform preliminary tasks before cutover
| Task | Description | Skills required |
| --- | --- | --- |
| Fix object count discrepancies. | Collect object counts from the source database and new target database. Fix discrepancies in the target database. | DBA |
| Check dependencies. | Check whether dependencies (links) to and from other databases are valid and work as expected. | DBA |
| Perform tests. | If this is a testing cycle, perform query testing, gather metrics, and fix issues. | DBA |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch to the target database. | Switch client applications to the new infrastructure. | DBA, App owner, Systems administrator |
| Provide testing support. | Provide support for functional application testing. | DBA |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down resources. | Shut down the temporary AWS resources you created for the migration. | DBA, Systems administrator |
| Validate project documents. | Review and validate the project documents. | DBA, App owner, Systems administrator |
| Gather metrics. | Gather metrics such as time to migrate, percentage of manual versus automated effort, cost savings, and so on. | DBA, App owner, Systems administrator |
| Close out the project. | Close out the project and provide feedback. | DBA, App owner, Systems administrator |
| Decommission the source database. | When all migration and cutover tasks are complete, decommission the on-premises database. | DBA, Systems administrator |
## Related resources
**References**
+ [Migration strategy for relational databases](https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-database-migration/welcome.html)
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/)
+ [Amazon RDS documentation](https://docs.aws.amazon.com/rds/)
+ [Amazon RDS pricing](https://aws.amazon.com/rds/pricing/)
+ [Amazon VPC and Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.html)
+ [Amazon RDS Multi-AZ deployments](https://aws.amazon.com/rds/details/multi-az/)
+ [Migrate on-premises MySQL databases to Aurora MySQL using Percona XtraBackup, Amazon EFS, and Amazon S3](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-on-premises-mysql-databases-to-aurora-mysql-using-percona-xtrabackup-amazon-efs-and-amazon-s3.html)
+ [Amazon RDS DB instance storage](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html)
**Tutorials**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
# Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server
*Henrique Lobao, Jonathan Pereira Cruz, and Vishal Singh, Amazon Web Services*
## Summary
This pattern provides guidance for migrating from an on-premises Microsoft SQL Server database to Amazon Relational Database Service (Amazon RDS) for SQL Server. It describes two options for migration: using AWS Data Migration Service (AWS DMS) or using native Microsoft SQL Server tools such as Copy Database Wizard.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source Microsoft SQL Server database in an on-premises data center
**Limitations**
+ Database size limit: 16 TB
**Product versions**
+ For the latest list of supported versions and features, see [Microsoft SQL Server on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.FeatureSupport) in the AWS documentation. If you're using AWS DMS, see also [Using a Microsoft SQL Server Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.SQLServer.html) for SQL Server versions supported by AWS DMS.
## Architecture
**Source technology stack**
+ An on-premises Microsoft SQL Server database
**Target technology stack**
+ An Amazon RDS for SQL Server DB instance
**Source and target architecture**
*Using AWS DMS: *
![\[Architecture for migration from on-premises SQL Server to Amazon RDS using AWS DMS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/27942833-c294-405c-90e6-32cc197e36ee/images/69b9877c-2d56-4d64-8475-a3dae789c5de.png)
*Using native SQL Server tools: *
![\[Architecture for migration from on-premises SQL Server to Amazon RDS using SQL Server tools\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/27942833-c294-405c-90e6-32cc197e36ee/images/45ee14e4-3c7e-4b35-a2c9-3e8e3c7e6cee.png)
## Tools
+ [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) supports several types of source and target databases. For details, see [AWS DMS Step-by-Step Walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html). If AWS DMS doesn't support the source database, select another method for migrating the data.
+ Native Microsoft SQL Server tools include backup and restore, Copy Database Wizard, copy and attach database.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database version and engine. | | DBA |
| Identify the hardware requirements for the target server instance. | | DBA, Systems administrator |
| Identify the storage requirements (storage type and capacity). | | DBA, Systems administrator |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, Systems administrator |
| Identify the network access security requirements for source and target databases. | | DBA, Systems administrator |
| Identify the application migration strategy. | | DBA, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | | Systems administrator |
| Create security groups. | | Systems administrator |
| Configure and start an Amazon RDS DB instance. | | DBA, Systems administrator |
### Migrate data - option 1
| Task | Description | Skills required |
| --- | --- | --- |
| Use native SQL Server tools or third-party tools to migrate database objects and data. | | DBA |
### Migrate data - option 2
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate data with AWS DMS. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | | DBA, App owner, Systems administrator |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | | DBA, App owner, Systems administrator |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary AWS resources. | | DBA, Systems administrator |
| Review and validate the project documents. | | DBA, App owner, Systems administrator |
| Gather metrics such as time to migrate, percentage of manual versus automated tasks, and cost savings. | | DBA, App owner, Systems administrator |
| Close out the project and provide feedback. | | DBA, App owner, Systems administrator |
## Related resources
**References**
+ [Deploying Microsoft SQL Server on Amazon Web Services](https://d1.awsstatic.com/whitepapers/RDS/Deploying_SQLServer_on_AWS.pdf)
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [Amazon RDS Pricing](https://aws.amazon.com/rds/pricing/)
+ [Microsoft Products on AWS](https://aws.amazon.com/windows/products/)
+ [Microsoft Licensing on AWS](https://aws.amazon.com/windows/resources/licensing/)
+ [Microsoft SQL Server on AWS](https://aws.amazon.com/windows/products/sql/)
+ [Using Windows Authentication with a Microsoft SQL Server DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServerWinAuth.html)
+ [Amazon RDS Multi-AZ Deployments](https://aws.amazon.com/rds/details/multi-az/)
**Tutorials and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
+ [AWS DMS (video)](https://www.youtube.com/watch?v=zb4GcjEdl8U)
+ [Amazon RDS (video)](https://www.youtube.com/watch?v=igRfulrrYCo)
# Migrate data from Microsoft Azure Blob to Amazon S3 by using Rclone
*Suhas Basavaraj, Aidan Keane, and Corey Lane, Amazon Web Services*
## Summary
This pattern describes how to use [Rclone](https://rclone.org/) to migrate data from Microsoft Azure Blob object storage to an Amazon Simple Storage Service (Amazon S3) bucket. You can use this pattern to perform a one-time migration or an ongoing synchronization of the data. Rclone is a command-line program written in Go and is used to move data across various storage technologies from cloud providers.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Data stored in Azure Blob container service
## Architecture
**Source technology stack**
+ Azure Blob storage container
**Target technology stack**
+ Amazon S3 bucket
+ Amazon Elastic Compute Cloud (Amazon EC2) Linux instance
**Architecture**
![\[Migrating data from Microsoft Azure to Amazon S3\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6ead815d-7768-4726-b27d-97a70cd21081/images/abe69eee-632f-4ca2-abf6-3223f3f3ec94.png)
## Tools
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Rclone](https://rclone.org/) is an open-source command-line program inspired by **rsync**. It is used to manage files across many cloud storage platforms.
## Best practices
When you migrate data from Azure to Amazon S3, be mindful of these considerations to avoid unnecessary costs or slow transfer speeds:
+ Create your AWS infrastructure in the same geographical Region as the Azure storage account and Blob container—for example, AWS Region `us-east-1` (N. Virginia) and Azure region `East US`.
+ Avoid using NAT Gateway if possible, because it accrues data transfer fees for both ingress and egress bandwidth.
+ Use a [VPC gateway endpoint for Amazon S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) to increase performance.
+ Consider using an AWS Graviton2 (ARM) processor-based EC2 instance for lower cost and higher performance over Intel x86 instances. Rclone is heavily cross-compiled and provides a precompiled ARM binary.
## Epics
### Prepare AWS and Azure cloud resources
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare a destination S3 bucket. | [Create a new S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the appropriate AWS Region or choose an existing bucket as the destination for the data you want to migrate. | AWS administrator |
| Create an IAM instance role for Amazon EC2. | [Create a new AWS Identity and Access Management (IAM) role for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#working-with-iam-roles). This role gives your EC2 instance write access to the destination S3 bucket. | AWS administrator |
| Attach a policy to the IAM instance role. | Use the IAM console or AWS Command Line Interface (AWS CLI) to create an inline policy for the EC2 instance role that allows write access permissions to the destination S3 bucket. For an example policy, see the [Additional information](#migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone-additional) section. | AWS administrator |
| Launch an EC2 instance. | Launch an Amazon Linux EC2 instance that is configured to use the newly created IAM service role. This instance will also need access to Azure public API endpoints through the internet. Consider using [AWS Graviton-based EC2 instances](https://docs.aws.amazon.com/compute-optimizer/latest/ug/graviton-recommendations.html) to lower costs. Rclone provides ARM-compiled binaries. | AWS administrator |
| Create an Azure AD service principal. | Use the Azure CLI to create an Azure Active Directory (Azure AD) service principal that has read-only access to the source Azure Blob storage container. For instructions, see the [Additional information](#migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone-additional) section. Store these credentials on your EC2 instance to the location `~/azure-principal.json`. | Cloud administrator, Azure |
### Install and configure Rclone
| Task | Description | Skills required |
| --- | --- | --- |
| Download and install Rclone. | Download and install the Rclone command-line program. For installation instructions, see the [Rclone installation documentation](https://rclone.org/install/). | General AWS, Cloud administrator |
| Configure Rclone. | Copy the following `rclone.conf` sample file. Replace `AZStorageAccount` with your Azure Storage account name and `us-east-1` with the AWS Region where your S3 bucket is located. Save this file to the location `~/.config/rclone/rclone.conf` on your EC2 instance.
[AZStorageAccount] type = azureblob account = AZStorageAccount service_principal_file = azure-principal.json
[s3] type = s3 provider = AWS env_auth = true region = us-east-1
| General AWS, Cloud administrator |
| Verify Rclone configuration. | To confirm that Rclone is configured and permissions are working properly, verify that Rclone can parse your configuration file and that objects inside your Azure Blob container and S3 bucket are accessible. See the following for example validation commands.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone.html) | General AWS, Cloud administrator |
### Migrate data using Rclone
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate data from your containers. | Run the Rclone [copy](https://rclone.org/commands/rclone_copy/) or [sync](https://rclone.org/commands/rclone_sync/) command. **Example: copy**This command copies data from the source Azure Blob container to the destination S3 bucket.
When you use the **sync **command, data that isn't present in the source container will be deleted from the destination S3 bucket. | General AWS, Cloud administrator |
| Synchronize your containers. | After the initial copy is complete, run the Rclone **sync** command for ongoing migration so that only new files that are missing from the destination S3 bucket will be copied. | General AWS, Cloud administrator |
| Verify that data has been migrated successfully. | To check that data was successfully copied to the destination S3 bucket, run the Rclone [lsd](https://rclone.org/commands/rclone_lsd/) and [ls](https://rclone.org/commands/rclone_ls/) commands. | General AWS, Cloud administrator |
## Related resources
+ [Amazon S3 User Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) (AWS documentation)
+ [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) (AWS documentation)
+ [Creating a Microsoft Azure Blob container](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal) (Microsoft Azure documentation)
+ [Rclone commands](https://rclone.org/commands/) (Rclone documentation)
## Additional information
**Example role policy for EC2 instances**
This policy gives your EC2 instance read and write access to a specific bucket in your account. If your bucket uses a customer managed key for server-side encryption, the policy might need additional access to AWS Key Management Service (AWS KMS) .
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::amzn-s3-demo-bucket/*",
"arn:aws:s3:::amzn-s3-demo-bucket"
]
},
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
}
]
}
```
**Creating a read-only Azure AD service principal**
An Azure service principal is a security identity that is used by customer applications, services, and automation tools to access specific Azure resources. Think of it as a user identity (login and password or certificate) with a specific role and tightly controlled permissions to access your resources. To create a read-only service principal to follow least privilege permissions and protect data in Azure from accidental deletions, follow these steps:
1. Log in to your Microsoft Azure cloud account portal and launch Cloud Shell in PowerShell or use the Azure Command-Line Interface (CLI) on your workstation.
1. Create a service principal and configure it with [read-only](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-reader) access to your Azure Blob storage account. Save the JSON output of this command to a local file called `azure-principal.json`. The file will be uploaded to your EC2 instance. Replace the placeholder variables that are shown in braces (`{` and `}`) with your Azure subscription ID, resource group name, and storage account name.
```
az ad sp create-for-rbac `
--name AWS-Rclone-Reader `
--role "Storage Blob Data Reader" `
--scopes /subscriptions/{Subscription ID}/resourceGroups/{Resource Group Name}/providers/Microsoft.Storage/storageAccounts/{Storage Account Name}
```
# Migrate from Couchbase Server to Couchbase Capella on AWS
*Battulga Purevragchaa and Saurabh Shanbhag, Amazon Web Services*
*Mark Gamble, None*
## Summary
Couchbase Capella is a fully managed, NoSQL database as a service (DBaaS) for mission-critical applications (for example, user profiles or online catalogs and inventory management). Couchbase Capella manages your DBaaS workload in a Couchbase-managed Amazon Web Services (AWS) account. Capella makes it easy to run and manage multiple-cluster, multiple–AWS Region, multicloud, and hybrid-cloud replication within a single interface.
Couchbase Capella helps you instantly scale your Couchbase Server applications, helping you create multi-node clusters in minutes. Couchbase Capella supports all Couchbase Server features, including [SQL\$1\$1](https://www.couchbase.com/products/n1ql), [Full Text Search](https://www.couchbase.com/products/full-text-search), [Eventing Service](https://docs.couchbase.com/server/current/eventing/eventing-overview.html), and [Analytics Service](https://www.couchbase.com/products/analytics). It also removes the need to manage installations, upgrades, backups, and general database maintenance.
This pattern describes the steps and best practices for migrating a self-managed [Couchbase Server](https://www.couchbase.com/products/server) environment to the AWS Cloud. The pattern provides a repeatable process for migrating data and indexes from Couchbase Server clusters, running either on premises or in the cloud, to Couchbase Capella. Using these steps helps you avoid issues during your migration and speeds up your overall migration process.
This pattern provides the following two migration options:
+ **Option 1** is appropriate if you have fewer than 50 indexes to migrate.
+ **Option 2** is appropriate if you have more than 50 indexes to migrate.
You can also [set up sample data](https://docs.couchbase.com/server/current/manage/manage-settings/install-sample-buckets.html) on your self-managed Couchbase Server to follow along with the migration guide.
If you choose migration **option 2**, or if you are using scopes or collections other than the default value, you must use the example configuration file, which is in the *Additional information* section.
## Prerequisites and limitations
**Prerequisites **
+ An existing Couchbase Capella paid account. You can also create a [Couchbase Capella account on AWS](https://aws.amazon.com/marketplace/pp/prodview-xrhx5zgue5c26) and use the Couchbase Capella free trial, and then upgrade to a paid account to configure your cluster for the migration.. To start with the trial version, follow the instructions in [Getting Started with Couchbase Capella](https://docs.couchbase.com/cloud/get-started/create-account.html).
+ An existing self-managed Couchbase Server environment either on premises or deployed on a cloud service provider.
+ For migration option 2, Couchbase Shell and a configuration file. To create the configuration file, you can use the example file that’s in the *Additional information* section.
+ Familiarity with administering Couchbase Server and Couchbase Capella.
+ Familiarity with opening TCP ports and running commands in a command line interface (CLI).
The migration process also requires the roles and expertise described in the following table.
|
|
| Role | Expertise | Responsibilities |
| --- |--- |--- |
| Couchbase administrator | Familiarity with Couchbase Server and Couchbase CapellaBasic command line knowledge is helpful but not required | Couchbase Server and Capella–specific tasks |
| Systems administrator, IT administrator | Familiarity with the self-managed Couchbase Server system environment and administration | Opening ports and determining IP addresses on self-managed Couchbase Server cluster nodes |
**Limitations **
+ This pattern is used to migrate data, indexes, and [Couchbase Full Text Search](https://docs.couchbase.com/server/current/fts/full-text-intro.html) indexes from Couchbase Server to Couchbase Capella on AWS. The pattern doesn’t apply to migrating [Couchbase Eventing Service](https://docs.couchbase.com/server/current/eventing/eventing-overview.html), or to [Couchbase Analytics](https://docs.couchbase.com/server/current/analytics/introduction.html).
+ Couchbase Capella is available in multiple AWS Regions. For up-to-date information on the Regions that Capella supports, see [Amazon Web Services](https://docs.couchbase.com/cloud/reference/aws.html) in the Couchbase documentation.
**Product versions**
+ [Couchbase Server (Community or Enterprise) Edition version 5.x or later](https://docs.couchbase.com/server/current/release-notes/relnotes.html)
## Architecture
**Source technology stack**
+ Couchbase Server
**Target technology stack**
+ Couchbase Capella
**Target architecture**
![\[Couchbase Capella migration to Couchbase cluster in the Capella data plane on AWS in four steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/14ac5a81-eade-4708-9335-f5602fa07824/images/95cd7f33-742e-4d10-8e2c-37c7b4d9df45.png)
1. You access Couchbase Capella by using the **Capella Control Plane**. You can use the Capella Control Plane to do the following:
+ Control and monitor your account.
+ Manage clusters and data, indexes, users and groups, access permissions, monitoring, and events.
1. Clusters are created.
1. The **Capella Data Plane** is in the Couchbase-managed AWS account. After you create a new cluster, Couchbase Capella deploys it across multiple Availability Zones in the selected AWS Region.
1. You can develop and deploy Couchbase applications in a VPC in your AWS account. Typically, this VPC accesses the Capella Data Plane through [VPC peering](https://docs.couchbase.com/cloud/clouds/private-network.html).
## Tools
+ [Couchbase Cross Data Center Replication (XDCR)](https://docs.couchbase.com/cloud/current/clusters/xdcr/xdcr.html) helps replicate data across clusters that are located in different cloud providers and different data centers. It is used to migrate data into Couchbase Capella from self-managed Couchbase Server clusters.
**Note**
XDCR cannot be used with Couchbase Server Community Edition to migrate to Couchbase Capella. Instead, you can use [cbexport](https://docs.couchbase.com/server/current/tools/cbexport.html). For more information, see the *Migrate data from Community Edition* epic.
+ [Couchbase Shell](https://couchbase.sh/docs/) is a command line shell for Couchbase Server and Couchbase Capella to access local and remote Couchbase clusters. In this pattern, Couchbase Shell is used to migrate indexes.
+ [cbexport](https://docs.couchbase.com/server/current/tools/cbexport.html) is a Couchbase utility for exporting data from Couchbase cluster. Included in [Couchbase Server CLI tools](https://docs.couchbase.com/server/current/cli/cli-intro.html).
## Epics
### Prepare the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Evaluate the size of the self-managed Couchbase Server cluster. | Log in to the [Couchbase Web Console](https://docs.couchbase.com/server/current/manage/manage-ui/manage-ui.html) for Couchbase Server, and assess your self-managed cluster’s nodes and buckets. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html)You will use your self-managed Couchbase Server cluster configurations as a general guide for sizing and configuring the target cluster on Couchbase Capella.For help with a more detailed Couchbase Capella sizing exercise, [contact Couchbase](https://www.couchbase.com/contact). | Couchbase administrator |
| Record Couchbase Service distribution on the self-managed Couchbase Server cluster. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
| Record the IP addresses of the self-managed Couchbase Server cluster nodes. | (Ignore this step if you are using Community Edition.) Record the IP address for each node in your cluster. They will be added to the allow list on your Couchbase Capella cluster later. | Couchbase administrator, Systems administrator |
### Deploy and configure resources on Couchbase Capella
| Task | Description | Skills required |
| --- | --- | --- |
| Choose a template. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
| Choose and configure the nodes. | Choose and configure the nodes to match your self-managed Couchbase Server cluster environment, including the number of nodes, services distribution, compute or RAM, and storage.Couchbase Capella uses [multidimensional scaling](https://docs.couchbase.com/cloud/clusters/scale-cluster.html#scale-a-cluster) best practices. Services and nodes can be chosen only according to deployment best practices. This might mean that you can’t exactly match your self-managed Couchbase Server cluster’s configurations. | Couchbase administrator |
| Deploy the cluster. | Choose a support zone and support package, and then deploy the cluster. For detailed steps and instructions, see [Create a cluster](https://docs.couchbase.com/cloud/clusters/create-cluster.html) in the Couchbase documentation.If you are using the Couchbase Capella free trial, you must convert it to a paid account before beginning your migration. To convert your account, open the **Billing** section of the Couchbase Capella Control Plane, and then choose **Add Activation ID**. The Activation ID is sent to your billing contact email address after you complete a purchase agreement with Couchbase Sales, or after you make a purchase through [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-xrhx5zgue5c26). | Couchbase administrator |
| Create a database credential user. | A database credential user is specific to a cluster and consists of a user name, password, and a set of bucket privileges. This user is required for creating buckets and accessing bucket data. In the Couchbase Capella Control Plane, create a database credential for the new cluster by following the instructions in [Configure database credentials](https://docs.couchbase.com/cloud/clusters/manage-database-users.html) in the Couchbase Capella documentation.An organization user needs organizational role credentials assigned to them if they want to access bucket data on a particular cluster, either remotely or through the Couchbase Capella UI. This is separate from database credentials, which are typically used by apps and integrations. Creating the organizational user allows you to create and manage the target buckets on your Couchbase Capella cluster. | Couchbase administrator |
| If using migration option 2, install Couchbase Shell. | You can install Couchbase Shell on any system that has network access to both your self-managed Couchbase Server and the Couchbase Capella clusters. For more information, see [Install Couchbase Shell version 1.0.0-beta.5](https://couchbase.sh/docs/#_installation) in the Couchbase Shell documentation.Confirm that Couchbase Shell is installed by [testing a connection to your self-managed cluster](https://couchbase.sh/docs/#_connecting_to_a_cluster) in a command line terminal. | Couchbase administrator, Systems administrator |
| Allow IP addresses. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html)For more information about allowed IP addresses, see [Configure allowed IP addresses](https://docs.couchbase.com/cloud/get-started/configure-cluster-access.html#allow-ip-address) in the Couchbase documentation. | Couchbase administrator, Systems administrator |
| Configure certificates. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator, Systems administrator |
| Create the configuration file for Couchbase Shell. | Create a configuration dotfile in the Couchbase Shell installation’s home directory (for example, `//.cbsh/config`). For more information, see [Config dotfiles](https://couchbase.sh/docs/#_the_config_dotfiles) in the Couchbase documentation.Add connection properties for the source and target clusters to the configuration file. You can use the example configuration file that’s in the *Additional information* section and edit the settings for your clusters. Save the configuration file with the updated settings to the `.cbsh` folder (for example, `//.cbsh/config`). | Couchbase administrator, Systems administrator |
| Create target buckets. | For each source bucket, create one target bucket in your Couchbase Capella cluster by following the instructions in [Create a bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket) in the Couchbase documentation.Your target bucket configurations must match the bucket names, memory settings, and conflict resolution settings of the buckets in your self-managed Couchbase Server cluster. | Couchbase administrator |
| Create scopes and collections. | Every bucket contains a default scope and collection with the keyspace `_default._default`. If you are using any other keyspaces for your scope and collection, you must create identical keyspaces in the target Capella cluster.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html)
| Couchbase administrator |
### Migrate the data from Enterprise Edition
| Task | Description | Skills required |
| --- | --- | --- |
| Open TCP ports on the self-managed Couchbase Server cluster nodes. | Make sure that the appropriate ports are open for XDCR communication on the self-managed Couchbase Server cluster's nodes. For more information, see the [Couchbase Server ports documentation](https://docs.couchbase.com/server/current/install/install-ports.html#ports-listed-by-communication-path). | Couchbase administrator, Systems administrator |
| If you are using Couchbase Server Enterprise Edition, set up Couchbase XDCR. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
| Start Couchbase XDCR. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
### Migrate the indexes by using option 1
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate self-managed cluster indexes to Couchbase Capella. | We recommend this process if you have fewer than 50 indexes to migrate. If you have more than 50 indexes to migrate, we recommend that you use migration option 2.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator, Systems administrator |
### Migrate the indexes by using option 2
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate the index definitions. | We recommend this process if you have more than 50 indexes to migrate. If you have fewer than 50 indexes to migrate, we recommend that you use migration option 1.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator, Systems administrator |
| Build the index definitions. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator, Systems administrator |
### Migrate full-text search indexes
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate self-managed cluster full-text search indexes to Couchbase Capella. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
### Migrate data from Couchbase Community Edition
| Task | Description | Skills required |
| --- | --- | --- |
| Export data from self-managed Couchbase Server Community Edition. | Encrypted XDCR is not available in Couchbase Community Edition. You can export data from Couchbase Community Edition and then manually import the data into Couchbase Capella.To export data from the source bucket, use `cbexport` at the command line.The following command is provided as an example.
Note that `cbkey`, `cbscope`, `cbcoll`, and `cbexported_data.json` are arbitrary labels. They will be referenced later in the process, so if you choose to name them differently, make note of it. | Couchbase administrator |
| Import data into Couchbase Capella. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html)For large files, Couchbase Capella supports command line import using cURL. You can explore Import options in more detail at [Import data](https://docs.couchbase.com/cloud/clusters/data-service/import-data-documents.html) in the Couchbase Capella documentation. | Couchbase administrator |
### Test and verify the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Verify data migration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
| Verify index migration. | In the Couchbase Capella Control Plane, in the **Tools** dropdown list for your target cluster, choose **Indexes**. Verify that the indexes are migrated and built. | Couchbase administrator |
| Verify query results. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
| Verify full-text search results (applicable if you migrated FTS indexes). | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-couchbase-server-to-couchbase-capella-on-aws.html) | Couchbase administrator |
## Related resources
**Prepare the migration**
+ [Get started with the Couchbase Capella free trial](https://cloud.couchbase.com/sign-up)
+ [Cloud provider requirements for Couchbase Capella](https://docs.couchbase.com/cloud/reference/aws.html)
+ [Couchbase Capella sizing guidelines](https://docs.couchbase.com/cloud/clusters/sizing.html)
**Migrate the data and indexes**
+ [Couchbase XDCR](https://docs.couchbase.com/cloud/clusters/xdcr/xdcr.html)
+ [Couchbase Shell documentation](https://couchbase.sh/docs/)
**Couchbase Capella SLAs and support**
+ [Couchbase Capella service-level agreements (SLAs](https://www.couchbase.com/capellasla))
+ [Couchbase Capella Service support policy](https://www.couchbase.com/support-policy/cloud)
## Additional information
The following code is an example [configuration file for Couchbase Shell](https://couchbase.sh/docs/#_the_config_dotfiles).
```
Version = 1
[[clusters]]
identifier = "On-Prem-Cluster"
hostnames = [""]
default-bucket = "travel-sample"
username = ""
password = ""
tls-cert-path = "/"
data-timeout = "2500ms"
connect-timeout = "7500ms"
query-timeout = "75s"
[[clusters]]
identifier = "Capella-Cluster"
hostnames = [""]
default-bucket = "travel-sample"
username = ""
password = ""
tls-cert-path = "/"
data-timeout = "2500ms"
connect-timeout = "7500ms"
query-timeout = "75s"
```
Before you save the configuration file, use the following table to make sure that you added your own source and target cluster information.
| | |
| --- |--- |
| | Use the IP address for your self-managed Couchbase Server cluster. |
| | Use the administrator user for your self-managed Couchbase Server cluster. |
| | Use the absolute path to the saved root certificate file for your self-managed Couchbase Server cluster. |
| | Use the connection endpoint for your Couchbase Capella cluster. |
| | Use the database user for your Couchbase Capella cluster. |
| | Use the database user password for your Couchbase Capella cluster. |
| | Use the absolute path to the saved root certificate file for your Couchbase Capella cluster. |
# Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2
*Neal Ardeljan and Afroz Khan, Amazon Web Services*
## Summary
This pattern walks you through the steps for migrating from an on-premises Red Hat Enterprise Linux (RHEL) 6.9 or later system that’s running IBM WebSphere Application Server (WAS) to RHEL 8 running Apache Tomcat on an Amazon Elastic Compute Cloud (Amazon EC2) instance.
The pattern can be applied to the following source and target versions:
+ WebSphere Application Server 7.x to Apache Tomcat 8 (with Java 7 or later)
+ WebSphere Application Server 8.x to Apache Tomcat 8 (with Java 7 or later)
+ WebSphere Application Server 8.5.5.x to Apache Tomcat 9 (with Java 8 or later)
+ WebSphere Application Server 8.5.5.x to Apache Tomcat 10 (with Java 8 or later)
## Prerequisites and limitations
**Prerequisites**** **
+ An active AWS account
+ Source Java code, with the following assumptions:
+ Uses the Java Development Kit (JDK) version of Java 7 or later
+ Uses the Spring or Apache Struts framework
+ Doesn't use the Enterprise Java Beans (EJB) framework or any other WebSphere server functionality that's not readily available for Tomcat
+ Primarily uses servlets or Java Server Pages (JSPs)
+ Uses Java Database Connectivity (JDBC) connectors to connect to databases
+ Source IBM WebSphere Application Server version 7.x or higher
+ Target Apache Tomcat version 8.5 or higher
## Architecture
** ****Source technology stack**
+ A web application built using the Apache Struts Model-View-Controller (MVC) framework
+ A web application running on IBM WebSphere Application Server version 7.x or 8.x
+ A web application that uses a Lightweight Directory Access Protocol (LDAP) connector to connect to an LDAP directory (iPlanet/eTrust)
+ An application that uses IBM Tivoli Access Manager (TAM) connectivity to update the TAM user password (in the present implementation, applications use PD.jar)
** ****On-premises databases**
+ Oracle Database 21c (21.0.0.0)
+ Oracle Database 19c (19.0.0.0)
+ Oracle Database 12c Release 2 (12.2.0.1)
+ Oracle Database 12c Release 1 (12.1.0.2)
**Target technology stack**
+ Apache Tomcat version 8 (or later) running on RHEL on an EC2 instance
+ Amazon Relational Database Service (Amazon RDS) for Oracle
For more information about the Oracle versions supported by Amazon RDS, see the [Amazon RDS for Oracle](https://aws.amazon.com/rds/oracle/) website.
**Target architecture**
![\[Architecture for migrating from IBM WebSphere to Apache Tomcat on Amazon EC2\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/96f91201-e0a6-4d3f-a94e-7bd68a59cc4e/images/11afe7c0-b400-423b-9dfe-02a915fe47ff.png)
## Tools
+ Application tier: Rebuilding Java application into a WAR file.
+ Database tier: Oracle native backup and restore.
+ Apache Tomcat migration tool for Jakarta EE. This tool takes a web application written for Java EE 8 that runs on Apache Tomcat 9 and converts it automatically to run on Apache Tomcat 10, which implements Jakarta EE 9.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Complete the application discovery, current state footprint, and performance baseline. | | BA, Migration Lead |
| Validate the source and target database versions. | | DBA |
| Identify the hardware requirements for the target server EC2 instance. | | DBA, SysAdmin |
| Identify storage requirements (storage type and capacity). | | DBA, SysAdmin |
| Choose the proper EC2 instance type based on capacity, storage features, and network features. | | DBA, SysAdmin |
| Identify the network access security requirements for the source and target databases. | | DBA, SysAdmin |
| Identify the application migration strategy and tooling. | | DBA, Migration Lead |
| Complete the migration design and migration guide for the application. | | Build Lead, Migration Lead |
| Complete the application migration runbook. | | Build Lead, Cutover Lead, Testing Lead, Migration Lead |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | | SysAdmin |
| Create the security groups. | | SysAdmin |
| Configure and start Amazon RDS for Oracle. | | DBA, SysAdmin |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Create or obtain access to the endpoints to fetch the database backup files. | | DBA |
| Use the native database engine or a third-party tool to migrate database objects and data. | For details, see "Migrating database objects and data" in the *Additional information *section. | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Lodge the change request (CR) for migration. | | Cutover Lead |
| Obtain the CR approval for migration. | | Cutover Lead |
| Follow the application migration strategy per the application migration runbook. | For details, see "Setting up the application tier" in the *Additional information* section. | DBA, Migration Engineer, App owner |
| Upgrade the application (if necessary). | | DBA, Migration Engineer, App owner |
| Complete the functional, non-functional, data validation, SLA, and performance tests. | | Testing Lead, App Owner, App Users |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Obtain signoff from the application owner or business owner. | | Cutover Lead |
| Switch the application clients to the new infrastructure. | | DBA, Migration Engineer, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary AWS resources. | | DBA, Migration Engineer, SysAdmin |
| Review and validate the project documents. | | Migration Lead |
| Gather metrics such as time to migrate, percentage of manual versus automated tasks, and cost savings. | | Migration Lead |
| Close out the project and provide feedback. | | Migration Lead, App Owner |
## Related resources
**References**
+ [Apache Tomcat 10.0 documentation](https://tomcat.apache.org/tomcat-10.0-doc/index.html)
+ [Apache Tomcat 9.0 documentation](https://tomcat.apache.org/tomcat-9.0-doc/index.html)
+ [Apache Tomcat 8.0 documentation](https://tomcat.apache.org/tomcat-8.0-doc)
+ [Apache Tomcat 8.0 installation guide](https://tomcat.apache.org/tomcat-8.0-doc/setup.html)
+ [Apache Tomcat JNDI documentation ](https://tomcat.apache.org/tomcat-8.0-doc/jndi-datasource-examples-howto.html)
+ [Amazon RDS for Oracle website](https://aws.amazon.com/rds/oracle/)
+ [Amazon RDS pricing](https://aws.amazon.com/rds/pricing/)
+ [Oracle and Amazon Web Services](https://aws.amazon.com/oracle/)
+ [Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html)
+ [Amazon RDS Multi-AZ Deployments](https://aws.amazon.com/rds/details/multi-az/)
**Tutorials and videos**
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
## Additional information
**Migrating database objects and data**
For example, if you're using native Oracle backup/restore utilities:
1. Create the Amazon Simple Storage Service (Amazon S3) backup for database backup files (optional).
1. Back up the Oracle DB data to the network shared folder.
1. Log in to the migration staging server to map the network share folder.
1. Copy data from the network share folder to the S3 bucket.
1. Request an Amazon RDS Multi-AZ deployment for Oracle.
1. Restore the on-premises database backup to Amazon RDS for Oracle.
**Setting up the application tier**
1. Install Tomcat 8 (or 9/10) from the Apache Tomcat website.
1. Package the application and shared libraries into a WAR file.
1. Deploy the WAR file in Tomcat.
1. Monitor the start log to `Linux cat` any missing shared libraries from WebSphere.
1. Watch the start record to `Linux cat` any WebSphere-specific deployment descriptor extensions.
1. Collect any missing dependent Java libraries from the WebSphere server.
1. Amend WebSphere-specific deployment descriptor elements with Tomcat-compatible equivalents.
1. Rebuild the WAR file with the dependent Java libraries and updated deployment descriptors.
1. Update the LDAP configuration, database configuration, and test connections (see [Realm Configuration HOW-TO](https://tomcat.apache.org/tomcat-8.0-doc/realm-howto.html) and [JNDI Datasource HOW-TO](https://tomcat.apache.org/tomcat-8.0-doc/jndi-datasource-examples-howto.html) in the Apache Tomcat documentation).
1. Test the installed application against the restored Amazon RDS for Oracle database.
1. Create an Amazon Machine Image (AMI) for Linux from the EC2 instance.
1. Launch the completed architecture with the Application Load Balancer and Auto Scaling group.
1. Update the URLs (by using the WebSEAL junction) to point to the Application Load Balancer.
1. Update the configuration management database (CMDB).
# Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2 with Auto Scaling
*Kevin Yung and Afroz Khan, Amazon Web Services*
## Summary
This pattern provides guidance for migrating a Java application from IBM WebSphere Application Server to Apache Tomcat on an Amazon Elastic Compute Cloud (Amazon EC2) instance with Amazon EC2 Auto Scaling enabled.
By using this pattern, you can achieve:
+ A reduction in IBM licensing costs
+ High availability using Multi-AZ deployment
+ Improved application resiliency with Amazon EC2 Auto Scaling
## Prerequisites and limitations
**Prerequisites**
+ Java applications (version 7.*x* or 8.*x*) should be developed in LAMP stacks.
+ The target state is to host Java applications on Linux hosts. This pattern has been successfully implemented in a Red Hat Enterprise Linux (RHEL) 7 environment. Other Linux distributions can follow this pattern, but the configuration of the Apache Tomcat distribution should be referenced.
+ You should understand the Java application's dependencies.
+ You must have access to the Java application source code to make changes.
**Limitations and replatforming changes**
+ You should understand the enterprise archive (EAR) components and verify that all libraries are packaged in the web component WAR files. You need to configure the [Apache Maven WAR Plugin](https://maven.apache.org/plugins/maven-war-plugin/) and produce WAR file artifacts.
+ When using Apache Tomcat 8, there is a known conflict between servlet-api.jar and the application package built-in jar files. To resolve this issue, delete servlet-api.jar from the application package.
+ You must configure WEB-INF/resources located in the *classpath* of the [Apache Tomcat configuration](https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html). By default, the JAR libraries are not loaded in the directory. Alternatively, you can deploy all the resources under src/main/resources*.*
+ Check for any hard-coded context roots within the Java application, and update the new [context root of Apache Tomcat.](https://tomcat.apache.org/tomcat-8.0-doc/config/context.html#Defining_a_context)
+ To set JVM runtime options, you can create the configuration file setenv.sh in the Apache Tomcat bin folder; for example, JAVA\$1OPTS, JAVA\$1HOME**,** etc.
+ Authentication is configured at the container level and is set up as a realm in Apache Tomcat configurations. Authentication is established for any of the following three realms:
+ [JDBC Database Realm](https://tomcat.apache.org/tomcat-8.0-doc/config/realm.html#JDBC_Database_Realm_-_org.apache.catalina.realm.JDBCRealm) looks up users in a relational database accessed by the JDBC driver.
+ [DataSource Database Realm](https://tomcat.apache.org/tomcat-8.0-doc/config/realm.html#DataSource_Database_Realm_-_org.apache.catalina.realm.DataSourceRealm) looks up users in a database that is accessed by JNDI.
+ [JNDI Directory Realm](https://tomcat.apache.org/tomcat-8.0-doc/config/realm.html#JNDI_Directory_Realm_-_org.apache.catalina.realm.JNDIRealm) looks up users in the Lightweight Directory Access Protocol (LDAP) directory that is accessed by the JNDI provider. The look-ups require:
+ LDAP connection details: user search base, search filter, role base, role filter
+ The key JNDI Directory Realm: Connects to LDAP, authenticates users, and retrieves all groups in which a user is a member
+ Authorization: In the case of a container with a role-based authorization that checks the authorization constraints in web.xml, web resources must be defined and compared to the roles defined in the constraints. If LDAP doesn't have group-role mapping, you must set the attribute in web.xml to achieve group-role mapping. To see an example of a configuration document, see the [Oracle documentation](https://docs.oracle.com/cd/E19226-01/820-7627/bncav/index.html).
+ Database connection: Create a resource definition in Apache Tomcat with an Amazon Relational Database Service (Amazon RDS) endpoint URL and connection details. Update the application code to reference a DataSource by using JNDI lookup. An existing DB connection defined in WebSphere would not work, as it uses WebSphere’s JNDI names. You can add a entry in web.xml with the JNDI name and DataSource type definition. To see a sample configuration document, see the [Apache Tomcat documentation](https://tomcat.apache.org/tomcat-8.0-doc/jndi-resources-howto.html#JDBC_Data_Sources).
+ Logging: By default, Apache Tomcat logs to the console or to a log file. You can enable realm-level tracing by updating *logging.properties* (see [Logging in Tomcat](https://tomcat.apache.org/tomcat-8.0-doc/logging.html)). If you are using Apache Log4j to append logs to a file, you must download tomcat-juli and add it to the *classpath*.
+ Session management: If you are retaining IBM WebSEAL for application load balancing and session management, no change is required. If you are using an Application Load Balancer or Network Load Balancer on AWS to replace the IBM WebSEAL component, you must set up session management by using an Amazon ElastiCache instance with a Memcached cluster and set up Apache Tomcat to use [open-source session management](https://github.com/magro/memcached-session-manager).
+ If you are using the IBM WebSEAL forward proxy, you must set up a new Network Load Balancer on AWS. Use the IPs provided by the Network Load Balancer for WebSEAL junction configurations.
+ SSL configuration: We recommend that you use Secure Sockets Layer (SSL) for end-to-end communications. To set up an SSL server configuration in Apache Tomcat, follow the instructions in the [Apache Tomcat documentation](https://tomcat.apache.org/tomcat-8.0-doc/ssl-howto.html).
## Architecture
**Source technology stack**
+ IBM WebSphere Application Server
**Target technology stack**
+ The architecture uses [Elastic Load Balancing (version 2](https://docs.aws.amazon.com/elasticloadbalancing/)). If you are using IBM WebSEAL for Identify management and load balancing, you can select a Network Load Balancer on AWS to integrate with the IBM WebSEAL reverse proxy.
+ Java applications are deployed to an Apache Tomcat application server, which runs on an EC2 instance in an [Amazon EC2 Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html). You can set up a [scaling policy](https://docs.aws.amazon.com/autoscaling/ec2/userguide/scaling_plan.html) based on Amazon CloudWatch metrics such as CPU utilization.
+ If you're retiring the use of IBM WebSEAL for load balancing, you can use [Amazon ElastiCache for Memcached ](https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/WhatIs.html) for session management.
+ For the back-end database, you can deploy [High Availability (Multi-AZ) for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) and select a database engine type.
**Target architecture**
![\[AWS Cloud architecture with VPC, two availability zones, load balancer, and database components.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/52b91dab-7b3b-4751-abe2-25e7c7cd8d89/images/25125023-9a81-452a-9ada-184e2416cc02.png)
## Tools
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)
+ Apache Tomcat (version 7.*x* or 8.*x*)
+ RHEL 7 or Centos 7
+ [Amazon RDS Multi-AZ deployment](https://aws.amazon.com/rds/details/multi-az/)
+ [Amazon ElastiCache for Memcached](https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/WhatIs.html) (optional)
## Epics
### Set up the VPC
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | | |
| Create subnets. | | |
| Create routing tables if necessary. | | |
| Create network access control lists (ACLs). | | |
| Set up AWS Direct Connect or a corporate VPN connection. | | |
### Replatform the application
| Task | Description | Skills required |
| --- | --- | --- |
| Refactor the application build Maven configuration to generate the WAR artifacts. | | |
| Refactor the application dependency data sources in Apache Tomcat. | | |
| Refactor the application source codes to use JNDI names in Apache Tomcat. | | |
| Deploy the WAR artifacts into Apache Tomcat. | | |
| Complete application validations and tests. | | |
### Configure the network
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the corporate firewall to allow the connection to dependency services. | | |
| Configure the corporate firewall to allow end-user access to Elastic Load Balancing on AWS. | | |
### Create the application infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create and deploy the application on an EC2 instance. | | |
| Create an Amazon ElastiCache for Memcached cluster for session management. | | |
| Create an Amazon RDS Multi-AZ instance for the backend database. | | |
| Create SSL certificates and import them into AWS Certificate Manager (ACM). | | |
| Install SSL certificates on load balancers. | | |
| Install SSL certificates for Apache Tomcat servers. | | |
| Complete application validations and tests. | | |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the existing infrastructure. | | |
| Restore the database from production to Amazon RDS. | | |
| Cut over the application by making DNS changes. | | |
## Related resources
**References**
+ [Apache Tomcat 7.0 documentation](https://tomcat.apache.org/tomcat-7.0-doc/realm-howto.html)
+ [Apache Tomcat 7.0 installation guide](https://tomcat.apache.org/tomcat-7.0-doc/appdev/installation.html)
+ [Apache Tomcat JNDI documentation](https://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html)
+ [Amazon RDS Multi-AZ Deployments](https://aws.amazon.com/rds/details/multi-az/)
+ [Amazon ElastiCache for Memcached](https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/WhatIs.html)
**Tutorials and videos**
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
# Migrate a .NET application from Microsoft Azure App Service to AWS Elastic Beanstalk
*Raghavender Madamshitti, Amazon Web Services*
## Summary
This pattern describes how to migrate a .NET web application hosted on Microsoft Azure App Service to AWS Elastic Beanstalk. There are two ways to migrate applications to Elastic Beanstalk:
+ Use AWS Toolkit for Visual Studio - This plugin for the Microsoft Visual Studio IDE provides the easiest and most straightforward way to deploy custom .NET applications to AWS. You can use this approach to deploy .NET code directly to AWS and to create supporting resources, such as Amazon Relational Database Service (Amazon RDS) for SQL Server databases, directly from Visual Studio.
+ Upload and deploy to Elastic Beanstalk - Each Azure App Service includes a background service called Kudu, which is useful for capturing memory dumps and deployment logs, viewing configuration parameters, and accessing deployment packages. You can use the Kudu console to access Azure App Service content, extract the deployment package, and then upload the package to Elastic Beanstalk by using the upload and deploy option in the Elastic Beanstalk console.
This pattern describes the second approach (uploading your application to Elastic Beanstalk through Kudu). The pattern also uses the following AWS services: AWS Elastic Beanstalk, Amazon Virtual Private Cloud (Amazon VPC), Amazon CloudWatch, Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling, Amazon Simple Storage Service (Amazon S3), and Amazon Route 53.
The .NET web application is deployed to AWS Elastic Beanstalk, which runs in an Amazon EC2 Auto Scaling Group. You can set up a scaling policy based on Amazon CloudWatch metrics such as CPU utilization. For a database, you can use Amazon RDS in a Multi-AZ environment, or Amazon DynamoDB, depending on your application and business requirements.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A .NET web application running in Azure App Service
+ Permission to use the Azure App Service Kudu console
**Product versions**
+ .NET Core (x64) 1.0.1, 2.0.0, or later, or .NET Framework 4.x, 3.5 (see [.NET on Windows Server platform history](https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platform-history-dotnet.html))
+ Internet Information Services (IIS) version 8.0 or later, running on Windows Server 2012 or later
+ .NET 2.0 or 4.0 Runtime.
## Architecture
**Source technology stack **
+ Application developed using .NET Framework 3.5 or later, or .NET Core 1.0.1, 2.0.0, or later, and hosted on Azure App Service (web app or API app)
**Target technology stack**
+ AWS Elastic Beanstalk running in an Amazon EC2 Auto Scaling group
**Migration architecture**
![\[Kudu accesses Azure App Service content, gets deployment package, uploads it to Elastic Beanstalk.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/df606a2d-b0a8-4035-b377-0a760e7300c9/images/dd15f97b-9cf2-4bcc-af45-44df1c4ca4a5.png)
**Deployment workflow**
![\[Deployment workflow to create app, publish it to launch environment, and then manage environment.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/df606a2d-b0a8-4035-b377-0a760e7300c9/images/accec77d-c753-4166-8f27-bd4932b3d884.png)
## Tools
**Tools**
+ .NET Core or .NET Framework
+ C\$1
+ IIS
+ Kudu console
**AWS services and features**
+ [AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html) – Elastic Beanstalk is an easy-to-use service for deploying and scaling .NET web applications. Elastic Beanstalk automatically manages capacity provisioning, load balancing, and auto scaling.
+ [Amazon EC2 Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) – Elastic Beanstalk includes an Auto Scaling group that manages the Amazon EC2 instances in the environment. In a single-instance environment, the Auto Scaling group ensures that there is always one instance running. In a load-balanced environment, you can configure the group with a range of instances to run, and Amazon EC2 Auto Scaling adds or removes instances as needed, based on load.
+ [Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) – When you enable load balancing in AWS Elastic Beanstalk, it creates a load balancer that distributes traffic among the EC2 instances in the environment.
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) – Elastic Beanstalk automatically uses Amazon CloudWatch to provide information about your application and environment resources. Amazon CloudWatch supports standard metrics, custom metrics, and alarms.
+ [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) – Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. You can use Route 53 alias records to map custom domain names to AWS Elastic Beanstalk environments.
## Epics
### Set up a VPC
| Task | Description | Skills required |
| --- | --- | --- |
| Set up a virtual private cloud (VPC). | In your AWS account, create a VPC with the required information. | System administrator |
| Create subnets. | Create two or more subnets in your VPC. | System administrator |
| Create a route table. | Create a route table, based on your requirements. | System administrator |
### Set up Elastic Beanstalk
| Task | Description | Skills required |
| --- | --- | --- |
| Access the Azure App Service Kudu console. | Access Kudu through the Azure portal by navigating to the App Service dashboard, and then choosing **Advanced Tools**, **Go**. Or, you can modify the Azure App Service URL as follows: `https://.scm.azurewebsites.net`. | App developer, System administrator |
| Download the deployment package from Kudu. | Navigate to Windows PowerShell by choosing the **DebugConsole** option. This will open the Kudo console. Go to the `wwwroot` folder and download it. This will download the Azure App Service deployment package as a zip file. For an example, see the attachment. | App developer, System administrator |
| Create a package for Elastic Beanstalk. | Unzip the deployment package that you downloaded from Azure App Service. Create a JSON file called `aws-windows-deployment-manifest.json` (this file is required only for .NET Core applications). Create a zip file that includes `aws-windows-deployment-manifest.json` and the Azure App Service deployment package file. For an example, see the attachment. | App developer, System administrator |
| Create a new Elastic Beanstalk application. | Open the Elastic Beanstalk console. Choose an existing application or create a new application. | App developer, System administrator |
| Create the environment. | In the Elastic Beanstalk console **Actions** menu, choose **Create environment**. Select the web server environment and .NET/IIS platform. For application code, choose **Upload**. Upload the zip file that you prepared for Elastic Beanstalk, and then choose **Create Environment**. | App developer, System administrator |
| Configure Amazon CloudWatch. | By default, basic CloudWatch monitoring is enabled. If you want to change the configuration, in the Elastic Beanstalk wizard, choose the published application, and then choose **Monitoring**. | System administrator |
| Verify that the deployment package is in Amazon S3. | When the application environment has been created, you can find the deployment package in the S3 bucket. | App developer, System administrator |
| Test the application. | When the environment has been created, use the URL provided in the Elastic Beanstalk console to test the application. | System administrator |
## Related resources
+ [AWS Elastic Beanstack concepts](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.html) (Elastic Beanstalk documentation)
+ [Getting Started with .NET on Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/dotnet-getstarted.html) (Elastic Beanstalk documentation)
+ [Kudu console](https://github.com/projectkudu/kudu/wiki/Kudu-console) (GitHub)
+ [Using "Kudu" to Manage Azure Web Apps](https://www.gslab.com/blogs/kudu-azure-web-app/) (GS Lab article)
+ [Custom ASP.NET Core Elastic Beanstalk Deployments](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/deployment-beanstalk-custom-netcore.html) (AWS Toolkit for Visual Studio user guide)
+ [Elastic Load Balancing documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html)
+ [AWS Elastic Beanstalk Supported Platforms](https://docs.amazonaws.cn/en_us/elasticbeanstalk/latest/platforms/platforms-supported.html) (Elastic Beanstalk documentation)
+ [Deploy a Web Application to AWS](https://www.c-sharpcorner.com/article/deploying-a-web-application-to-aws/) (C\$1 Corner article)
+ [Scaling the Size of Your Auto Scaling Group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/scaling_plan.html) (Amazon EC2 documentation)
+ [High Availability (Multi-AZ) for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) (Amazon RDS documentation)
## Additional information
**Notes**
+ If you are migrating an on-premises or Azure SQL Server database to Amazon RDS, you must update the database connection details as well.
+ For testing purposes, a sample demo application is attached.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/df606a2d-b0a8-4035-b377-0a760e7300c9/attachments/attachment.zip)
# Migrate from Oracle WebLogic to Apache Tomcat (TomEE) on Amazon ECS
*Anya Epishcheva and Harshad Gohil, Amazon Web Services*
## Summary
This pattern discusses the steps for migrating an on-premises Oracle Solaris SPARC system running Oracle WebLogic to a Docker container-based installation running [Apache TomEE](http://tomee.apache.org/) (Apache Tomcat with added container support) with Amazon Elastic Container Service (Amazon ECS).
For information about migrating databases that are associated with the applications you are migrating from Oracle WebLogic to Tomcat, see the database migration patterns in this catalog.
**Best practices**
Steps for migrating Java and Java Enterprise Edition (Java EE) web applications vary, depending on the number of container-specific resources used by the application. Spring-based applications are typically easier to migrate, because they have a small number of dependencies on the deployment container. In contrast, Java EE applications that use Enterprise JavaBeans (EJBs) and managed container resources such as thread pools, Java Authentication and Authorization Service (JAAS), and Container-Managed Persistence (CMP) require more effort.
Applications developed for Oracle Application Server frequently use the Oracle Identity Management suite. Customers migrating to open-source application servers frequently choose to re-implement identity and access management using SAML-based federation. Others use Oracle HTTP Server Webgate for cases when migrating from the Oracle Identity Management suite isn't an option.
Java and Java EE web applications are great candidates for deployment on AWS services that are Docker-based, such as AWS Fargate and Amazon ECS. Customers frequently choose a Docker image with the latest version of the target application server (such as TomEE) and the Java Development Kit (JDK) pre-installed. They install their applications on top of the base Docker image, publish it in their Amazon Elastic Container Registry (Amazon ECR) registry, and use it for scalable deployment of their applications on AWS Fargate or Amazon ECS.
Ideally, application deployment is elastic; that is, the number of application instances scales in or out, depending on traffic or workload. This means that application instances need to come online or be terminated to adjust capacity for demand.
When moving a Java application to AWS, consider making it stateless. This is a key architectural principle of the AWS Well-Architected Framework that will enable horizontal scaling using containerization. For example, most Java-based web applications store user-session information locally. To survive application instance termination due to automatic scaling in Amazon Elastic Compute Cloud (Amazon EC2) or for other reasons, user-session information should be stored globally so that web application users can continue to work seamlessly and transparently without reconnecting or relogging into a web application. There are several architectural options for this approach, including Amazon ElastiCache for Redis, or storing session state in a global database. Application servers such as TomEE have plugins, which enable session storage and management via Redis, databases, and other global data stores.
Use a common, centralized logging and debugging tool that is easily integrated with Amazon CloudWatch and AWS X-Ray. Migration provides an opportunity to improve application lifecycle capabilities. For example, you might want to automate the build process so that changes are easily made using a continuous integration and continuous delivery (CI/CD) pipeline. This may require changes to the application so that it can be deployed without any downtime.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Source Java code and JDK
+ Source application built with Oracle WebLogic
+ Defined solution for identity and access management (SAML or Oracle Webgate)
+ Defined solution for application session management (moving like-for-like or with Amazon ElastiCache, or making the application stateless if needed)
+ Understanding if the team needs to refactor J2EE-specific libraries for portability to Apache TomEE (see [Java EE 7 Implementation Status](http://tomee.apache.org/javaee7-status.html) on the Apache website)
+ Hardened TomEE image based on your security requirements
+ Container image with pre-installed target TomEE
+ Application remediation agreed and implemented if needed (for example, logging debug build, authentication)
**Product versions**
+ Oracle WebLogic OC4J, 9i, 10g
+ Tomcat 7 (with Java 1.6 or later)
## Architecture
**Source technology stack **
+ Web application built using Oracle WebLogic
+ Web application using Oracle Webgate or SAML authentication
+ Web applications connected to Oracle Database version 10g and later
**Target technology stack **
+ TomEE (Apache Tomcat with added container support) running on Amazon ECS (see also [Deploying Java Web Applications](https://aws.amazon.com/answers/web-applications/aws-web-app-deployment-java/) and [Java Microservices on Amazon ECS](https://aws.amazon.com/blogs/compute/deploying-java-microservices-on-amazon-ec2-container-service/))
+ Amazon Relational Database Service (Amazon RDS) for Oracle; for Oracle versions supported by Amazon RDS, see [Amazon RDS for Oracle](https://aws.amazon.com/rds/oracle/)** **
** Target architecture **
![\[AWS Cloud architecture diagram showing VPC, application subnets, and shared services account components.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/d5e7b3fa-062f-4559-af56-aa6058f96755/images/762193cf-aa68-4195-b3c7-6541caee61c9.png)
## Tools
To operate on TomEE, a Java application must be rebuilt into a .war file. In some cases, application changes may be required to operate the application on TomEE; you should check to make sure that the necessary configuration options and environment properties are defined correctly.
Also, Java Naming and Directory Interface (JNDI) lookups and JavaServer Pages (JSP) namespaces should be defined correctly. Consider checking file names used by the application to avoid naming collisions with built-in T libraries. For example, persistence.xml is a file name used by the Apache OpenJPA framework (which is bundled with OpenEJB in TomEE) for configuration purposes. The persistence.xml file in PUI contains Spring framework bean declarations.
TomEE version 7.0.3 and later (Tomcat 8.5.7 and later) returns an HTTP 400 response (bad request) for raw (unencoded) URLs with special characters. The server response appears as a blank page to the end-user. Earlier versions of TomEE and Tomcat allowed the use of certain unencoded special characters in URLs; however, it’s considered unsafe, as stated on the [CVE-2016-6816 website](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6816). To resolve the URL encoding issue, the URLs passed to the browser directly via JavaScript must be encoded with the **encodeURI()** method instead of being used as raw strings.
After you deploy the .war file in TomEE, monitor *start log* to *Linux cat* for any missing shared libraries and Oracle-specific extensions to add missing components from Tomcat libraries.
**General procedure**
+ Configure the application on TomEE.
+ Identify and reconfigure application server-specific configuration files and resources from source to target format.
+ Identify and reconfigure JNDI resources.
+ Adjust EJB namespace and lookups to the format required by the target application server (if applicable).
+ Reconfigure JAAS application container-specific security roles and principle mappings (if applicable).
+ Package the application and shared libraries into a .war file.
+ Deploy the .war file in TomEE by using the Docker container provided.
+ Monitor *start log* to identify any missing shared library and deployment descriptor extensions. If any are found, go back to the first task.
+ Test the installed application against the restored Amazon RDS database.
+ Launch the complete architecture with a load balancer and Amazon ECS cluster by following the instructions in [Deploy Docker Containers](https://aws.amazon.com/getting-started/tutorials/deploy-docker-containers/).
+ Update the URLs to point to the load balancer.
+ Update the configuration management database (CMDB).
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Perform application discovery (current state footprint and performance baseline). | | BA, Migration Lead |
| Validate source and target database versions and engines. | | DBA |
| Validate the source and target application design (identity and session management). | | DBA, Migration Engineer, App owner |
| Identify hardware and storage requirements for the target server instance. | | DBA, SysAdmin |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, SysAdmin |
| Identify network access security requirements for the source and target databases. | | DBA, SysAdmin |
| Identify application migration strategy and tooling. | | DBA, Migration Lead |
| Complete the migration design and migration guide for the application. | | Build Lead, Migration Lead |
| Complete the application migration runbook. | | Build Lead, Cutover Lead, Testing Lead, Migration Lead |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | | SysAdmin |
| Create security groups. | | SysAdmin |
| Configure and start the Amazon RDS DB instance. | | DBA, SysAdmin |
| Configure Amazon ECS deployment. | | SysAdmin |
| Package your application as a Docker image. | | SysAdmin |
| Push the image to the Amazon ECR registry (or skip this step and push it to the Amazon ECS cluster). | | SysAdmin |
| Configure the task definition for the application and Amazon ECS service options. | | SysAdmin |
| Configure your cluster, review security settings, and set AWS Identity and Access Management (IAM) roles. | | SysAdmin |
| Launch your setup and run tests according to your application migration runbook. | | SysAdmin |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Get your security assurance team's permission to move production data to AWS. | | DBA, Migration Engineer, App owner |
| Create and obtain access to endpoints to fetch database backup files. | | DBA |
| Use the native database engine or third-party tools to migrate database objects and data. | | DBA |
| Run necessary tests from the application migration runbook to confirm successful data migration. | | DBA, Migration Engineer, App owner |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Create a change request (CR) for migration. | | Cutover Lead |
| Obtain CR approval for migration. | | Cutover Lead |
| Follow the application migration strategy from the application migration runbook. | | DBA, Migration Engineer, App owner |
| Upgrade the application (if required). | | DBA, Migration Engineer, App owner |
| Complete functional, non-functional, data validation, SLA, and performance tests. | | Testing Lead, App Owner, App Users |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Obtain signoff from the application or business owner. | | Cutover Lead |
| Run a table topic exercise to walk through all the steps of the cutover runbook. | | DBA, Migration Engineer, App owner |
| Switch application clients to the new infrastructure. | | DBA, Migration Engineer, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down temporary AWS resources. | | DBA, Migration Engineer, SysAdmin |
| Review and validate the project documents. | | Migration Lead |
| Gather metrics around time to migrate, % of manual vs. tool, cost savings, etc. | | Migration Lead |
| Close out the project and provide feedback. | | Migration Lead, App Owner |
## Related resources
**References**
+ [Apache Tomcat 7.0 documentation](https://tomcat.apache.org/tomcat-7.0-doc/realm-howto.html)
+ [Apache Tomcat 7.0 installation guide](https://tomcat.apache.org/tomcat-7.0-doc/appdev/installation.html)
+ [Apache Tomcat JNDI documentation](https://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html)
+ [Apache TomEE documentation](http://tomee.apache.org/)
+ [Amazon RDS for Oracle](https://aws.amazon.com/rds/oracle/)
+ [Amazon RDS pricing](https://aws.amazon.com/rds/pricing/)
+ [Oracle and AWS](https://aws.amazon.com/oracle/)
+ [Oracle on Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html)
+ [Amazon RDS Multi-AZ deployments](https://aws.amazon.com/rds/details/multi-az/)
+ [Getting started with Amazon ECS](https://aws.amazon.com/ecs/getting-started/)
+ [Getting started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
**Tutorials and videos**
+ [Best Practices for Running Oracle Databases on Amazon RDS](https://www.youtube.com/watch?v=j2wqT0EPDbw) (re:Invent 2018 presentation)
# Migrate an Oracle database from Amazon EC2 to Amazon RDS for Oracle using AWS DMS
*Chethan Gangadharaiah and Brian motzer, Amazon Web Services*
## Summary
This pattern describes the steps for migrating an Oracle database on Amazon Elastic Compute Cloud (Amazon EC2) to Amazon Relational Database Service (Amazon RDS) for Oracle by using AWS Database Migration Service (AWS DMS). The pattern also uses Oracle SQL Developer or SQL \$1Plus to connect to your Oracle DB instance, and includes an AWS CloudFormation template that automates some of the tasks.
Migrating to Amazon RDS for Oracle enables you to focus on your business and applications while Amazon RDS takes care of database administration tasks such as provisioning databases, backup and recovery, security patches, version upgrades, and storage management.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An Amazon Machine Image (AMI) for Oracle Database on Amazon EC2
**Product versions**
+ AWS DMS supports Oracle versions 11g (version 11.2.0.3.v1 and later), 12c, and 18c for Amazon RDS instance databases for the Enterprise, Standard, Standard One, and Standard Two editions. For the latest information about supported versions, see [Using an Oracle Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html) in the AWS documentation. (The attached AWS CloudFormation templates use Oracle version 12c as the source database.)
+ Oracle SQL Developer 4.0.3
## Architecture
**Source architecture**
+ Oracle Database on Amazon EC2
**Target architecture**
+ Amazon RDS for Oracle
**Migration architecture**
![\[AWS Cloud architecture showing Oracle database migration from EC2 to RDS across availability zones.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4db0c731-0897-4eb8-a06f-b648c3d94b2c/images/636c2a69-5a78-482d-ae81-55e9ec975ead.png)
## Tools
+ [AWS DMS](https://docs.aws.amazon.com/dms/index.html) – AWS Database Migration Service (AWS DMS) helps you migrate databases to AWS quickly and securely. It supports both homogeneous and heterogeneous migrations. For information about the Oracle database versions and editions that are supported, see [Using an Oracle Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and [Using an Oracle Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html) in the AWS documentation.
+ **Oracle SQL Developer or SQL \$1Plus** – These tools enable you to connect to the Amazon RDS for Oracle DB instance.
## Epics
### Set up your target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon RDS for Oracle DB instance. | Sign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/. Create an Oracle DB instance by selecting the appropriate engine, template, database credentials setting, instance type, storage, Multi-AZ settings, virtual private cloud (VPC) and configuration, login credentials, and additional settings for the Oracle database. For instructions, view the links in the "Related resources" section. Or use the AWS CloudFormation template (Create\$1RDS.yaml) in the attachment to create the Amazon RDS for Oracle DB instance. | Developer |
| Connect to Amazon RDS and grant privileges to the Oracle user. | Modify the security group to open the appropriate ports to connect from the local machine and the AWS DMS replication instance. When you configure connectivity, make sure that the "Publicly accessible" option is selected so you can connect to the database from outside the VPC. Connect to Amazon RDS with Oracle SQL Developer or SQL \$1Plus by using the login credentials, create an AWS DMS user, and provide the required privileges to the AWS DMS user to modify the database. | Developer |
### Configure the security group of the source EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Check if the Oracle database is up and running. | Use Secure Shell (SSH) to connect to the EC2 instance, and try connecting to the Oracle database by using SQL \$1Plus. | Developer |
| Modify the security group. | Modify the security group of the EC2 instance to open appropriate ports, so you can connect from your local machine and the AWS DMS replication instance. | Developer |
### Set up AWS DMS
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS DMS replication instance. | In AWS DMS, create a replication instance in the same VPC as your Amazon RDS for Oracle DB instance. Specify the name and description for the replication instance, choose the instance class and replication engine version (use the default), choose the VPC in which you created the Amazon RDS DB instance, set Multi-AZ settings if required, allocate storage, specify the Availability Zone, and configure additional settings. Alternatively, you can use the AWS CloudFormation template (DMS.yaml) in the attachment to implement this step. | DBA |
| Connect to the source and target database endpoints. | Create the source and target database endpoints by specifying the endpoint identifier, engine, server, port, login credentials, and extra connection attributes. For the source server, use the public DNS of the EC2 instance that's hosting the Oracle database. For the target server, use the endpoint of Amazon RDS for Oracle. Run a test to verify that the source and target connections are working. Alternatively, you can use the AWS CloudFormation template (DMS.yaml) in the attachment to implement this step. | DBA |
| Create an AWS DMS task. | Create an AWS DMS task to migrate data from the source endpoint to the target endpoint, to set up replication between the source and destination endpoint, or both. When creating the AWS DMS task, specify the replication instance, source endpoint, target endpoint, migration type (data only, replication only, or both), table mapping, and filter. Run the AWS DMS task, monitor the task, check the table statistics, and check logs in Amazon CloudWatch. Alternatively, you can use the AWS CloudFormation template (DMS.yaml) in the attachment to implement this step. | DBA |
## Related resources
+ [Creating an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateOracleInstance.html)
+ [Connecting to a DB Instance Running the Oracle Database Engine](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToOracleInstance.html)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [AWS DMS Step-by-Step Walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html)
+ [Migrating Oracle Databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/4db0c731-0897-4eb8-a06f-b648c3d94b2c/attachments/attachment.zip)
# Migrate an on-premises Oracle database to Amazon OpenSearch Service using Logstash
*Aditya Goteti, Amazon Web Services*
## Summary
This pattern describes how to move data from an on-premises Oracle database to Amazon OpenSearch Service using Logstash. It includes architectural considerations and some required skill sets and recommendations. The data can be from a single table or from multiple tables in which a full-text search will need to be performed.
OpenSearch Service can be configured within a virtual private cloud (VPC), or it can be placed publicly with IP-based restrictions. This pattern describes a scenario where OpenSearch Service is configured within a VPC. Logstash is used to collect the data from the Oracle database, parse it to JSON format, and then feed the data into OpenSearch Service.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Java 8 (required by Logstash 6.4.3)
+ Connectivity between the on-premises database servers and Amazon Elastic Compute Cloud (Amazon EC2) instances in a VPC, established using AWS Virtual Private Network (AWS VPN)
+ A query to retrieve the required data to be pushed to OpenSearch Service from the database
+ Oracle Java Database Connectivity (JDBC) drivers
**Limitations**
+ Logstash cannot identify records that are hard-deleted from the database
**Product versions**
+ Oracle Database 12c
+ OpenSearch Service 6.3
+ Logstash 6.4.3
## Architecture
**Source technology stack**
+ On-premises Oracle database
+ On-premises AWS VPN
**Target technology stack**
+ VPC
+ EC2 instance
+ OpenSearch Service
+ Logstash
+ NAT Gateway (for operating system updates on EC2 instances and to install Java 8, Logstash, and plugins)
**Data migration architecture**
![\[How to move data from an on-premises Oracle database to Amazon OpenSearch Service using Logstash.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/09f6d2de-de2f-4ed6-af93-34b71b75a263/images/df6a61fb-09fb-49d4-a7e8-b04e88c003df.png)
## Tools
+ Logstash 6.4.3
+ JDBC input plugin ([download and more information](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html))
+ Logstash output plugin ([logstash-output-amazon\$1es](https://github.com/awslabs/logstash-output-amazon_es))
+ Oracle JDBC drivers
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Identify the size of the source data. | The size of the source data is one of the parameters that you use to determine the number of shards to be configured in an index. | DBA, Database developer |
| Analyze the data types of each column and corresponding data. | OpenSearch Service dynamically maps the data type when a previously unseen field is found in the document. If there are any specific data types or formats (for example, date fields) that need to be explicitly declared, identify the fields and define the mapping for those fields during index creation. | App owner, Developer, Database developer |
| Determine if there are any columns with primary or unique keys. | To avoid duplication of records in Amazon OpenSearch Service during updates or inserts, you need to configure the `document_id` setting in the output section of the `amazon_es` plugin (for example, `document_id => "%{customer_id}"` where `customer_id` is a primary key). | App owner, Developer |
| Analyze the number and frequency of new records added; check how frequently the records are deleted. | This task is required to understand the growth rate of source data. If data is intensively read and insertions are rare, you can have a single index. If new records are inserted frequently and there are no deletions, the shard size can easily exceed the recommended maximum size of 50 GB. In this case, you can dynamically create an index by configuring index patterns in Logstash and in the code where you can access it by using an alias. | App owner, Developer |
| Determine how many replicas are required. | | App owner, Developer |
| Determine the number of shards to be configured on the index. | | App owner, Developer |
| Identify the instance types for dedicated master nodes, data nodes, and the EC2 instance. | For more information, see the [Related resources](#migrate-an-on-premises-oracle-database-to-amazon-opensearch-service-using-logstash-resources) section. | App owner, Developer |
| Determine the number of dedicated master nodes and data nodes required. | For more information, see the [Related resources](#migrate-an-on-premises-oracle-database-to-amazon-opensearch-service-using-logstash-resources) section. | |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Launch an EC2 instance. | Launch an EC2 instance within the VPC to which AWS VPN is connected. | Amazon VPC constructs, AWS VPN |
| Install Logstash on the EC2 instance. | | Developer |
| Install the Logstash plugins. | Install the required Logstash plugins `jdbc-input` and` logstash-output-amazon_es`. | Developer |
| Configure Logstash. | Create the Logstash keystore to store sensitive information such as AWS Secrets Manager keys and database credentials, and then place the references in a Logstash configuration file. | Developer |
| Configure the dead letter queue and persistent queue. | By default, when Logstash encounters an event that it cannot process because the data contains a mapping error or some other issue, the Logstash pipeline either hangs or drops the unsuccessful event. To protect against data loss in this situation, you can configure Logstash to write unsuccessful events to a dead letter queue instead of dropping them. To protect against data loss during abnormal termination, Logstash has a persistent queue feature that will store the message queue on disk. Persistent queues provide the data durability in Logstash. | Developer |
| Create the Amazon OpenSearch Service domain. | Create the Amazon OpenSearch Service domain with an access policy that doesn't require signing requests with AWS Identity and Access Management (IAM) credentials. The Amazon OpenSearch Service domain must be created within the same VPC. You should also select the instance types and set the number of dedicated and master nodes based on your analysis. | Developer |
| Configure the required Amazon OpenSearch Service logs. | For more information, see the [OpenSearch Service documentation](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/cloudwatch-alarms.html). | |
| Create the index. | | Developer |
| Start Logstash. | Run Logstash as a background service. Logstash runs the configured SQL query, pulls the data, converts it to JSON format, and feeds it to OpenSearch Service. For the initial load, do not configure the scheduler in the Logstash configuration file. | Developer |
| Check documents. | Check the number of documents on the index and whether all documents are present in the source database. During initial load, they are added to the index and used to stop Logstash. Change the Logstash configuration to add a scheduler that runs at a fixed interval depending on client requirements, and restart Logstash. Logstash will pick only the records that were updated or added after the last run, and the last run timestamp is stored in the file that is configured with the property `last_run_metadata_path => "/usr/share/logstash/.logstash_jdbc_last_run"` in the Logstash configuration file. | Developer |
## Related resources
+ [Recommended CloudWatch Alarms](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/cloudwatch-alarms.html)
+ [Dedicated Amazon OpenSearch Service Master Nodes](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-dedicatedmasternodes.html)
+ [Sizing Amazon OpenSearch Service Domains](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sizing-domains.html)
+ [Logstash documentation](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html)
+ [JDBC input plugin](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html)
+ [Logstash output plugin](https://github.com/awslabs/logstash-output-amazon_es)
+ [Amazon OpenSearch Service website](https://aws.amazon.com/elasticsearch-service/)
# Migrate an on-premises Oracle database to Amazon RDS for Oracle
*Baji Shaik and Pavan Pusuluri, Amazon Web Services*
## Summary
This pattern describes the steps for migrating on-premises Oracle databases to Amazon Relational Database Service (Amazon RDS) for Oracle. As part of the migration process, you create a migration plan and consider important factors about your target database infrastructure based on your source database. You can choose one of two migration options based on your business requirements and use case:
+ AWS Database Migration Service (AWS DMS) – You can use AWS DMS to migrate databases to the AWS Cloud quickly and securely. Your source database remains fully operational during the migration, which minimizes downtime to applications that rely on the database. You can reduce migration time by using AWS DMS to create a task that captures ongoing changes after you complete an initial full-load migration through a process called [change data capture (CDC)](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html).
+ Native Oracle tools – You can migrate databases by using native Oracle tools, such as Oracle and [Data Pump Export](https://docs.oracle.com/cd/E11882_01/server.112/e22490/dp_export.htm#SUTIL200) and [Data Pump Import](https://docs.oracle.com/cd/E11882_01/server.112/e22490/dp_import.htm#SUTIL300) with [Oracle GoldenGate](https://docs.oracle.com/goldengate/c1230/gg-winux/GGCON/introduction-oracle-goldengate.htm#GGCON-GUID-EF513E68-4237-4CB3-98B3-2E203A68CBD4) for CDC. You can also use native Oracle tools such as the original [Export utility](https://docs.oracle.com/cd/E11882_01/server.112/e22490/original_export.htm#SUTIL3634) and original [Import utility](https://docs.oracle.com/cd/E11882_01/server.112/e22490/original_import.htm#SUTIL001) to reduce the full-load time.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An on-premises Oracle database
+ An Amazon RDS Oracle database (DB) instance
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ Oracle versions 11g (versions 11.2.0.3.v1 and later) and up to 12.2 and 18c. For the latest list of supported versions and editions, see [Amazon RDS for Oracle ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html)in the AWS documentation. For Oracle versions supported by AWS DMS, see [Using an Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) in the AWS DMS documentation.
## Architecture
**Source technology stack**
+ On-premises Oracle databases
**Target technology stack**
+ Amazon RDS for Oracle
**Source and target architecture**
The following diagram shows how to migrate an on-premises Oracle database to Amazon RDS for Oracle by using AWS DMS.
![\[Workflow for migrating Oracle databases to Amazon RDS for Oracle by using AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/25912997-0ac0-4303-9ce5-0621a7e12406/images/20f94a5c-1095-4182-b964-c379414c9a36.png)
The diagram shows the following workflow:
1. Create or use an existing database user, grant the required [AWS DMS permissions](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Self-Managed) to that user, turn on [ARCHIVELOG mode](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Self-Managed.Configuration.ArchiveLogMode), and then set up [supplemental logging](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Self-Managed.Configuration.SupplementalLogging).
1. Configure the internet gateway between the on-premises and AWS network.
1. Configure [source and target endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html) for AWS DMS.
1. Configure [AWS DMS replication tasks](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.html) to migrate the data from the source database to the target database.
1. Complete the post-migration activities on the target database.
The following diagram shows how to migrate an on-premises Oracle database to Amazon RDS for Oracle by using native Oracle tools.
![\[Workflow for migrating Oracle databases to Amazon RDS for Oracle by using Oracle tools.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/25912997-0ac0-4303-9ce5-0621a7e12406/images/af8e0e1a-d4c8-4d99-9780-3e093ad9a257.png)
The diagram shows the following workflow:
1. Create or use an existing database user and grant the required permissions to back up the Oracle database by using Oracle Export (`exp`) and Import (`imp`) utilities.
1. Configure the internet gateway between the on-premises and AWS network.
1. Configure the Oracle client on the [Bastion](https://www.oracle.com/security/cloud-security/bastion/) host to take the backup database.
1. Upload the backup database to an Amazon Simple Storage Service (Amazon S3) bucket.
1. Restore the database backup from Amazon S3 to an Amazon RDS for Oracle database.
1. Configure Oracle GoldenGate for CDC.
1. Complete the post-migration activities on the target database.
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups.
+ Native Oracle tools help you perform a homogeneous migration. You can use [Oracle Data Pump](https://docs.oracle.com/cd/B19306_01/server.102/b14215/dp_overview.htm) to migrate data between your source and target databases. This pattern uses Oracle Data Pump to perform the full load from the source database to the target database.
+ [Oracle GoldenGate](https://docs.oracle.com/goldengate/c1230/gg-winux/GGCON/introduction-oracle-goldengate.htm#GGCON-GUID-EF513E68-4237-4CB3-98B3-2E203A68CBD4) helps you perform logical replication between two or more databases. This pattern uses GoldenGate to replicate the delta changes after the initial load by using Oracle Data Pump.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Create project documents and record database details. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA |
| Identify storage requirements. | Identify and document your storage requirements, including the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html)For [General Purpose (gp2) SSD volumes](https://aws.amazon.com/ebs/volume-types/), you get three IOPS per 1 GB of storage. Allocate storage by calculating the total number of read and write IOPS on the source database. | DBA, SysAdmin |
| Choose the proper instance type based on compute requirements. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | SysAdmin |
| Identify network access security requirements. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA, SysAdmin |
| Identify the application migration strategy. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA, SysAdmin, App owner |
| Identify migration risks. | Assess the database and document migration specific risks and mitigations. For example:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a VPC. | [Create a new Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html) for the target DB instance. | SysAdmin |
| Create security groups. | [Create a security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#creating-security-group) in your new VPC to allow inbound connections to the DB instance. | SysAdmin |
| Create an Amazon RDS for Oracle DB instance. | [Create the target DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html) with the new VPC and security group, and then start the instance. | SysAdmin |
### Option 1 - Use native Oracle or third-party tools to migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the source database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA, SysAdmin |
| Prepare the target database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA, SysAdmin |
### Option 2 - Use AWS DMS to migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA |
| Migrate the data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA |
### Cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients to the new infrastructure. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA, SysAdmin, App owner |
| Implement your rollback plan. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html) | DBA, App owner |
### Close out the migration project
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up resources. | Shut down or remove the temporary AWS resources, such as the AWS DMS replication instance and S3 bucket. | DBA, SysAdmin |
| Review project documents. | Review your migration planning documents and goals, and then confirm that you completed all required migration steps. | DBA, SysAdmin, App owner |
| Gather metrics. | Record key migration metrics, including how long it took to complete the migration, the percentage of manual vs. tool-based tasks, cost savings, and other relevant metrics. | DBA, SysAdmin, App owner |
| Close out the project. | Close out the migration project and capture feedback about the effort. | DBA, SysAdmin, App owner |
## Related resources
**References**
+ [Migrating Oracle databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html) (AWS Prescriptive Guidance)
+ [AWS Database Migration Service](https://aws.amazon.com/dms/) (AWS DMS documentation)
+ [Amazon RDS Pricing](https://aws.amazon.com/rds/pricing/) (Amazon RDS documentation)
**Tutorials and videos**
+ [Getting Started with AWS Database Migration Service](https://aws.amazon.com/dms/getting-started/) (AWS DMS documentation)
+ [Amazon RDS resources](https://aws.amazon.com/rds/getting-started/) (Amazon RDS documentation)
+ [AWS Database Migration Service (DMS)](https://www.youtube.com/watch?v=zb4GcjEdl8U) (YouTube)
# Migrate an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump
*Mohan Annam and Brian motzer, Amazon Web Services*
## Summary
This pattern describes how to migrate an Oracle database from an on-premises data center to an Amazon Relational Database Service (Amazon RDS) for Oracle DB instance by using Oracle Data Pump.
The pattern involves creating a data dump file from the source database, storing the file in an Amazon Simple Storage Service (Amazon S3) bucket, and then restoring the data to an Amazon RDS for Oracle DB instance. This pattern is useful when you encounter limitations using AWS Database Migration Service (AWS DMS) for the migration.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ The required permissions to create roles in AWS Identity and Access Management (IAM) and for an Amazon S3 multipart upload
+ The required permissions to export data from the source database
+ AWS Command Line Interface (AWS CLI) [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
**Product versions**
+ Oracle Data Pump is available only for Oracle Database 10g Release 1 (10.1) and later versions.
## Architecture
**Source technology stack**
+ On-premises Oracle databases
**Target technology stack**
+ Amazon RDS for Oracle
+ SQL client (Oracle SQL Developer)
+ An S3 bucket
**Source and target architecture**
![\[Amazon S3 multipart upload from an on-premises Oracle DB to Amazon RDS by using Oracle Data Pump.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/d8d6e00f-753e-4ecc-80e5-e60e279a699b/images/1bb6095a-0a95-4469-be0e-7b7bd59b35ae.png)
## Tools
**AWS services**
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. In this pattern, IAM is used to create the roles and policies necessary for migrating data from Amazon S3 to Amazon RDS for Oracle.
+ [Amazon Relational Database Service (Amazon RDS) for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) helps you set up, operate, and scale an Oracle relational database in the AWS Cloud.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Other tools**
+ [Oracle Data Pump](https://docs.oracle.com/cd/B19306_01/server.102/b14215/dp_overview.htm) helps you move data and metadata from one database to another at high speeds. In this pattern, Oracle Data Pump is used to export the data dump (.dmp) file to the Oracle server, and to import it into Amazon RDS for Oracle. For more information, see [Importing data into Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html#Oracle.Procedural.Importing.DataPump.S3) in the Amazon RDS documentation.
+ [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) is an integrated development environment that simplifies the development and management of Oracle databases in both traditional and cloud-based deployments. It interacts with both the on-premises Oracle database and Amazon RDS for Oracle to run the SQL commands required for exporting and importing data.
## Epics
### Create an S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create the bucket. | To create the S3 bucket, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). | AWS systems administrator |
### Create the IAM role and assign policies
| Task | Description | Skills required |
| --- | --- | --- |
| Configure IAM permissions. | To configure permissions, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-s3-integration.html#oracle-s3-integration.preparing). | AWS systems administrator |
### Create the target Amazon RDS for Oracle DB instance and associate the Amazon S3 integration role
| Task | Description | Skills required |
| --- | --- | --- |
| Create the target Amazon RDS for Oracle DB instance. | To create the Amazon RDS for Oracle instance, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.Oracle.html). | AWS systems administrator |
| Associate the role with the DB instance. | To associate the role with the instance, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-s3-integration.html#oracle-s3-integration.preparing.instance). | DBA |
### Create the database user on the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the user. | Connect to the target Amazon RDS for Oracle database from Oracle SQL Developer or SQL\$1Plus, and run the following SQL command to create the user to import the schema into.
create user SAMPLE_SCHEMA identified by ; grant create session, resource to ; alter user quota 100M on users;
| DBA |
### Create the export file from the source Oracle database
| Task | Description | Skills required |
| --- | --- | --- |
| Create a data dump file. | To create a dump file named `sample.dmp` in the `DATA_PUMP_DIR` directory for exporting the `SAMPLE_SCHEMA` user, use the following script.
Review the export details by reviewing the `export.log` file in your local `DATA_PUMP_DIR` directory. | DBA |
### Upload the dump file to the S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Upload the data dump file from the source to the S3 bucket. | Using AWS CLI, run the following command.
aws s3 cp sample.dmp s3:///
| DBA |
### Download the export file from the S3 bucket to the RDS instance
| Task | Description | Skills required |
| --- | --- | --- |
| Download the data dump file to Amazon RDS | To copy the dump file `sample.dmp` from the S3 bucket to the Amazon RDS for Oracle database, run the following SQL command. In this example, the `sample.dmp` file is downloaded from the S3 bucket `my-s3-integration1` to the Oracle directory `DATA_PUMP_DIR`. Make sure that you have sufficient disk space allocated to your RDS instance to accommodate both the database and the export file.
-- If you want to download all the files in the S3 bucket remove the p_s3_prefix line.
SELECT rdsadmin.rdsadmin_s3_tasks.download_from_s3( p_bucket_name => 'my-s3-integration', p_s3_prefix => 'sample.dmp', p_directory_name => 'DATA_PUMP_DIR') AS TASK_ID FROM DUAL;
The previous command outputs a task ID. To review the status of the download by reviewing the data in the task ID, run the following command.
SELECT text FROM table(rdsadmin.rds_file_util.read_text_file('BDUMP','dbtask-.log'));
To see the files in the `DATA_PUMP_DIR` directory, run the following command.
SELECT filename,type,filesize/1024/1024 size_megs,to_char(mtime,'DD-MON-YY HH24:MI:SS') timestamp FROM TABLE(rdsadmin.rds_file_util.listdir(p_directory => upper('DATA_PUMP_DIR'))) order by 4;
| AWS systems administrator |
### Import the dump file into the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Restore the schema and data to Amazon RDS. | To import the dump file into the `sample_schema` database schema, run the following SQL command from SQL Developer or SQL\$1Plus.
To see the log file from the import, run the following command.
SELECT text FROM table(rdsadmin.rds_file_util.read_text_file('DATA_PUMP_DIR','import.log'));
| DBA |
### Remove the dump file from the DATA\$1PUMP\$1DIR directory
| Task | Description | Skills required |
| --- | --- | --- |
| List and clean up the export files. | List and remove the export files in the `DATA_PUMP_DIR` directory, run the following commands.
-- List the files SELECT filename,type,filesize/1024/1024 size_megs,to_char(mtime,'DD-MON-YY HH24:MI:SS') timestamp FROM TABLE(rdsadmin.rds_file_util.listdir(p_directory => upper('DATA_PUMP_DIR'))) order by 4;
-- Remove the files EXEC UTL_FILE.FREMOVE('DATA_PUMP_DIR','sample.dmp'); EXEC UTL_FILE.FREMOVE('DATA_PUMP_DIR','import.log');
| AWS systems administrator |
## Related resources
+ [Amazon S3 integration](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-s3-integration.html#oracle-s3-integration.preparing)
+ [Create a DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Tutorials.WebServerDB.CreateDBInstance.html)
+ [Importing data into Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html#Oracle.Procedural.Importing.DataPump.S3)
+ [Amazon S3 documentation](https://docs.aws.amazon.com/s3/index.html)
+ [IAM documentation](https://docs.aws.amazon.com/iam/index.html)
+ [Amazon RDS documentation](https://docs.aws.amazon.com/rds/index.html)
+ [Oracle Data Pump documentation](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-data-pump-overview.html)
+ [Oracle SQL Developer](https://www.oracle.com/database/sqldeveloper/)
# Migrate from PostgreSQL on Amazon EC2 to Amazon RDS for PostgreSQL using pglogical
*Rajesh Madiwale, Amazon Web Services*
## Summary
This pattern outlines steps for migrating a PostgreSQL database (version 9.5 and later) from Amazon Elastic Compute Cloud (Amazon EC2) to Amazon Relational Database Service (Amazon RDS) for PostgreSQL by using the PostgreSQL **pglogical **extension. Amazon RDS now supports the pglogical extension for PostgreSQL version 10.
## Prerequisites and limitations
**Prerequisites **
+ Choose the right type of Amazon RDS instance. For more information, see [Amazon RDS Instance Types](https://aws.amazon.com/rds/instance-types/).
+ Make sure that the source and target versions of PostgreSQL are the same.
+ Install and integrate the [**pglogical** extension with PostgreSQL](https://github.com/2ndQuadrant/pglogical) on Amazon EC2.
**Product versions**
+ PostgreSQL version 10 and later on Amazon RDS, with the features supported on Amazon RDS (see [PostgreSQL on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts) in the AWS documentation). This pattern was tested by migrating PostgreSQL 9.5 to PostgreSQL version 10 on Amazon RDS, but it also applies to later versions of PostgreSQL on Amazon RDS.
## Architecture
**Data migration architecture**
![\[Data migration architecture for PostgreSQL on Amazon RDS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/29af3931-48de-499f-9c4b-e10a98e4bba5/images/5f5b906f-dc1a-49a5-ae3f-3e10ae854784.png)
## Tools
+ [https://github.com/2ndQuadrant/pglogical](https://github.com/2ndQuadrant/pglogical) extension
+ PostgreSQL native utilities: [https://www.postgresql.org/docs/9.5/app-pgdump.html](https://www.postgresql.org/docs/9.5/app-pgdump.html) and [https://www.postgresql.org/docs/9.6/app-pgrestore.html](https://www.postgresql.org/docs/9.6/app-pgrestore.html)
## Epics
### Migrate data by using the pglogical extension
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon RDS PostgreSQL DB instance. | Set up a PostgreSQL DB instance in Amazon RDS. For instructions, see the [Amazon RDS for PostgreSQL documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html). | DBA |
| Obtain a schema dump from the source PostgreSQL database and restore it into the target PostgreSQL database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-postgresql-on-amazon-ec2-to-amazon-rds-for-postgresql-using-pglogical.html) | DBA |
| Turn on logical decoding. | In the Amazon RDS DB parameter group, set the `rds.logical_replication` static parameter to 1. For instructions, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.LogicalDecoding). | DBA |
| Create the pglogical extension on the source and target databases. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-postgresql-on-amazon-ec2-to-amazon-rds-for-postgresql-using-pglogical.html) | DBA |
| Create a publisher on the source PostgreSQL database. | To create a publisher, run:
| DBA |
| Create a replication set, add tables and sequences. | To create a replication set on the source PostgreSQL database, and to add tables and sequences to the replication set, run:
| DBA |
### Validate your data
| Task | Description | Skills required |
| --- | --- | --- |
| Check source and target databases. | Check the source and target databases to confirm that data is being replicated successfully. You can perform basic validation by using `select count(1)` from the source and target tables. | DBA |
## Related resources
+ [Amazon RDS](https://aws.amazon.com/rds/)
+ [Logical replication for PostgreSQL on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication) (Amazon RDS documentation)
+ [pglogical](https://github.com/2ndQuadrant/pglogical) (GitHub repository)
+ [Limitations of pglogical](https://github.com/2ndQuadrant/pglogical#limitations-and-restrictions) (GitHub repository README file)
+ [Migrating PostgreSQL from on-premises or Amazon EC2 to Amazon RDS using logical replication](https://aws.amazon.com/blogs/database/migrating-postgresql-from-on-premises-or-amazon-ec2-to-amazon-rds-using-logical-replication/) (AWS Database blog)
# Migrate an on-premises PostgreSQL database to Aurora PostgreSQL
*Baji Shaik and Jitender Kumar, Amazon Web Services*
## Summary
Amazon Aurora PostgreSQL-Compatible Edition combines the performance and availability of high-end commercial databases with the simplicity and cost-effectiveness of open-source databases. Aurora provides these benefits by scaling storage across three Availability Zones in the same AWS Region, and supports up to 15 read replica instances for scaling out read workloads and providing high availability within a single Region. By using an Aurora global database, you can replicate PostgreSQL databases in up to five Regions for remote read access and disaster recovery in the event of a Region failure. This pattern describes the steps for migrating an on-premises PostgreSQL source database to an Aurora PostgreSQL-Compatible database. The pattern includes two migration options: using AWS Data Migration Service (AWS DMS) or using native PostgreSQL tools (such as [pg\$1dump](https://www.postgresql.org/docs/current/app-pgdump.html), [pg\$1restore](https://www.postgresql.org/docs/current/app-pgrestore.html), and [psql](https://www.postgresql.org/docs/current/app-psql.html)) or third-party tools.
The steps described in this pattern also apply to target PostgreSQL databases on Amazon Relational Database Service (Amazon RDS) and Amazon Elastic Compute Cloud (Amazon EC2) instances.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A PostgreSQL source database in an on-premises data center
+ [An Aurora PostgreSQL-Compatible DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or an [Amazon RDS for PostgreSQL DB instance](https://aws.amazon.com/getting-started/hands-on/create-connect-postgresql-db/)
**Limitations**
+ Database size limits are 64 TB for Amazon RDS for PostgreSQL and 128 TB for Aurora PostgreSQL-Compatible.
+ If you’re using the AWS DMS migration option, review [AWS DMS limitations on using a PostgreSQL database as a source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.Limitations).
**Product versions**
+ For PostgreSQL major and minor version support in Amazon RDS, see [Amazon RDS for PostgreSQL updates](https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-versions.html) in the Amazon RDS documentation.
+ For PostgreSQL support in Aurora, see [Amazon Aurora PostgreSQL updates](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQLReleaseNotes/AuroraPostgreSQL.Updates.html) in the Aurora documentation.
+ If you’re using the AWS DMS migration option, see[ supported PostgreSQL versions](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html) in the AWS DMS documentation.
## Architecture
**Source technology stack**
+ On-premises PostgreSQL database
**Target technology stack**
+ Aurora PostgreSQL-Compatible DB instance
**Source architecture**
![\[Source architecture for on-premises PostgreSQL database\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/82114165-8102-44a2-8b12-485ac9eb8989/images/a8621ad3-781b-45a9-86a8-d0b0ec5c79ea.png)
**Target architecture**
![\[Target architecture for PostgreSQL database on Amazon Aurora\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/82114165-8102-44a2-8b12-485ac9eb8989/images/fc2ec0cb-7b9b-4cc0-b70c-40e47c2f4c45.png)
**Data migration architecture**
*Using AWS DMS*
![\[Migrating an on-premises PostgreSQL database to Aurora by using AWS DMS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/82114165-8102-44a2-8b12-485ac9eb8989/images/5336adb4-e9eb-47d0-a5b5-d149261b1638.png)
*Using native PostgreSQL tools*
![\[Migrating an on-premises PostgreSQL database to Aurora by using pg_dump and pg_restore\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/82114165-8102-44a2-8b12-485ac9eb8989/images/3c6fb533-45ff-443e-bfb1-97e60cbdd583.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises configurations. This service supports different sources and target databases. For information about how to validate the PostgreSQL source and target database versions and editions supported for use with AWS DMS, see [Using a PostgreSQL database as an AWS DMS source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support.
+ Native PostgreSQL tools include [pg\$1dump](https://www.postgresql.org/docs/current/app-pgdump.html), [pg\$1restore](https://www.postgresql.org/docs/current/app-pgrestore.html), and [psql](https://www.postgresql.org/docs/current/app-psql.html).
## Epics
### Analyze the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database versions. | If you are using AWS DMS, make sure that you’re using a [supported version of PostgreSQL](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html). | DBA |
| Identify the storage type and capacity requirements. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA, Systems administrator |
| Choose the proper instance type, capacity, storage features, and network features. | Determine the compute requirements of the target database instance. Review known performance issues that might need additional attention. Consider the following factors to determine the appropriate instance type:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html)For more information, see [Aurora DB instance classes](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.DBInstanceClass.html) in the Aurora documentation. | DBA, Systems administrator |
| Identify the network access security requirements for the source and target databases. | Determine the appropriate security groups that would enable the application to talk to the database. | DBA, Systems administrator |
| Identify the application migration strategy. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA, App owner, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a VPC. | Create a new virtual private cloud (VPC) for the target database instance. | Systems administrator |
| Create security groups. | Create a security group within the VPC (as determined in the previous epic) to allow inbound connections to the database instance. | Systems administrator |
| Configure and start the Aurora DB cluster. | Create the target database instance with the new VPC and security group and start the instance. | Systems administrator |
### Migrate data ‒ option 1 (using AWS DMS)
| Task | Description | Skills required |
| --- | --- | --- |
| Complete pre-migration steps. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA |
| Complete migration steps. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA |
| Validate data. | To ensure that your data was migrated accurately from the source to the target, follow the [data validation steps](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html) in the AWS DMS documentation. | DBA |
### Migrate data ‒ option 2 (using pg\$1dump and pg\$1restore)
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the source database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html)For more information, see the [pg\$1dump](https://www.postgresql.org/docs/current/app-pgdump.html) documentation and the [walkthrough](https://docs.aws.amazon.com/dms/latest/sbs/chap-manageddatabases.postgresql-rds-postgresql-full-load-pd_dump.html) in the AWS DMS documentation. | DBA |
| Prepare the target database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html)For more information, see the [pg\$1restore](https://www.postgresql.org/docs/current/app-pgrestore.html) documentation and the [walkthrough](https://docs.aws.amazon.com/dms/latest/sbs/chap-manageddatabases.postgresql-rds-postgresql-full-load-pd_dump.html) in the AWS DMS documentation. | DBA |
| Validate data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | Implement the application migration strategy that you created in the first epic. | DBA, App owner, Systems administrator |
### Cut over to the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA, App owner, Systems administrator |
| If you need to roll back the migration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-postgresql-database-to-aurora-postgresql.html) | DBA, App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down resources. | Shut down the temporary AWS resources. | DBA, Systems administrator |
| Validate documents. | Review and validate the project documents. | DBA, App owner, Systems administrator |
| Gather metrics. | Gather metrics around time to migrate, percent of manual versus tool cost savings, and so on. | DBA, App owner, Systems administrator |
| Close the project. | Close the project and provide any feedback. | DBA, App owner, Systems administrator |
## Related resources
**References**
+ [AWS Data Migration Service](https://aws.amazon.com/dms/)
+ [VPCs and Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_VPC.html)
+ [Amazon Aurora pricing](https://aws.amazon.com/rds/aurora/pricing/)
+ [Using a PostgreSQL database as an AWS DMS source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html)
+ [How to create an AWS DMS replication instance](https://aws.amazon.com/premiumsupport/knowledge-center/create-aws-dms-replication-instance/)
+ [How to create source and target endpoints using AWS DMS](https://aws.amazon.com/premiumsupport/knowledge-center/create-source-target-endpoints-aws-dms/)
**Additional resources**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Data migration step-by-step walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html)
+ [Amazon Aurora resources](https://aws.amazon.com/rds/aurora/getting-started/)
# Migrate an on-premises Microsoft SQL Server database to Microsoft SQL Server on Amazon EC2 running Linux
*Tirumala Dasari, Amazon Web Services*
## Summary
This pattern describes how to migrate from an on-premises Microsoft SQL Server database running on Microsoft Windows, to Microsoft SQL Server on an Amazon Elastic Compute Cloud (Amazon EC2) Linux instance by using backup and restore utilities.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Amazon EC2 Linux AMI (Amazon Machine Image) with Microsoft SQL Server
+ AWS Direct Connect between on-premises Windows and Microsoft SQL Server on the Linux EC2 instance
## Architecture
**Source technology stack**
+ On-premises Microsoft SQL Server database
**Target technology stack**
+ Linux EC2 instance with a Microsoft SQL Server database
**Database migration architecture**
![\[Architecture diagram to migrate an on-premises SQL Server database to a Linux EC2 instance.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f331ad15-2d41-4087-a6d1-60e3443e2acf/images/f50a779a-ce5d-44b1-8d37-dedd6400a12c.png)
## Tools
+ **WinSCP** - This tool enables Windows users to easily share files with Linux users.
+ **Sqlcmd** - This command-line utility lets you submit T-SQL statements or batches to local and remote instances of SQL Server. The utility is extremely useful for repetitive database tasks such as batch processing or unit testing.
## Epics
### Prepare the EC2 Linux instance with SQL Server
| Task | Description | Skills required |
| --- | --- | --- |
| Select an AMI that provides the Linux operating system and includes Microsoft SQL Server. | | Sysadmin |
| Configure the AMI to create an EC2 instance. | | Sysadmin |
| Create inbound and outbound rules for security groups. | | Sysadmin |
| Configure the Linux EC2 instance for a Microsoft SQL Server database. | | DBA |
| Create users and provide permissions as in the source database. | | Appowner, DBA |
| Install SQL Server tools and the sqlcmd utility on the Linux EC2 instance. | | DBA |
### Back up the database and move backup file to Linux EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Back up the on-premises SQL Server database. | | DBA |
| Install WinSCP on Microsoft SQL Server. | | DBA |
| Move the backup file to the Linux EC2 instance running Microsoft SQL Server. | | DBA |
### Restore the database on Linux EC2 instance running SQL Server
| Task | Description | Skills required |
| --- | --- | --- |
| Restore the database from the database backup file by using the sqlcmd utility. | | DBA |
| Validate database objects and data. | | Developer, Test engineer |
### Cut over from Windows SQL Server to Windows SQL Server on Linux EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Validate database objects and data. | | Developer, Test engineer |
| Cut over from the on-premises Microsoft SQL Server database to the Linux EC2 instance running Microsoft SQL Server. | | DBA |
## Related resources
+ [How to configure SQL Server 2017 on Amazon Linux and Ubuntu AMIs](https://aws.amazon.com/blogs/database/configuring-sql-server-2017-on-amazon-linux-2-and-ubuntu-amis/)
+ [Installation of SQL tools on a Linux instance](https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-tools?view=sql-server-2017#RHEL)
+ [Backup and restoration from an on-premises Microsoft SQL Server database to Microsoft SQL Server on a Linux EC2 instance](https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-migrate-restore-database?view=sql-server-2017#create-a-backup-on-windows)
# Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using linked servers
*Kevin Yung, Viqash Adwani, and Vishal Singh, Amazon Web Services*
## Summary
Linked servers enable Microsoft SQL Server to run SQL statements on other instances of database servers. This pattern describes how you can migrate your on-premises Microsoft SQL Server database to Amazon Relational Database Service (Amazon RDS) for Microsoft SQL Server to achieve lower cost and higher availability. Currently, Amazon RDS for Microsoft SQL Server doesn't support connections outside an Amazon Virtual Private Cloud (Amazon VPC) network.
You can use this pattern to achieve the following objectives:
+ To migrate Microsoft SQL Server to Amazon RDS for Microsoft SQL Server without breaking linked server capabilities.
+ To prioritize and migrate linked Microsoft SQL Server in different waves.
## Prerequisites and limitations
**Prerequisites**
+ Check whether [Microsoft SQL Server on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html) supports the features you require.
+ Make sure that you can use either [Amazon RDS for Microsoft SQL Server with default collations or collations set over database levels](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.SQLServer.CommonDBATasks.Collation.html).
## Architecture
**Source technology stack**
+ On-premises databases (Microsoft SQL Server)
**Target technology stack**
+ Amazon RDS for SQL Server
**Source state architecture**
![\[Diagram showing data replication between two data centers with primary and secondary SQL servers.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/95234758-cb8b-46e5-afd2-3d4aaf6ed668/images/776b453a-7fa0-43fd-b1ca-fb9e5cc21820.png)
**Target state architecture**
In the target state, you migrate Microsoft SQL Server to Amazon RDS for Microsoft SQL Server by using linked servers. This architecture uses a Network Load Balancer to proxy the traffic from Amazon RDS for Microsoft SQL Server to on-premises servers running Microsoft SQL Server. The following diagram shows the reverse proxy capability for the Network Load Balancer.
![\[AWS Cloud architecture with RDS SQL Server instances in two availability zones and on-premises databases.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/95234758-cb8b-46e5-afd2-3d4aaf6ed668/images/6bdbdfbf-b048-4fbd-acef-0aeb826edb50.png)
## Tools
+ AWS CloudFormation
+ Network Load Balancer
+ Amazon RDS for SQL Server in multiple Availability Zones (Multi-AZs)
+ AWS Database Migration Service (AWS DMS)
## Epics
### Create a landing zone VPC
| Task | Description | Skills required |
| --- | --- | --- |
| Create the CIDR allocation. | | AWS SysAdmin |
| Create a virtual private cloud (VPC). | | AWS SysAdmin |
| Create the VPC subnets. | | AWS SysAdmin |
| Create the subnet access control lists (ACLs). | | AWS SysAdmin |
| Create the subnet route tables. | | AWS SysAdmin |
| Create a connection with AWS Direct Connect or AWS Virtual Private Network (VPN). | | AWS SysAdmin |
### Migrate the database to Amazon RDS
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon RDS for Microsoft SQL Server DB instance. | | AWS SysAdmin |
| Create an AWS DMS replication instance. | | AWS SysAdmin |
| Create the source and target database endpoints in AWS DMS. | | AWS SysAdmin |
| Create the migration task and set continuous replication to ON after a full load. | | AWS SysAdmin |
| Request a firewall change to allow Amazon RDS for Microsoft SQL Server to access the on-premises SQL Server databases. | | AWS SysAdmin |
| Create a Network Load Balancer. | | AWS SysAdmin |
| Create a target group that targets the database servers in your data center | We recommend that you use hostnames in the target setup to incorporate data center (DC) failover events. | AWS SysAdmin |
| Run the SQL statement for linked server setup. | Run the SQL statements for adding a linked server by using the Microsoft SQL management tool against the Amazon RDS for Microsoft SQL Server DB instance. In the SQL statement, set @datasrc to use the Network Load Balancer hostname. Add linked server login credentials by using the Microsoft SQL management tool against the Amazon RDS for Microsoft SQL Server DB instance. | AWS SysAdmin |
| Test and validate the SQL Server functions. | | AWS SysAdmin |
| Create a cutover. | | AWS SysAdmin |
## Related resources
+ [Common Management Tasks for Microsoft SQL Server on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General)
+ [Collations and Character Sets for Microsoft SQL Server](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.SQLServer.CommonDBATasks.Collation.html)
+ [Network Load Balancer documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html)
+ [Implement Linked Servers with Amazon RDS for Microsoft SQL Server (blog post)](https://aws.amazon.com/blogs/database/implement-linked-servers-with-amazon-rds-for-microsoft-sql-server/)
# Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using native backup and restore methods
*Tirumala Dasari, David Queiroz, and Vishal Singh, Amazon Web Services*
## Summary
This pattern describes how to migrate an on-premises Microsoft SQL Server database to an Amazon Relational Database Service (Amazon RDS) for SQL Server DB instance (homogeneous migration). The migration process is based on native SQL Server backup and restore methods. It uses SQL Server Management Studio (SSMS) to create a database backup file, and an Amazon Simple Storage Service (Amazon S3) bucket to store the backup file before restoring it in Amazon RDS for SQL Server.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ AWS Identity and Access Management (IAM) role policies to access the S3 bucket and the Amazon RDS for SQL Server DB instance.
**Limitations**
+ The process described in this pattern migrates only the database. SQL logins or database users, including any SQL Server Agent jobs, aren’t migrated because they require additional steps.
**Product versions**
+ SQL Server 2012-2017. For the latest list of supported versions and features, see [Microsoft SQL Server on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.FeatureSupport) in the AWS documentation.
## Architecture
**Source technology stack**
+ An on-premises Microsoft SQL Server database
**Target technology stack**
+ Amazon RDS for SQL Server DB instance
**Data migration**** architecture**
![\[Architecture to migrate an on-premises SQL Server DB to an Amazon RDS for SQL Server DB instance.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/c2dcd6ab-deb1-4d5e-b3c5-3bf48c02ca4e/images/29f90473-6dd4-4574-bfbd-5c6a0481c40e.png)
## Tools
+ Microsoft SQL Server Management Studio (SSMS) is an integrated environment for managing SQL Server infrastructure. It provides a user interface and a group of tools with rich script editors that interact with SQL Server.
## Epics
### Create an Amazon RDS for SQL Server DB instance
| Task | Description | Skills required |
| --- | --- | --- |
| Select SQL Server as the database engine in Amazon RDS for SQL Server. | | DBA |
| Choose the SQL Server Express Edition. | | DBA |
| Specify database details. | For more information about creating a DB instance, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html). | DBA, App owner |
### Create a backup file from the on-premises SQL Server database
| Task | Description | Skills required |
| --- | --- | --- |
| Connect to the on-premises SQL Server database through SSMS. | | DBA |
| Create a backup of the database. | For instructions, see the [SSMS documentation](https://learn.microsoft.com/en-us/sql/ssms/sql-server-management-studio-ssms). | DBA, App owner |
### Upload the backup file to Amazon S3
| Task | Description | Skills required |
| --- | --- | --- |
| Create a bucket in Amazon S3. | For more information, see the [Amazon S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). | DBA |
| Upload the backup file to the S3 bucket. | For more information, see the [Amazon S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). | SysOps administrator |
### Restore the database in Amazon RDS for SQL Server
| Task | Description | Skills required |
| --- | --- | --- |
| Add the option group to Amazon RDS. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-native-backup-and-restore-methods.html)For more information, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithOptionGroups.html). | SysOps administrator |
| Restore the database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-native-backup-and-restore-methods.html) | DBA |
### Validate the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Validate objects and data. | Validate the objects and data between the source database and Amazon RDS for SQL Server.This task migrates the database only. Logins and jobs will not be migrated. | App owner, DBA |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Redirect application traffic. | After validation, redirect application traffic to the Amazon RDS for SQL Server DB instance. | App owner, DBA |
## Related resources
+ [Amazon S3 documentation](https://docs.aws.amazon.com/s3/)
+ [Amazon RDS for SQL Server documentation ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html)
+ [Options for the Microsoft SQL Server Database Engine](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.SQLServer.Options.html)
# Migrate a Microsoft SQL Server database to Aurora MySQL by using AWS DMS and AWS SCT
*Mark Szalkiewicz and Pavan Pusuluri, Amazon Web Services*
## Summary
This pattern describes how to migrate a Microsoft SQL Server database that is either on premises or on an Amazon Elastic Compute Cloud (Amazon EC2) instance to Amazon Aurora MySQL. The pattern uses AWS Database Migration Service (AWS DMS) and AWS Schema Conversion Tool (AWS SCT) for data migration and schema conversion.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A Microsoft SQL Server source database in an on-premises data center or on an EC2 instance
+ Java Database Connectivity (JDBC) drivers for AWS SCT connectors, installed on either a local machine or an EC2 instance where AWS SCT is installed
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ Microsoft SQL Server 2008, 2008R2, 2012, 2014, 2016, and 2017 for the Enterprise, Standard, Workgroup, and Developer editions. The Web and Express editions aren't supported by AWS DMS. For the latest list of supported versions, see [Using a Microsoft SQL Server Database as a Source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support. For information about Microsoft SQL Server versions supported by AWS SCT, see the [AWS SCT documentation](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html).
+ MySQL versions 5.5, 5.6, and 5.7. For the latest list of supported versions, see [Using a MySQL-Compatible Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html).
## Architecture
**Source technology stack**
One of the following:
+ An on-premises Microsoft SQL Server database
+ A Microsoft SQL Server database on an EC2 instance
**Target technology stack**
+ Aurora MySQL
**Data migration architecture**
+ From a Microsoft SQL Server database running in the AWS Cloud
![\[AWS Cloud architecture showing VPC with private subnet containing SQL Server and Aurora MySQL databases.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e2de4507-82a8-4bd6-b25b-1e830b197b9f/images/c675ada4-e92c-4ddb-b49f-69668f532504.png)
+ From a Microsoft SQL Server database running in an on-premises data center
![\[AWS Cloud architecture diagram showing on-premises to cloud migration using AWS SCT, DMS, and Aurora MySQL.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e2de4507-82a8-4bd6-b25b-1e830b197b9f/images/b6ce0199-fc56-4bf2-a8cc-67de161e3cf0.png)
## Tools
+ **AWS DMS** - [AWS Data Migration Service](http://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html) (AWS DMS) helps you migrate your data to and from widely used commercial and open-source databases, including Oracle, SQL Server, MySQL, and PostgreSQL. You can use AWS DMS to migrate your data into the AWS Cloud, between on-premises instances (through an AWS Cloud setup), or between combinations of cloud and on-premises setups.
+ **AWS SCT** - [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) (AWS SCT) makes heterogeneous database migrations easy by automatically converting the source database schema and a majority of the custom code to a format compatible with the target database.
## Epics
### Prepare for the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the source and target database version and engine. | | DBA |
| Create an outbound security group for the source and target databases. | | SysAdmin |
| Create and configure an EC2 instance for AWS SCT, if required. | | DBA |
| Download the latest version of AWS SCT and associated drivers. | | DBA |
| Add and validate the prerequisite users and grants in the source database. | | DBA |
| Create an AWS SCT project for the workload and connect to the source database. | | DBA |
| Generate an assessment report and evaluate feasibility. | | DBA |
### Prepare the target database
| Task | Description | Skills required |
| --- | --- | --- |
| Create a target Amazon RDS DB instance, using Amazon Aurora as the database engine. | | DBA |
| Extract the list of users, roles, and grants from the source. | | DBA |
| Map the existing database users to the new database users. | | App owner |
| Create users in the target database. | | DBA |
| Apply roles from the previous step to the target database. | | DBA |
| Review the database options, parameters, network files, and database links in the source database, and then evaluate their applicability to the target database. | | DBA |
| Apply any relevant settings to the target. | | DBA |
### Transfer objects
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS SCT connectivity to the target database. | | DBA |
| Convert the schema using AWS SCT. | AWS SCT automatically converts the source database schema and most of the custom code to a format that is compatible with the target database. Any code that the tool cannot convert automatically is clearly marked so that you can convert it yourself. | DBA |
| Review the generated SQL report and save any errors and warnings. | | DBA |
| Apply automated schema changes to the target or save them as a .sql file. | | DBA |
| Validate that AWS SCT created the objects on the target. | | DBA |
| Manually rewrite, reject, or redesign any items that failed to convert automatically. | | DBA |
| Apply the generated role and user grants and review any exceptions. | | DBA |
### Migrate the data
| Task | Description | Skills required |
| --- | --- | --- |
| Determine the migration method. | | DBA |
| Create a replication instance from the AWS DMS console. | For detailed information on using AWS DMS, see the links in the "Related resources" section. | DBA |
| Create the source and target endpoints. | | DBA |
| Create a replication task. | | DBA |
| Start the replication task and monitor the logs. | | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Use AWS SCT to analyze and convert the SQL items within the application code. | When you convert your database schema from one engine to another, you also need to update the SQL code in your applications to interact with the new database engine instead of the old one. You can view, analyze, edit, and save the converted SQL code. For detailed information on using AWS SCT, see the links in the "Related resources" section. | App owner |
| Create the new application servers on AWS. | | App owner |
| Migrate the application code to the new servers. | | App owner |
| Configure the application server for the target database and drivers. | | App owner |
| Fix any code that's specific to the source database engine in the application. | | App owner |
| Optimize the application code for the target engine. | | App owner |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Apply any new users, grants, and code changes to the target. | | DBA |
| Lock the application for any changes. | | App owner |
| Validate that all changes were propagated to the target database. | | DBA |
| Point the new application server to the target database. | | App owner |
| Recheck everything. | | App owner |
| Go live. | | App owner |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary AWS resources (AWS DMS replication instance and EC2 instance used for AWS SCT). | | DBA, App owner |
| Update feedback on the AWS DMS process for internal teams. | | DBA, App owner |
| Revise the AWS DMS process and improve the template if necessary. | | DBA, App owner |
| Review and validate the project documents. | | DBA, App owner |
| Gather metrics around time to migrate, percent of manual versus tool cost savings, and so on. | | DBA, App owner |
| Close the project and provide any feedback. | | DBA, App owner |
## Related resources
**References**
+ [AWS DMS User Guide](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [AWS SCT User Guide](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon Aurora Pricing](https://aws.amazon.com/rds/aurora/pricing/)
**Tutorials and videos**
+ [Getting Started with AWS Database Migration Service](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with the AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
+ [Amazon RDS resources](https://aws.amazon.com/rds/getting-started/)
+ [AWS DMS Step-by-Step Walkthroughs](http://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html)
# Migrate an on-premises MariaDB database to Amazon RDS for MariaDB using native tools
*Shyam Sunder Rakhecha, Amazon Web Services*
## Summary
This pattern provides guidance for migrating an on-premises MariaDB database to Amazon Relational Database Service (Amazon RDS) for MariaDB by using native tools. If you have MySQL tools installed, you can use **mysql **and **mysqldump**. If you have MariaDB tools installed, you can use **mariadb** and **mariadb-dump**. MySQL and MariaDB tools have the same origin, but there are minor differences in MariaDB version 10.6 and later.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A MariaDB source database in an on-premises data center
**Limitations**
+ Database size limit: 64 TB
**Product versions**
+ MariaDB versions 10.0-10.6 (for the latest list of supported versions, see [MariaDB on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html#MariaDB.Concepts.VersionMgmt) in the AWS documentation)
## Architecture
**Source technology stack**
+ MariaDB database in an on-premises data center
**Target technology stack**
+ Amazon RDS for MariaDB DB instance
**Target architecture**
![\[Architecture diagram with primary and standby RDS DB instances in different Availability Zones.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7be644e0-da42-4515-87b7-04da7a054adb/images/eca8eb55-579a-42e2-96ce-9b14b097b4c9.png)
**Data migration architecture**
![\[Architecture diagram of migrating an on-premises MariaDB database to Amazon RDS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7be644e0-da42-4515-87b7-04da7a054adb/images/daba40e2-a2b1-44f8-8e69-31458206a823.png)
## Tools
+ Native MySQL tools: **mysql** and **mysqldump**
+ Native MariaDB tools: **mariadb** and **mariadb-dump**
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate source and target database versions and engines. | | DBA |
| Identify hardware requirements for the target server instance. | | DBA, Systems administrator |
| Identify storage requirements (storage type and capacity). | | DBA, Systems administrator |
| Choose the proper instance type based on capacity, storage features, and network features. | | DBA, Systems administrator |
| Identify the network access security requirements for source and target databases. | | DBA, Systems administrator |
| Identify the application migration strategy. | | DBA, App owner, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | | Systems administrator |
| Create security groups. | | Systems administrator |
| Configure and start an Amazon RDS DB instance running MariaDB. | | Systems administrator |
### Migrate data
| Task | Description | Skills required |
| --- | --- | --- |
| Use native tools to migrate database objects and data. | In the source database, use **mysqldump **or** mariadb-dump** to create an output file that contains database objects and data. In the target database, use **mysql **or **mariadb **to restore the data. | DBA |
| Validate the data. | Check the source and target databases to confirm that the data migration was successful. | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the application migration strategy. | | DBA, App owner, Systems administrator |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch the application clients over to the new infrastructure. | | DBA, App owner, Systems administrator |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down the temporary AWS resources. | | Systems administrator |
| Review and validate the project documents. | | DBA, App owner, Systems administrator |
| Gather metrics around time to migrate, cost savings provided by tools, and so on. | | DBA, App owner, Systems administrator |
| Close out the project and provide feedback. | | DBA, App owner, Systems administrator |
## Related resources
**Amazon RDS references**
+ [Amazon RDS for MariaDB](https://aws.amazon.com/rds/mariadb/)
+ [Amazon Virtual Private Cloud VPCs and Amazon RDS](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.html)
+ [Amazon RDS Multi-AZ Deployments](https://aws.amazon.com/rds/details/multi-az/)
+ [Amazon RDS Pricing](https://aws.amazon.com/rds/pricing/)
**MySQL and MariaDB references**
+ [mariadb-dump/mysqldump](https://mariadb.com/kb/en/mariadb-dumpmysqldump/)
+ [mysql Command-line Client](https://mariadb.com/kb/en/mysql-command-line-client/)
**Tutorials and videos**
+ [Getting Started with Amazon RDS](https://aws.amazon.com/rds/getting-started/)
# Migrate an on-premises MySQL database to Aurora MySQL
*Igor Obradovic, Amazon Web Services*
## Summary
This pattern explains how to migrate an on-premises MySQL source database to Amazon Aurora MySQL-Compatible Edition. It describes two options for migration: using AWS Database Migration Service (AWS DMS) or using native MySQL tools such as **mysqldbcopy** and **mysqldump**.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A source MySQL database in an on-premises data center
**Limitations**
+ Database size limit: 128 TB
**Product versions**
+ MySQL version 8.0 (Aurora MySQL version 3) is available under standard support.
+ MySQL version 5.7 (Aurora MySQL version 2) is available under extended support, for an additional cost.
For the latest list of supported versions, see [Amazon Aurora versions](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraMySQLReleaseNotes/AuroraMySQL.release-calendars.html) in the AWS documentation. If you're using AWS DMS, see also [Using a MySQL-Compatible Database as a Target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html) for MySQL versions supported by AWS DMS.
## Architecture
**Source technology stack**
+ An on-premises MySQL database
**Target technology stack**
+ Amazon Aurora MySQL-Compatible Edition
**Target architecture**
Aurora data is stored in a cluster volume, which is a single, virtual volume that uses solid state drives (SSDs). A cluster volume consists of copies of the data across three Availability Zones in a single AWS Region. Because the data is automatically replicated across Availability Zones, it is highly durable with less possibility of data loss.
Aurora automatically divides your database volume into 10 GB segments spread across many disks. Each 10 GB chunk of your database volume is replicated six ways, across three Availability Zones. The following diagram illustrates the relationship between the cluster volume, the writer DB instance, and reader DB instances in an Aurora DB cluster, and the separation of compute capacity and storage. For more information about this architecture, see the [Aurora documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Overview.html) and [FAQ](https://aws.amazon.com/rds/aurora/faqs/#product-faqs).
![\[Aurora MySQL DB instances and shared storage volume on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/22729803-e4ff-45a2-ab5b-8ba2445e5e21/images/0d7d8ebd-e0f2-4bcf-b296-8bdfb2f12b64.png)
**Data migration architecture**
*Using AWS DMS:*
The following diagram illustrates the migration of an on-premises MySQL database to an Aurora MySQL-Compatible cluster in the AWS Cloud, using AWS DMS.
![\[Migrating an on-premises MySQL database to Aurora MySQL by using AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/22729803-e4ff-45a2-ab5b-8ba2445e5e21/images/e5d72ebd-d157-45d7-8844-d1011f1646c0.png)
*Using native MySQL tools:*
The following diagram illustrates the migration of an on-premises MySQL database to an Aurora MySQL-Compatible cluster in the AWS Cloud, using native MySQL tools such as **mysqldbcopy** and **mysqldump**.
![\[Migrating an on-premises MySQL database to Aurora MySQL by using mysqldbcopy and mysqldump.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/22729803-e4ff-45a2-ab5b-8ba2445e5e21/images/26258752-24f6-4241-a49f-59c15e946314.png)
## Tools
+ [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) supports several source and target database engines. For information about MySQL source and target databases supported by AWS DMS, see [Migrating MySQL-Compatible Databases to AWS](https://docs.aws.amazon.com/dms/latest/sbs/CHAP_MySQL.html). We recommend that you use the latest version of AWS DMS for the most comprehensive version and feature support.
+ [mysqldbcopy](https://manpages.ubuntu.com/manpages/focal/man1/mysqldbcopy.1.html) is a MySQL utility that copies a MySQL database on a single server or between servers.
+ [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) is a MySQL utility that creates a dump file from a MySQL database for backup or migration purposes.
## Epics
### Plan the migration
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the version and engine. | Validate the database version and engine for the source and target databases. | DBA |
| Identify hardware requirements. | Identify hardware requirements for the target server instance. | DBA, Systems administrator |
| Identify storage requirements. | Identify storage requirements (storage type and capacity). | DBA, Systems administrator |
| Choose the instance type. | Choose the proper instance type based on your compute, storage, and network requirements. | DBA, Systems administrator |
| Determine network access security requirements. | Identify the network access security requirements for the source and target databases. | DBA, Systems administrator |
| Determine strategy. | Identify the application migration strategy. | DBA, App owner, Systems administrator |
### Configure the infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create a virtual private cloud (VPC). | For instructions, see [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html) in the Amazon Virtual Private Cloud (Amazon VPC) documentation. | Systems administrator |
| Create security groups. | For instructions, see [Create a security group for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/creating-security-groups.html) in the Amazon VPC documentation. | Systems administrator |
| Configure and start an Aurora MySQL-Compatible DB cluster in your AWS account. | For instructions, see [Creating an Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html) in the Aurora documentation. | Systems administrator |
### Migrate data - option 1
| Task | Description | Skills required |
| --- | --- | --- |
| Use native MySQL tools or third-party tools to migrate database objects and data. | For instructions, see the documentation for MySQL tools such as [mysqldbcopy](https://manpages.ubuntu.com/manpages/focal/man1/mysqldbcopy.1.html) and [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html). | DBA |
### Migrate data - option 2
| Task | Description | Skills required |
| --- | --- | --- |
| Migrate data with AWS DMS. | For instructions, see [Using a MySQL-compatible database as a source](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html) and [Using a MySQL-compatible database as a target](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html) in the AWS DMS documentation. | DBA |
### Migrate the application
| Task | Description | Skills required |
| --- | --- | --- |
| Follow the strategy. | Follow the application migration strategy. | DBA, App owner, Systems administrator |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Switch application clients. | Switch the application clients over to connect to the new Aurora cluster endpoint. | DBA, App owner, Systems administrator |
### Close the project
| Task | Description | Skills required |
| --- | --- | --- |
| Shut down resources. | Shut down temporary AWS resources. | DBA, Systems administrator |
| Review documentation. | Review and validate the project documents. | DBA, App owner, Systems administrator |
| Collect metrics. | Gather metrics around time to migrate, percentage of manual steps versus tool usage, cost savings, and so on. | DBA, App owner, Systems administrator |
| Complete migration project. | Close out the project and provide feedback. | App owner, DBA, Systems administrator |
## Related resources
**References**
+ [Migrating data to Amazon Aurora MySQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.html)
+ [AWS DMS website](https://aws.amazon.com/dms/)
+ [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html)
+ [Amazon Aurora Pricing](https://aws.amazon.com/rds/aurora/pricing/)
+ [Creating and connecting to an Aurora MySQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html)
+ [Amazon VPC and Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.html)
+ [Amazon Aurora documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html)
**Tutorials and videos**
+ [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/)
+ [Getting Started with Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.html)
# Migrate on-premises MySQL databases to Aurora MySQL using Percona XtraBackup, Amazon EFS, and Amazon S3
*Rohan Jamadagni, Udayasimha Theepireddy, and sajith menon, Amazon Web Services*
## Summary
This pattern describes how to migrate large, on-premises MySQL databases efficiently to Amazon Aurora MySQL by using Percona XtraBackup. Percona XtraBackup is an open-source, non-blocking backup utility for MySQL-based servers. The pattern shows how to use Amazon Elastic File System (Amazon EFS) to reduce the time to upload the backup to Amazon Simple Storage Service (Amazon S3) and to restore the backup to Amazon Aurora MySQL. The pattern also provides details on how to make incremental Percona backups to minimize the number of binary logs to be applied to the target Aurora MySQL database.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ Permissions to create AWS Identity and Access Management (IAM) roles and policies
+ Network connectivity between the on-premises MySQL database and the virtual private cloud (VPC) on AWS
**Limitations **
+ The source servers must be Linux-based systems that can install a Network File System (NFS) client (nfs-utils/nfs-common).
+ The S3 bucket used for uploading backup files supports server-side encryption (SSE-S3/SSE-KMS) only.
+ Amazon S3 limits the size of the backup files to 5 TB. If your backup file exceeds 5 TB, you can split it into multiple, smaller files.
+ The number of source files uploaded to the S3 bucket cannot exceed one million files.
+ The pattern supports Percona XtraBackup full backup and incremental backup only. It doesn't support partial backups that use `--tables`, `--tables-exclude`, `--tables-file`, `--databases`, `--databases-exclude`, or `--databases-file`.
+ Aurora doesn't restore users, functions, stored procedures, or time zone information from the source MySQL database.
**Product versions**
+ The source database must be MySQL version 5.5, 5.6, or 5.7.
+ For MySQL 5.7, you must use Percona XtraBackup 2.4.
+ For MySQL 5.6 and 5.6, you must use Percona XtraBackup 2.3 or 2.4.
## Architecture
**Source technology stack**
+ Linux-based operating system
+ MySQL server
+ Percona XtraBackup
**Target technology stack**
+ Amazon Aurora
+ Amazon S3
+ Amazon EFS
**Target architecture**
![\[Architecture to migrate large MySQL databases to Amazon Aurora MySQL by using Percona XtraBackup.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bf327776-bafd-484d-9ae2-a6f5c8af6edd/images/7a410539-1511-4106-90e2-8c0c8e95f92b.png)
## Tools
*AWS services*
+ [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMySQL.html) is a fully managed relational database engine that makes it simple and cost-effective to set up, operate, and scale MySQL deployments. Aurora MySQL is a drop-in replacement for MySQL.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Other tools**
+ [Percona XtraBackup](https://www.percona.com/doc/percona-xtrabackup/2.4/index.html) is an open-source utility that performs streaming, compressed, and incremental backups of MySQL databases without disrupting or blocking your databases.
## Epics
### Create an Amazon EFS file system
| Task | Description | Skills required |
| --- | --- | --- |
| Create a security group to associate with Amazon EFS mount targets. | Create a security group in the VPC that is configured with a VPN attachment to the on-premises database over AWS Transit Gateway. For more information about the commands and steps described in this and other stories, see the links in the "Related resources" section at the end of this pattern. | AWS DevOps/database administrator |
| Edit the security group rules. | Add an inbound rule, using type NFS, port 2049, and the IP range of the on-premises database server as the source. By default, the outbound rule allows all the traffic to leave. If this is not the case , add an outbound rule to open a connection for the NFS port. Add two more inbound rules: port 2049 (source: security group ID of this same security group) and port 22 (source: IP range from where you will connect to an EC2 instance). | AWS DevOps/database administrator |
| Create a file system. | In the mount targets, use the VPC and security group you created in the previous story. Choose the throughput mode and performance based on the I/O requirements of the on-premises database. Optionally, enable encryption at rest. | AWS DevOps/database administrator |
### Mount the file system
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM instance profile role to be associated with an EC2 instance. | Create an IAM role that has permissions to upload and access objects in Amazon S3. Choose the S3 bucket where the backup will be stored as a policy resource. | AWS DevOps |
| Create an EC2 instance. | Launch an Linux-based EC2 instance and attach the IAM instance profile role that you created in the previous step, and the security group you created earlier. | AWS DevOps |
| Install the NFS client. | Install the NFS client on the on-premises database server and on the EC2 instance. For installation instructions, see the "Additional information" section. | DevOps |
| Mount the Amazon EFS file system. | Mount the Amazon EFS file system on premises and on the EC2 instance. On each server, create a directory for storing the backup, and mount the file system by using the mount target endpoint. For an example, see the "Additional information" section. | DevOps |
### Make a backup of the MySQL source database
| Task | Description | Skills required |
| --- | --- | --- |
| Install Percona XtraBackup. | Install Percona XtraBackup 2.3 or 2.4 (depending on the version of your MySQL database) on the on-premises database server. For installation links, see the "Related resources" section. | Database administrator |
| Count the schemas and tables in the source database. | Gather and note the number of schemas and objects in the source MySQL database. You will use these counts to validate the Aurora MySQL database after migration. | Database administrator |
| (Optional) Note the latest binary log sequence from the source database. | Perform this step if you want to establish binary log replication between the source database and Aurora MySQL to minimize downtime. log-bin must be enabled, and server\$1id must be unique. Note the current binary log sequence from the source database, just before initiating a backup. Perform this step just before full backup if you're planning to use only full backup. If you're planning to make incremental backups after a full backup, perform this step just before the final incremental backup that you will restore on the Aurora MySQL DB instance. | Database administrator |
| Start a full backup of the source MySQL database. | Make a full backup of the MySQL source database using Percona XtraBackup. For example commands for full and incremental backups, see the "Additional information" section. | Database administrator |
| (Optional) Make incremental backups using Percona XtraBackup. | Incremental backups can be used to reduce the amount of binary logs you need to apply to sync the source database with Aurora MySQL. Large-size and transaction-heavy databases might generate a large number of binary logs during backups. By taking incremental backups and storing them on a shared Amazon EFS file system, you can significantly reduce the time for backing up and uploading your database. For details, see the "Additional information" section. Continue to make incremental backups until you're ready to begin the migration process to Aurora. | Database administrator |
| Prepare backups. | In this step, transactional logs are applied to the backup for transactions that were in flight during the backup. Continue to apply transactional logs (--apply-log-only) to each incremental backup to merge the backups, except for the last backup. For examples, see the "Additional information" section. After this step, the complete, merged backup will be in \$1//fullbackup. | Database administrator |
| Zip and split the final merged backup. | After you prepare the final, merged backup, use tar, zip, and split commands to create smaller zipped files of the backup. For examples, see the "Additional information" section. | Database administrator |
### Restore the backup to an Aurora MySQL DB cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Upload the backup to Amazon S3. | The Amazon EFS file system where the backup files are stored is mounted on both the on-premises database and an EC2 instance, so the backup files are readily available to the EC2 instance. Connect to the EC2 instance by using Secure Shell (SSH) and upload the zipped backup files to a new or existing S3 bucket; for example: aws s3 sync \$1//fullbackup s3:///fullbackup. For additional details, see the links in the "Related resources" section. | AWS DevOps |
| Create a service role for Aurora to access Amazon S3. | Create an IAM role with trust "rds.amazonaws.com" and a policy that will enable Aurora to access the S3 bucket where the backup files are stored. The required permissions are ListBucket, GetObject, and GetObjectVersion. | AWS DevOps |
| Create the networking configuration for Aurora. | Create a cluster DB subnet group with at least two Availability Zones and a subnet route table configuration that allows outbound connectivity to the source database. Create a security group that allows outbound connections to the on-premises database, and allows administrators to connect to the Aurora DB cluster. For more information, see the links in the "Related resources" section. | AWS DevOps/database administrator |
| Restore the backup to an Aurora MySQL DB cluster. | Restore your data from the backup that you uploaded to Amazon S3. Specify the MySQL version of your source database, provide the S3 bucket name and folder path prefix where you uploaded the backup file (for example, "fullbackup" for the examples in the "Additional information" section), and provide the IAM role you created to authorize Aurora to access Amazon S3. | AWS DevOps/database administrator |
| Validate the Aurora MySQL database. | Validate the count of schema and objects in the restored Aurora DB cluster against the count you obtained from the source database. | Database administrator |
| Set up binlog replication. | Use the binary log sequence that you noted earlier, before making the last backup that was restored to the Aurora DB cluster. Create a replication user on the source database, and follow the instructions in the "Additional information" section to provide the appropriate privileges, to enable replication on Aurora, and to confirm that the replication is in sync. | AWS DevOps/database administrator |
## Related resources
**Creating an Amazon EFS file system**
+ [Creating a security group](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#CreatingSecurityGroups) (Amazon VPC documentation)
+ [Transit gateway VPN attachments](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpn-attachments.html) (Amazon VPC documentation)
+ [Scaling VPN throughput using AWS Transit Gateway](https://aws.amazon.com/blogs/networking-and-content-delivery/scaling-vpn-throughput-using-aws-transit-gateway/) (Networking & Content Delivery blog)
+ [Creating an Amazon EFS file system](https://docs.aws.amazon.com/efs/latest/ug/efs-onpremises.html#wt5-step1-efs) (Amazon EFS documentation)
+ [Creating mount targets](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs.html) (Amazon EFS documentation)
+ [Encrypting data at rest](https://docs.aws.amazon.com/efs/latest/ug/encryption-at-rest.html) (Amazon EFS documentation)
**Mounting the file system**
+ [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) (Amazon EC2 documentation)
+ [Launching an Amazon EC2 Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance) (Amazon EC2 documentation)
+ [Installing the NFS client](https://docs.aws.amazon.com/efs/latest/ug/efs-onpremises.html#wt5-step4-install-nfs) (Amazon EFS documentation)
+ [Mounting the Amazon EFS file system on your on-premises client](https://docs.aws.amazon.com/efs/latest/ug/efs-onpremises.html#wt5-step3-connect) (Amazon EFS documentation)
+ [Mounting EFS File Systems](https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html) (Amazon EFS documentation)
**Making a backup of the MySQL source database**
+ [Installing Percona XtraBackup 2.3](https://www.percona.com/doc/percona-xtrabackup/2.3/installation.html) (Percona XtraBackup documentation)
+ [Installing Percona XtraBackup 2.4](https://www.percona.com/doc/percona-xtrabackup/2.4/installation.html) (Percona XtraBackup documentation)
+ [Setting the replication master configuration](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) (MySQL documentation)
+ [Migrating data from an external MySQL database to an Aurora MySQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.ExtMySQL.html) (Aurora documentation)
+ [Incremental backup](https://www.percona.com/doc/percona-xtrabackup/2.4/backup_scenarios/incremental_backup.html) (Percona XtraBackup documentation)
**Restoring the backup to Amazon Aurora MySQL**
+ [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#create-bucket-intro) (Amazon S3 documentation)
+ [Connecting to your Linux instance using SSH](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html) (Amazon Ec2 documentation)
+ [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) (AWS CLI documentation)
+ [sync command](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) (AWS CLI command reference)
+ [Creating an IAM policy to access Amazon S3 resources](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.Authorizing.IAM.S3CreatePolicy.html) (Aurora documentation)
+ [DB cluster prerequisites](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html#Aurora.CreateInstance.Prerequisites) (Aurora documentation)
+ [Working with DB subnet groups](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Subnets) (Aurora documentation)
+ [Creating a VPC security group for a private DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Tutorials.WebServerDB.CreateVPC.html#CHAP_Tutorials.WebServerDB.CreateVPC.SecurityGroupDB) (Aurora documentation)
+ [Restoring an Aurora MySQL DB cluster from an S3 bucket](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.ExtMySQL.html#AuroraMySQL.Migrating.ExtMySQL.S3.Restore) (Aurora documentation)
+ [Setting up replication with MySQL or another Aurora DB cluster ](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Replication.MySQL.html#AuroraMySQL.Replication.MySQL.SettingUp) (Aurora documentation)
+ [mysql.rds\$1set\$1external\$1master procedure](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql_rds_set_external_master.html) (MySQL on Amazon RDS SQL reference)
+ [mysql.rds\$1start\$1replication procedure](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql_rds_start_replication.html) (MySQL on Amazon RDS SQL reference)
**Additional references**
+ [Migrating data from an external MySQL database to an Aurora MySQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.ExtMySQL.html) (Aurora documentation)
+ [MySQL server downloads](https://downloads.mysql.com/archives/community/) (Oracle website)
**Tutorials and videos **
+ [Migrating MySQL data to an Aurora MySQL DB cluster using Amazon S3](https://aws.amazon.com/premiumsupport/knowledge-center/migrate-mysql-aurora-innobackup/) (AWS Knowledge Center)
+ [Amazon EFS setup and mount](https://www.youtube.com/watch?v=NR8rVsSn_dY) (video)
## Additional information
**Installing an NFS client**
+ If you are using Red Hat or a similar Linux operating system, use the command:
```
$ sudo yum -y install nfs-utils
```
+ If you are using Ubuntu or a similar Linux operating system, use the command:
```
$ sudo apt-get -y install nfs-common
```
For more information, see the [walkthrough](https://docs.aws.amazon.com/efs/latest/ug/efs-onpremises.html#wt5-step4-install-nfs) in the Amazon EFS documentation.
**Mounting the Amazon EFS file system**
Use the commands:
```
mkdir ~/
$ sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport mount-target-IP:/ ~/
```
For more information, see the [walkthrough](https://docs.aws.amazon.com/efs/latest/ug/efs-onpremises.html#wt5-step3-connect) and [Mounting EFS File Systems](https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html) in the Amazon EFS documentation.
**Making backups of the MySQL source database**
*Full backups*
Use a command like the following, which takes the backup, zips it, and splits it into smaller chunks of 1 GB each:
```
xtrabackup --backup --user=dbuser --password= --binlog-info=AUTO --stream=tar --target-dir=~//fullbackup | gzip - | split -d --bytes=1024MB - ~//fullbackup/backup.tar.gz &
```
If you're planning to make subsequent incremental backups after the full backup, do not zip and split the backup. Instead, use a command similar to the following:
```
xtrabackup --backup --user=dbuser --password= --target-dir=~//fullbackup/
```
*Incremental backups*
Use the full backup path for the `--incremental-basedir` parameter; for example:
```
xtrabackup --backup --user=dbuser --password= --target-dir=~//incremental/backupdate --incremental-basedir=~//fullbackup
```
where *basedir *is the path to the full backup and the xtrabackup\$1checkpoints file.
For more information about making backups, see [Migrating Data from an External MySQL Database to an Amazon Aurora MySQL DB Cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.ExtMySQL.html) in the Aurora documentation.
**Preparing backups**
To prepare a full backup:
```
xtrabackup --prepare --apply-log-only --target-dir=~//fullbackup
```
To prepare an incremental backup:
```
xtrabackup --prepare --apply-log-only --target-dir=~//fullbackup --incremental-dir=~//incremental/06062020
```
To prepare the final backup:
```
xtrabackup --prepare --target-dir=~//fullbackup --incremental-dir=~//incremental/06072020
```
For more information, see [Incremental backups](https://www.percona.com/doc/percona-xtrabackup/2.4/backup_scenarios/incremental_backup.html) in the Percona XtraBackup documentation.
**Zipping and splitting the merged backup**
To zip the merged backup at \$1//fullbackup:
```
tar -zcvf ~//fullbackup
```
To split the backup:
```
split -d -b1024M --verbose
```
**Setting up binlog replication**
To create a replication user on the source database and provide the appropriate privileges:
```
CREATE USER 'repl_user'@'' IDENTIFIED BY ''; GRANT REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'repl_user'@'';
```
To enable replication on Aurora by connecting to the Aurora DB cluster, enable binary logs in the DB cluster parameter group. Set `binlog_format = mixed` (mixed mode is preferred). This change requires that you restart the instance to apply the update.
```
CALL mysql.rds_set_external_master ('sourcedbinstanceIP', sourcedbport, 'repl_user', '', 'binlog_file_name', binlog_file_position, 0); CALL mysql.rds_start_replication;
```
To confirm that the replication is in sync:
```
SHOW Slave Status \G;
```
The **Seconds behind master** field shows how far behind Aurora is from the on-premises database.
# Migrate on-premises Java applications to AWS using AWS App2Container
*Dhananjay Karanjkar, Amazon Web Services*
## Summary
AWS App2Container (A2C) is a command line tool that helps transform existing applications running in virtual machines into containers, without needing any code changes. A2C discovers applications running on a server, identifies dependencies, and generates relevant artifacts for seamless deployment to Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS).
This pattern provides the steps for remotely migrating on-premises Java applications deployed on an application server to AWS Fargate or Amazon EKS by using App2Container through the worker machine.
The worker machine can be used in the following use cases:
+ Docker installation is not allowed or not available on the application servers where the Java applications are running.
+ You must manage the migration of multiple applications deployed on different physical or virtual servers.
This pattern uses AWS CodeCommit, AWS CodePipeline, and AWS CodeBuild.
## Prerequisites and limitations
**Prerequisites **
+ An application server with a Java application running on a Linux server
+ A worker machine with a Linux operating system
+ A worker machine with at least 20 GB of available disk space
**Limitations **
+ Not all applications are supported. For more information, see [Supported applications for Linux](https://docs.aws.amazon.com/app2container/latest/UserGuide/supported-applications.html).
## Architecture
**Source technology stack **
+ Java applications running on Linux server
**Target technology stack **
+ AWS CodeBuild
+ AWS CodeCommit
+ AWS CodeDeploy
+ AWS CodePipeline
+ Amazon Elastic Container Registry
+ AWS Fargate
**Target architecture **
![\[Architecture for on-premises Java applications on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/61ed65a0-fab2-4fc8-8531-18bfd56a25b3/images/602cde7b-ab0c-46a5-8c37-afe304adf061.png)
## Tools
**Tools**
+ [AWS App2Container](https://docs.aws.amazon.com/app2container/latest/UserGuide/what-is-a2c.html) – AWS App2Container (A2C) is a command line tool to help you lift and shift applications that run in your on-premises data centers or on virtual machines, so that they run in containers that are managed by Amazon ECS or Amazon EKS.
+ [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html) – AWS CodeBuild is a fully managed build service in the cloud. CodeBuild compiles your source code, runs unit tests, and produces artifacts that are ready to deploy.
+ [AWS CodeCommit](https://docs.aws.amazon.com/codecommit/latest/userguide/welcome.html) – AWS CodeCommit is a version control service hosted by Amazon Web Services that you can use to privately store and manage assets (such as documents, source code, and binary files) in the cloud.
+ [AWS CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html) – AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software.
+ [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) – Amazon Elastic Container Service (Amazon ECS) is a highly scalable, fast container management service that for running, stopping, and managing containers on a cluster.
+ [Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) – Amazon Elastic Container Registry (Amazon ECR) is an AWS managed container image registry service that is secure, scalable, and reliable.
+ [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) – Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes.
+ [AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html) – AWS Fargate is a technology that you can use with Amazon ECS to run containers without having to manage servers or clusters of Amazon Elastic Compute Cloud (Amazon EC2) instances. With Fargate, you no longer have to provision, configure, or scale clusters of virtual machines to run containers.
## Epics
### Set up credentials
| Task | Description | Skills required |
| --- | --- | --- |
| Create a secret to access the application server. | To access the application server remotely from the worker machine, create a secret in AWS Secrets Manager. For your secret, you can use either the SSH private key or the Certificate and the SSH private key. For more information, see [Manage secrets for AWS App2Container](https://docs.aws.amazon.com/app2container/latest/UserGuide/manage-secrets.html). | DevOps, Developer |
### Set up the worker machine
| Task | Description | Skills required |
| --- | --- | --- |
| Install the tar file. | Run `sudo yum install -y tar`. | DevOps, Developer |
| Install the AWS CLI. | To install the Amazon Command Line Interface (AWS CLI), run `curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"`. Unzip `awscliv2.zip`.Run `sudo ./aws/install`. | DevOps, Developer |
| Install App2Container. | Run the following commands:`curl -o AWSApp2Container-installer-linux.tar.gz https://app2container-release-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/AWSApp2Container-installer-linux.tar.gz``sudo tar xvf AWSApp2Container-installer-linux.tar.gz``sudo ./install.sh` | DevOps, Developer |
| Configure the profiles. | To configure the AWS default profile, run `sudo aws configure`.To configure the named AWS default profile, run `sudo aws configure --profile `. | DevOps, Developer |
| Install Docker. | Run the following commands.`sudo yum install -y docker``sudo systemctl enable docker & sudo systemctl restart docker` | |
| Initialize App2Container. | To initialize App2Container, you need the following information:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-on-premises-java-applications-to-aws-using-aws-app2container.html)Run `sudo app2container init`. | DevOps, Developer |
### Configure the worker machine
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the worker machine to remotely connect and run App2Container commands on the application server. | To configure the worker machine, the following information is required:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-on-premises-java-applications-to-aws-using-aws-app2container.html)Run `sudo app2container remote configure`. | DevOps, Developer |
### Discover, analyze, and extract applications on the worker machine
| Task | Description | Skills required |
| --- | --- | --- |
| Discover the on-premises Java applications. | To remotely discover all the applications running on the application server, run the following command.`sudo app2container remote inventory --target `This command generates a list of deployed applications in `inventory.json`. | Developer, DevOps |
| Analyze the discovered applications. | To remotely analyze each application by using the application-id obtained in the inventory stage, run the following command.`sudo app2container remote analyze --application-id --target `This generates `analysis.json` file in the workspace location. After this file is generated, you can alter the containerization parameters based on your needs. | Developer, DevOps |
| Extract the analyzed applications. | To generate an application archive for the analyzed application, remotely run the following command, which will generate the tar bundle in the workspace location.`sudo app2container remote extract --application-id --target `Extracted artifacts can be generated on the local worker machine. | Developer, DevOps |
### Containerize the extracted artifacts on the worker machine
| Task | Description | Skills required |
| --- | --- | --- |
| Containerize the extracted artifacts. | Containerize the artifacts extracted in the previous step by running the following command.`sudo app2container containerize --input-archive ` | Developer, DevOps |
| Finalize the target. | To finalize the target, open `deployment.json`, which is created when the `containerize` command runs. To specify AWS Fargate as the target, set `createEcsArtifacts` to `true`. To set Amazon EKS as the target, set `createEksArtifacts` to true. | Developer, DevOps |
### Generate and provision AWS artifacts
| Task | Description | Skills required |
| --- | --- | --- |
| Generate AWS deployment artifacts on the worker machine. | To generate deployment artifacts, run the following command.`sudo app2container generate app-deployment --application-id `This generates the `ecs-master.yml` AWS CloudFormation template in the workspace. | DevOps |
| Provision the artifacts. | To further provision the generated artifacts, deploy the AWS CloudFormation template by running the following command.`aws cloudformation deploy --template-file --capabilities CAPABILITY_NAMED_IAM --stack-name –ECS` | DevOps |
| Generate the pipeline. | Modify `pipeline.json`, which was created in the previous story, based on your needs. Then run the `generate pipeline` command to generate the pipeline deployment artifacts. | DevOps |
## Related resources
+ [What is App2Container?](https://docs.aws.amazon.com/app2container/latest/UserGuide/what-is-a2c.html)
+ [AWS App2Container blog post](https://aws.amazon.com/blogs/aws/aws-app2container-a-new-containerizing-tool-for-java-and-asp-net-applications/)
+ [AWS CLI configuration basics](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)
+ [Docker basics for Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html)
+ [Docker commands](https://docs.docker.com/engine/reference/commandline/cli/)
# Migrate shared file systems in an AWS large migration
*Amit Rudraraju, Sam Apa, Bheemeswararao Balla, Wally Lu, and Sanjeev Prakasam, Amazon Web Services*
## Summary
Migrating 300 or more servers is considered a *large migration*. The purpose of a large migration is to migrate workloads from their existing, on-premises data centers to the AWS Cloud, and these projects typically focus on application and database workloads. However, shared file systems require focused attention and a separate migration plan. This pattern describes the migration process for shared file systems and provides best practices for migrating them successfully as part of a large migration project.
A *shared file system* (SFS), also known as a *network *or *clustered *file system, is a file share that is mounted to multiple servers. Shared file systems are accessed through protocols such as Network File System (NFS), Common Internet File System (CIFS), or Server Message Block (SMB).
These systems are not migrated with standard migration tools such as AWS Application Migration Service because they are neither dedicated to the host being migrated nor represented as a block device. Although most host dependencies are migrated transparently, the coordination and management of the dependent file systems must be handled separately.
You migrate shared file systems in the following phases: discover, plan, prepare, cut over, and validate. Using this pattern and the attached workbooks, you migrate your shared file system to an AWS storage service, such as Amazon Elastic File System (Amazon EFS), Amazon FSx for NetApp ONTAP, or Amazon FSx for Windows File Server. To transfer the file system, you can use AWS DataSync or a third-party tool, such as NetApp SnapMirror.
**Note**
This pattern is part of an AWS Prescriptive Guidance series about [large migrations to the AWS Cloud](https://aws.amazon.com/prescriptive-guidance/large-migrations/). This pattern includes best practices and instructions for incorporating SFSs into your wave plans for servers. If you are migrating one or more shared file systems outside of a large migration project, see the data transfer instructions in the AWS documentation for [Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/trnsfr-data-using-datasync.html), [Amazon FSx for Windows File Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/migrate-to-fsx.html), and [Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/migrating-fsx-ontap.html).
## Prerequisites and limitations
**Prerequisites**
Prerequisites can vary depending on your source and target shared file systems and your use case. The following are the most common:
+ An active AWS account.
+ You have completed application portfolio discovery for your large migration project and started developing wave plans. For more information, see [Portfolio playbook for AWS large migrations](https://docs.aws.amazon.com/prescriptive-guidance/latest/large-migration-portfolio-playbook/welcome.html).
+ Virtual private clouds (VPCs) and security groups that allow ingress and egress traffic between the on-premises data center and your AWS environment. For more information, see [Network-to Amazon VPC connectivity options](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/network-to-amazon-vpc-connectivity-options.html) and [AWS DataSync network requirements](https://docs.aws.amazon.com/datasync/latest/userguide/datasync-network.html).
+ Permissions to create AWS CloudFormation stacks or permissions to create Amazon EFS or Amazon FSx resources. For more information, see the [CloudFormation documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-template.html), [Amazon EFS documentation](https://docs.aws.amazon.com/efs/latest/ug/security-iam.html), or [Amazon FSx documentation](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/security-iam.html).
+ If you’re using AWS DataSync to perform the migration, you need the following permissions:
+ Permissions for AWS DataSync to send logs to an Amazon CloudWatch Logs log group. For more information, see [Allowing DataSync to upload logs to CloudWatch log groups](https://docs.aws.amazon.com/datasync/latest/userguide/monitor-datasync.html#cloudwatchlogs).
+ Permissions to access the CloudWatch Logs log group. For more information, see [Overview of managing access permissions to your CloudWatch Logs resources](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-access-control-overview-cwl.html).
+ Permissions to create agents and tasks in DataSync. For more information, see [Required IAM permissions for using AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/permissions-requirements.html).
**Limitations**
+ This pattern is designed to migrate SFSs as part of a large migration project. It includes best practices and instructions for incorporating SFSs into your wave plans for migrating applications. If you are migrating one or more shared file systems outside of a large migration project, see the data transfer instructions in the AWS documentation for [Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/trnsfr-data-using-datasync.html), [Amazon FSx for Windows File Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/migrate-to-fsx.html), and [Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/migrating-fsx-ontap.html).
+ This pattern is based on commonly used architectures, services, and migration patterns. However, large migration projects and strategies can vary between organizations. You might need to customize this solution or the provided workbooks based on your requirements.
## Architecture
**Source technology stack**
One or more of the following:
+ Linux (NFS) file server
+ Windows (SMB) file server
+ NetApp storage array
+ Dell EMC Isilon storage array
**Target technology stack**
One or more of the following:
+ Amazon Elastic File System
+ Amazon FSx for NetApp ONTAP
+ Amazon FSx for Windows File Server
**Target architecture**
![\[Architecture diagram of using AWS DataSync to migrate on-premises shared file systems to AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a30cf791-7a8a-4f71-8927-bc61f3b332f2/images/13232433-7d33-44c8-8998-b720f33f67b3.png)
The diagram shows the following process:
1. You establish a connection between the on-premises data center and the AWS Cloud by using an AWS service such as AWS Direct Connect or AWS Site-to-Site VPN.
1. You install the DataSync agent in the on-premises data center.
1. According to your wave plan, you use DataSync to replicate data from the source shared file system to the target AWS file share.
**Migration phases**
The following image shows the phases and high-level steps for migrating an SFS in a large migration project.
![\[Discover, plan, prepare, cut over, and validate phases of migrating shared file systems to AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a30cf791-7a8a-4f71-8927-bc61f3b332f2/images/f1e0c94d-0eea-46a8-bdec-3297b34c1d43.png)
The [Epics](#migrate-shared-file-systems-in-an-aws-large-migration-epics) section of this pattern contains detailed instructions for how to complete the migration and use the attached workbooks. The following is a high-level overview of the steps in this phased approach.
|
|
| Phase | Steps |
| --- |--- |
| Discover | 1. Using a discovery tool, you collect data about the shared file system, including servers, mount points, and IP addresses.2. Using a configuration management database (CMDB) or your migration tool, you collect details about the server, including information about the migration wave, environment, application owner, IT service management (ITSM) service name, organizational unit, and application ID. |
| Plan | 3. Using the collected information about the SFSs and the servers, create the SFS wave plan.4. Using the information in the build worksheet, for each SFS, choose a target AWS service and a migration tool. |
| Prepare | 5. Set up the target infrastructure in Amazon EFS, Amazon FSx for NetApp ONTAP, or Amazon FSx for Windows File Server.6. Set up the data transfer service, such as DataSync, and then start the initial data sync. When the initial sync is complete, you can set up reoccurring syncs to run on a schedule,7. Update the SFS wave plan with information about the target file share, such as the IP address or path. |
| Cut over | 8. Stop applications that actively access the source SFS.9. In the data transfer service, perform a final data sync.10. When the sync is complete, validate that it was completely successfully by reviewing the log data in CloudWatch Logs. |
| Validate | 11. On the servers, change the mount point to the new SFS path.12. Restart and validate the applications. |
## Tools
**AWS services**
+ [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) helps you centralize the logs from all your systems, applications, and AWS services so you can monitor them and archive them securely.
+ [AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html) is an online data transfer and discovery service that helps you move files or object data to, from, and between AWS storage services.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud.
+ [Amazon FSx](https://docs.aws.amazon.com/fsx/?id=docs_gateway) provides file systems that support industry-standard connectivity protocols and offer high availability and replication across AWS Regions.
**Other tools**
+ [SnapMirror](https://library.netapp.com/ecmdocs/ECMP1196991/html/GUID-BA1081BE-B2BB-4C6E-8A82-FB0F87AC514E.html) is a NetApp data replication tool that replicates data from specified source volumes or [qtrees](https://library.netapp.com/ecmdocs/ECMP1154894/html/GUID-8F084F85-2AB8-4622-B4F3-2D9E68559292.html) to target volumes or qtrees, respectively. You can use this tool to migrate a NetApp source file system to Amazon FSx for NetApp ONTAP.
+ [Robocopy](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy), which is short for *Robust File Copy*, is a command-line directory and command for Windows. You can use this tool to migrate a Windows source file system to Amazon FSx for Windows File Server.
## Best practices
**Wave planning approaches**
When planning waves for your large migration project, consider latency and application performance. When the SFS and dependent applications are operating in different locations, such as one in the cloud and one in the on-premises data center, this can increase latency and affect application performance. The following are the available options when creating wave plans:
1. **Migrate the SFS and all dependent servers within the same wave** – This approach prevents performance issues and minimizes rework, such as reconfiguring mount points multiple times. It is recommended when very low latency is required between the application and the SFS. However, wave planning is complex, and the goal is typically to remove variables from dependency groupings, not add to them. In addition, this approach isn’t recommended if many servers access the same SFS because it makes the wave too large.
1. **Migrate the SFS after the last dependent server has been migrated **– For example, if an SFS is accessed by multiple servers and those servers are scheduled to migrate in waves 4, 6, and 7, schedule the SFS to migrate in wave 7.
This approach is often the most logical for large migrations and is recommended for latency-sensitive applications. It reduces costs associated with data transfer. It also minimizes the period of latency between the SFS and higher-tier (such as production) applications because higher-tier applications are typically scheduled to migrate last, after development and QA applications.
However, this approach still requires discovery, planning, and agility. You might need to migrate the SFS in an earlier wave. Confirm that the applications can withstand the additional latency for the period of time between the first dependent wave and the wave containing the SFS. Conduct a discovery session with the application owners and migrate the application in same wave the most latency-sensitive application. If performance issues are discovered after migrating a dependent application, be prepared to pivot quickly to migrate the SFS as quickly as possible.
1. **Migrate the SFS at the end of the large migration project **– This approach is recommended if latency is not a factor, such as when the data in the SFS is infrequently accessed or not critical to application performance. This approach streamlines the migration and simplifies cutover tasks.
You can blend these approaches based on the latency-sensitivity of the application. For example, you can migrate latency-sensitive SFSs by using approaches 1 or 2 and then migrate the rest of the SFSs by using approach 3.
**Choosing an AWS file system service**
AWS offers several cloud services for file storage. Each offers different benefits and limitations for performance, scale, accessibility, integration, compliance, and cost optimization. There are some logical default options. For example, if your current on-premises file system is operating Windows Server, then Amazon FSx for Windows File Server is the default choice. Or if the on-premises file system is operating NetApp ONTAP, then Amazon FSx for NetApp ONTAP is the default choice. However, you might choose a target service based on the requirements of your application or to realize other cloud operating benefits. For more information, see [Choosing the right AWS file storage service for your deployment](https://d1.awsstatic.com/events/Summits/awsnycsummit/Choosing_the_right_AWS_file_storage_service_for_your_deployment_STG302.pdf) (AWS Summit presentation).
**Choosing a migration tool**
Amazon EFS and Amazon FSx support use of AWS DataSync to migrate shared file systems to the AWS Cloud. For more information about supported storage systems and services, benefits, and use cases, see [What is AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html). For an overview of the process of using DataSync to transfer your files, see [How AWS DataSync transfers work](https://docs.aws.amazon.com/datasync/latest/userguide/how-datasync-transfer-works.html).
There are also several third-party tools that are available, including the following:
+ If you choose Amazon FSx for NetApp ONTAP, you can use NetApp SnapMirror to migrate the files from the on-premises data center to the cloud. SnapMirror uses block-level replication, which can be faster than DataSync and reduce the duration of the data transfer process. For more information, see [Migrating to FSx for ONTAP using NetApp SnapMirror](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/migrating-fsx-ontap-snapmirror.html).
+ If you choose Amazon FSx for Windows File Server, you can use Robocopy to migrate files to the cloud. For more information, see [Migrating existing files to FSx for Windows File Server using Robocopy](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/migrate-files-to-fsx.html).
## Epics
### Discover
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the SFS discovery workbook. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead |
| Collect information about the source SFS. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead |
| Collect information about the servers. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead |
### Plan
| Task | Description | Skills required |
| --- | --- | --- |
| Build the SFS wave plan. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Build lead, Cutover lead, Migration engineer, Migration lead |
| Choose the target AWS service and migration tool. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead |
### Prepare
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the target file system. | According to the details recorded in your wave plan, set up the target file systems in the target AWS account, VPC, and subnets. For instructions, see the following AWS documentation:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead, AWS administrator |
| Set up the migration tool and transfer data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | AWS administrator, Cloud administrator, Migration engineer, Migration lead |
| Update the wave plan. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Stop applications. | If applications or clients are actively performing read and write operations in the source SFS, stop them before you perform the final data sync. For instructions, see the application documentation or your internal processes for stopping read and write activities. For example, see [Start or Stop the Web Server (IIS 8)](https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/jj635851(v=ws.11)) (Microsoft documentation) or [Managing system services with systemctl](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/managing-systemd_configuring-basic-system-settings#managing-system-services-with-systemctl_managing-systemd) (Red Hat documentation). | App owner, App developer |
| Perform the final data transfer. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | Migration engineer, Migration lead |
| Validate the data transfer. | If you’re using AWS DataSync, do the following to validate the final data transfer completed successfully:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html)If you’re using a third-party tool, see the data transfer validation instructions in the documentation for the selected migration tool. | Migration engineer, Migration lead |
### Validate
| Task | Description | Skills required |
| --- | --- | --- |
| Remount the file system and validate application function and performance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-shared-file-systems-in-an-aws-large-migration.html) | AWS systems administrator, App owner |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Cell values in Microsoft Excel don’t update. | Copy the formulas in the sample rows by dragging the fill handle. For more information, see instructions for [Windows](https://support.microsoft.com/en-us/office/fill-a-formula-down-into-adjacent-cells-041edfe2-05bc-40e6-b933-ef48c3f308c6) or for [Mac](https://support.microsoft.com/en-au/office/copy-a-formula-by-dragging-the-fill-handle-in-excel-for-mac-dd928259-622b-473f-9a33-83aa1a63e218) (Microsoft Support website). |
## Related resources
**AWS documentation**
+ [AWS DataSync documentation](https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html)
+ [Amazon EFS documentation](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html)
+ [Amazon FSx documentation](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/index.html)
+ [Large migrations to the AWS Cloud](https://aws.amazon.com/prescriptive-guidance/large-migrations/)
+ [Guide for AWS large migrations](https://docs.aws.amazon.com/prescriptive-guidance/latest/large-migration-guide/welcome.html)
+ [Portfolio playbook for AWS large migrations](https://docs.aws.amazon.com/prescriptive-guidance/latest/large-migration-portfolio-playbook/welcome.html)
**Troubleshooting**
+ [Troubleshooting AWS DataSync issues](https://docs.aws.amazon.com/datasync/latest/userguide/troubleshooting-datasync.html)
+ [Troubleshooting Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/troubleshooting.html)
+ [Troubleshooting Amazon FSx for Windows File Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/troubleshooting.html)
+ [Troubleshooting Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/troubleshooting.html)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/a30cf791-7a8a-4f71-8927-bc61f3b332f2/attachments/attachment.zip)
# Migrate an Oracle database to Amazon RDS for Oracle by using Oracle GoldenGate flat file adapters
*Dhairya Jindani and Baji Shaik, Amazon Web Services*
## Summary
Oracle GoldenGate is a real-time data capture and replication service for heterogeneous databases and IT environments. However, this service doesn’t currently support Amazon Relational Database Service (Amazon RDS) for Oracle. For a list of supported databases, see [Oracle GoldenGate for Heterogeneous Databases](https://docs.oracle.com/goldengate/c1230/gg-winux/GGHDB/12.3-what-is-oracle-goldengate-heterogeneous-databases.htm#GGHDB-GUID-08EAC588-F76C-4E37-BEBA-0DC57B98CA46) (Oracle documentation). This pattern describes how to use Oracle GoldenGate and Oracle GoldenGate flat file adapters to generate flat files from the source Oracle database, which can be on-premises or on an Amazon Elastic Compute Cloud (Amazon EC2) instance. You can then import those flat files to an Amazon RDS for Oracle database instance.
In this pattern, you use Oracle GoldenGate to extract the trail files from your source Oracle database. The data pump copies the trail files to an integration server, which is an Amazon EC2 instance. On the integration server, Oracle GoldenGate uses the flat file adapter to generate a series of sequential flat files based on the transational data capture of the trail files.Oracle GoldenGate formats the data as either delimiter-separated values or length-delimited values. You then use Oracle SQL\$1Loader to import the flat files into the target Amazon RDS for Oracle database instance.
**Intended audience**
This pattern is intended for those who have experience with and knowledge of an Oracle GoldenGate's fundamental building blocks. For more information, see [Overview of the Oracle GoldenGate Architecture](https://docs.oracle.com/goldengate/1212/gg-winux/GWUAD/wu_about_gg.htm#GWUAD115) (Oracle documentation).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An Oracle GoldenGate license.
+ A separate license for an Oracle GoldenGate adapter.
+ A source Oracle database, either running on-premises or on an Amazon EC2 instance.
+ An Amazon EC2 Linux instance that is used as the integration server. For more information, see [Get started with Amazon EC2 Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) (Amazon EC2 documentation).
+ A target Amazon RDS for Oracle database instance. For more information, see [Creating an Oracle DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.Oracle.html) (Amazon RDS documentation).
**Product versions**
+ Oracle Database Enterprise Edition version 10g, 11g, 12c, or later
+ Oracle GoldenGate version 12.2.0.1.1 or later
## Architecture
**Source technology stack**
An Oracle database (on premises or on an Amazon EC2 instance)
**Target technology stack**
Amazon RDS for Oracle
**Source and target architecture**
![\[Migrating an Oracle database to Amazon RDS for Oracle by using an Oracle GoldenGate adapter.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f34961f7-aa9a-41cb-b1ea-522e36ef2f67/images/21ef5177-e669-4591-aced-28d2f22decf2.png)
1. Oracle GoldenGate extracts trails from the source database logs.
1. The data pump extracts the trails and migrates them to an integration server.
1. The Oracle GoldenGate flat file adapter reads the trails, source definitions, and extract parameters.
1. You exit the extraction, which generates a control file and flat data files.
1. You migrate the flat data files to an Amazon RDS for Oracle database instance in the AWS Cloud.
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) for Oracle helps you set up, operate, and scale an Oracle relational database in the AWS Cloud.
**Other services**
+ [Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GWUAD/wu_about_gg.htm#GWUAD110) is a service that helps you to replicate, filter, and transform data from one database to another heterogeneous database or to another target topology, such as flat files.
+ [Oracle GoldenGate application adapters](https://docs.oracle.com/goldengate/gg121211/gg-adapter/GADAD/flatfile_config.htm#GADAD424) enable Oracle GoldenGate to produce a series of sequential flat files and control files from transactional data captured in the trail files of a source database. These adapters are widely used for extract, transform, and load (ETL) operations in data warehouse applications and proprietary or legacy applications. Oracle GoldenGate performs this capture and applies it in near real-time across heterogeneous databases, platforms, and operating systems. The adapters support different formats for the output files, such as CSV or Apache Parquet. You can load these generated files in order to load the data into different heterogeneous databases.
## Epics
### Set up Oracle GoldenGate on the source database server
| Task | Description | Skills required |
| --- | --- | --- |
| Download Oracle GoldenGate. | On the source database server, download Oracle GoldenGate version 12.2.0.1.1 or later. For instructions, see [Downloading Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/install.htm#GIORA164) (Oracle documentation). | DBA |
| Install Oracle GoldenGate. | For instructions, see [Installing Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/install.htm#GIORA162) (Oracle documentation). | DBA |
| Set up Oracle GoldenGate. | For instructions, see [Preparing the Database for Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/setup.htm#GIORA357) (Oracle documentation). | DBA |
### Set up Oracle GoldenGate on the integration server
| Task | Description | Skills required |
| --- | --- | --- |
| Download Oracle GoldenGate. | On the integration server, download Oracle GoldenGate version 12.2.0.1.1 or later. For instructions, see [Downloading Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/install.htm#GIORA164) (Oracle documentation). | DBA |
| Install Oracle GoldenGate. | Create directories, set up the manager process, and create the `defgen` file for a heterogeneous environment. For instructions, see [Installing Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/install.htm#GIORA162) (Oracle documentation). | DBA |
### Change the Oracle GoldenGate data capture configuration
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare the Oracle GoldenGate adapters. | On the integration server, set up the Oracle GoldenGate adapter software. Do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-database-to-amazon-rds-for-oracle-by-using-oracle-goldengate-flat-file-adapters.html) | DBA |
| Configure the data pump. | On the source server, configure the data pump to transfer the trail file from the source server to the integration server. Create the data pump parameter file and trails file directory. For instructions, see [Configuring the Flat File Adapter](https://docs.oracle.com/goldengate/gg12201/gg-adapter/GADAD/GUID-DF13488D-E0E9-497C-8AFF-70B839DE4843.htm#GADAD424) (Oracle documentation). | DBA |
### Generate and migrate the flat files
| Task | Description | Skills required |
| --- | --- | --- |
| Generate the flat files. | Create the extract file and control file, and then start the extraction process on the integration server. This extracts the database changes and writes the source database to the flat files. For instructions, see [Using the Flat File Adapter](https://docs.oracle.com/goldengate/gg12201/gg-adapter/GADAD/GUID-D30CC70D-B90F-4209-BEB5-9BA53EA869EF.htm#GADAD432) (Oracle documentation). | DBA |
| Load the flat files to the target database. | Load the flat files into the target Amazon RDS for Oracle database instance. For more information, see [Importing using Oracle SQL\$1Loader](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.SQLLoader.html) (Amazon RDS documentation). | DBA |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The Oracle GoldenGate flat file adapter generates an error. | For a description of the adapter errors, see [Locating Error Messages](https://docs.oracle.com/goldengate/gg12201/gg-adapter/GADAD/GUID-D30CC70D-B90F-4209-BEB5-9BA53EA869EF.htm#GADAD437) (Oracle documentation). For troubleshooting instructions, see [Troubleshooting the Flat File Adapter](https://docs.oracle.com/goldengate/gg12201/gg-adapter/GADAD/GUID-CB3D9B2C-49CC-408A-8C00-06E0C7923DD6.htm#GADAD552) (Oracle documentation). |
## Related resources
+ [Installing Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/install.htm#GIORA162) (Oracle documentation)
+ [Configuring Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/setup.htm#GIORA357) (Oracle documentation)
+ [Understanding Oracle GoldenGate Adapters](https://docs.oracle.com/goldengate/gg12201/gg-adapter/GADAD/GUID-F9105B02-9836-4F98-99F8-6E9C46D42764.htm#GADAD101) (Oracle documentation)
+ [Configuring the Flat File Adapter](https://docs.oracle.com/goldengate/gg12201/gg-adapter/GADAD/GUID-DF13488D-E0E9-497C-8AFF-70B839DE4843.htm#GADAD424) (Oracle documentation)
# Change Python and Perl applications to support database migration from Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible Edition
*Dwarika Patra and Deepesh Jayaprakash, Amazon Web Services*
## Summary
This pattern describes changes to application repositories that might be required when you migrate databases from Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible Edition. The pattern assumes that these applications are Python-based or Perl-based, and provides separate instructions for these scripting languages.
Migrating SQL Server databases to Aurora PostgreSQL-Compatible involves schema conversion, database object conversion, data migration, and data loading. Because of the differences between PostgreSQL and SQL Server (relating to data types, connection objects, syntax, and logic), the most difficult migration task involves making the necessary changes to the code base so that it works correctly with PostgreSQL.
For a Python-based application, connection objects and classes are scattered throughout the system. Also, the Python code base might use multiple libraries to connect to the database. If the database connection interface changes, the objects that run the application’s inline queries also require changes.
For a Perl-based application, changes involve connection objects, database connection drivers, static and dynamic inline SQL statements, and how the application handles complex dynamic DML queries and results sets.
When you migrate your application, you can also consider possible enhancements on AWS, such as replacing the FTP server with Amazon Simple Storage Service (Amazon S3) access.
The application migration process involves the following challenges:
+ Connection objects. If connection objects are scattered in the code with multiple libraries and function calls, you might have to find a generalized way to change them to support PostgreSQL.
+ Error or exception handling during record retrieval or updates. If you have conditional create, read, update, and delete (CRUD) operations on the database that return variables, results sets, or data frames, any errors or exceptions might result in application errors with cascading effects. These should be handled carefully with proper validations and save points. One such save point is to call large inline SQL queries or database objects inside `BEGIN...EXCEPTION...END` blocks.
+ Controlling transactions and their validation. These includes manual and automatic commits and rollbacks. The PostgreSQL driver for Perl requires you to always explicitly set the auto-commit attribute.
+ Handling dynamic SQL queries. This requires a strong understanding of the query logic and iterative testing to ensure that queries work as expected.
+ Performance. You should ensure that code changes don’t result in degraded application performance.
This pattern explains the conversion process in detail.
## Prerequisites and limitations
**Prerequisites**
+ Working knowledge of Python and Perl syntax.
+ Basic skills in SQL Server and PostgreSQL.
+ Understanding of your existing application architecture.
+ Access to your application code, SQL Server database, and PostgreSQL database.
+ Access to Windows or Linux (or other Unix) development environment with credentials for developing, testing, and validating application changes.
+ For a Python-based application, standard Python libraries that your application might require, such as **Pandas **to handle data frames, and **psycopg2** or **SQLAlchemy **for database connections.
+ For a Perl-based application, required Perl packages with dependent libraries or modules. The Comprehensive Perl Archive Network (CPAN) module can support most application requirements.
+ All required dependent customized libraries or modules.
+ Database credentials for read access to SQL Server and read/write access to Aurora.
+ PostgreSQL to validate and debug application changes with services and users.
+ Access to development tools during application migration such as Visual Studio Code, Sublime Text, or **pgAdmin**.
**Limitations**
+ Some Python or Perl versions, modules, libraries, and packages aren’t compatible with the cloud environment.
+ Some third-party libraries and frameworks used for SQL Server cannot be replaced to support PostgreSQL migration.
+ Performance variations might require changes to your application, to inline Transact-SQL (T-SQL) queries, database functions, and stored procedures.
+ PostgreSQL supports lowercase names for table names, column names, and other database objects.
+ Some data types, such as UUID columns, are stored in lowercase only. Python and Perl applications must handle such case differences.
+ Character encoding differences must be handled with the correct data type for the corresponding text columns in the PostgreSQL database.
**Product versions**
+ Python 3.6 or later (use the version that supports your operating system)
+ Perl 5.8.3 or later (use the version that supports your operating system)
+ Aurora PostgreSQL-Compatible Edition 4.2 or later (see [details](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Updates.20180305.html#AuroraPostgreSQL.Updates.20180305.42))
## Architecture
**Source technology stack**
+ Scripting (application programming) language: Python 2.7 or later, or Perl 5.8
+ Database: Microsoft SQL Server version 13
+ Operating system: Red Hat Enterprise Linux (RHEL) 7
**Target technology stack **
+ Scripting (application programming) language: Python 3.6 or later, or Perl 5.8 or later
+ Database: Aurora PostgreSQL-Compatible 4.2
+ Operating system: RHEL 7
**Migration architecture **
![\[Migrating a Perl or Python application with SQL Server to Aurora PostgreSQL-Compatible\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b64de64a-bd55-4db7-ba7b-0a2557862af1/images/b8fab3e2-ded5-4f58-86bf-3f645252e9fc.png)
## Tools
**AWS services and tools**
+ [Aurora PostgreSQL–Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, PostgreSQL-compatible, and ACID-compliant relational database engine that combines the speed and reliability of high-end commercial databases with the cost-effectiveness of open-source databases. Aurora PostgreSQL is a drop-in replacement for PostgreSQL and makes it easier and more cost-effective to set up, operate, and scale your new and existing PostgreSQL deployments.
+ [AWS Command Line Interface (AWS CLI) ](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)is an open-source tool that enables you to interact with AWS services by using commands in your command-line shell.
**Other tools**
+ [Python](https://www.python.org/) and PostgresSQL database connection libraries such as [psycopg2](https://pypi.org/project/psycopg2/) and [SQLAlchemy](https://www.sqlalchemy.org/)
+ [Perl](https://www.perl.org/) and its [DBI modules](https://metacpan.org/pod/DBD::Pg)
+ [PostgreSQL interactive terminal](https://www.postgresql.org/docs/13/app-psql.html) (psql)
## Epics
### Migrate your application repository to PostgreSQL – high-level steps
| Task | Description | Skills required |
| --- | --- | --- |
| Follow these code conversion steps to migrate your application to PostgreSQL. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html)The following epics provide detailed instructions for some of these conversion tasks for Python and Perl applications. | App developer |
| Use a checklist for each step of the migration. | Add the following to your checklist for each step of application migration, including the final step:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html) | App developer |
### Analyze and update your application – Python code base
| Task | Description | Skills required |
| --- | --- | --- |
| Analyze your existing Python code base. | Your analysis should include the following to facilitate the application migration process:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html) | App developer |
| Convert your database connections to support PostgreSQL. | Most Python applications use the **pyodbc **library to connect with SQL Server databases as follows.
import pyodbc .... try: conn_string = "Driver=ODBC Driver 17 for SQL Server;UID={};PWD={};Server={};Database={}".format (conn_user, conn_password, conn_server, conn_database) conn = pyodbc.connect(conn_string) cur = conn.cursor() result = cur.execute(query_string) for row in result: print (row) except Exception as e: print(str(e))
Convert the database connection to support PostgreSQL as follows.
import pyodbc import psycopg2 .... try: conn_string = ‘postgresql+psycopg2://’+ conn_user+’:’+conn_password+’@’+conn_server+’/’+conn_database conn = pyodbc.connect(conn_string, connect_args={‘options’:’-csearch_path=dbo’}) cur = conn.cursor() result = cur.execute(query_string) for row in result: print (row) except Exception as e: print(str(e))
| App developer |
| Change inline SQL queries to PostgreSQL. | Convert your inline SQL queries to a PostgreSQL-compatible format. For example, the following SQL Server query retrieves a string from a table.
dtype = "type1" stm = ‘"SELECT TOP 1 searchcode FROM TypesTable (NOLOCK) WHERE code="’ + "’" + str(dtype) + "’" # For Microsoft SQL Server Database Connection engine = create_engine(‘mssql+pyodbc:///?odbc_connect=%s’ % urllib.parse.quote_plus(conn_string), connect_args={‘connect_timeout’:login_timeout}) conn = engine_connect() rs = conn.execute(stm) for row in rs: print(row)
After conversion, the PostgreSQL-compatible inline SQL query looks like the following.
dtype = "type1" stm = ‘"SELECT searchcode FROM TypesTable WHERE code="’ + "’" + str(dtype) + "’ LIMIT 1" # For PostgreSQL Database Connection engine = create_engine(‘postgres+psycopg2://%s’ %conn_string, connect_args={‘connect_timeout’:login_timeout}) conn = engine.connect() rs = conn.execute(stm) for row in rs: print(row)
| App developer |
| Handle dynamic SQL queries. | Dynamic SQL can be present in one script or in multiple Python scripts. Earlier examples showed how to use Python’s string replace function to insert variables for constructing dynamic SQL queries. An alternate approach is to append the query string with variables wherever applicable. In the following example, the query string is constructed on the fly based on the values returned by a function.
query = ‘"SELECT id from equity e join issues i on e.permId=i.permId where e.id’" query += get_id_filter(ids) + " e.id is NOT NULL
These types of dynamic queries are very common during application migration. Follow these steps to handle dynamic queries:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html) | App developer |
| Handle results sets, variables, and data frames. | For Microsoft SQL Server, you use Python methods such as `fetchone()` or `fetchall()` to retrieve the results set from the database. You can also use `fetchmany(size)` and specify the number of records to return from the results set. To do this, you can use the **pyodbc **connection object as shown in the following example.**pyodbc (Microsoft SQL Server)**
import pyodbc server = 'tcp:myserver.database.windows.net' database = 'exampledb' username = 'exampleusername' password = 'examplepassword' conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password) cursor = conn.cursor() cursor.execute("SELECT * FROM ITEMS") row = cursor.fetchone() while row: print(row[0]) row = cursor.fetchone()
In Aurora, to perform similar tasks such as connecting to PostgreSQL and fetching results sets, you can use either **psycopg2 **or **SQLAlchemy**. These Python libraries provide the connection module and cursor object to traverse through the PostgreSQL database records, as shown in the following example.**psycopg2 (Aurora PostgreSQL-Compatible)**
| App developer |
| Test your application during and after migration. | Testing the migrated Python application is an ongoing process. Because the migration includes connection object changes (**psycopg2 **or **SQLAlchemy**), error handling, new features (data frames), inline SQL changes, bulk copy functionalities (`bcp` instead of `COPY`) and similar changes, it must be tested carefully during and after application migration. Check for:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html) | App developer |
### Analyze and update your application – Perl code base
| Task | Description | Skills required |
| --- | --- | --- |
| Analyze your existing Perl code base. | Your analysis should include the following to facilitate the application migration process. You should identify:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html) | App developer |
| Convert the connections from the Perl application and DBI module to support PostgreSQL. | Perl-based applications generally use the Perl DBI module, which is a standard database access module for the Perl programming language. You can use the same DBI module with different drivers for SQL Server and PostgreSQL.For more information about required Perl modules, installations, and other instructions, see the [DBD::Pg documentation](https://metacpan.org/pod/DBD::Pg). The following example connects to Aurora PostgreSQL-Compatible at `exampletest-aurorapg-database.cluster-sampleclusture.us-east.-rds.amazonaws.com`.
#!/usr/bin/perl use DBI; use strict; my $driver = "Pg"; my $hostname = "exampletest-aurorapg-database-sampleclusture.us-east.rds.amazonaws.com" my $dsn = "DBI:$driver: dbname = $hostname;host = 127.0.0.1;port = 5432"; my $username = "postgres"; my $password = "pass123"; $dbh = DBI->connect("dbi:Pg:dbname=$hostname;host=$host;port=$port;options=$options", $username, $password, {AutoCommit => 0, RaiseError => 1, PrintError => 0} );
| App developer |
| Change Inline SQL queries to PostgreSQL. | Your application might have inline SQL queries with `SELECT`, `DELETE`, `UPDATE`, and similar statements that include query clauses that PostgreSQL doesn’t support. For example, query keywords such as `TOP` and `NOLOCK` aren’t supported in PostgreSQL. The following examples show how you can handle `TOP`, `NOLOCK`, and Boolean variables.In SQL Server:
$sqlStr = $sqlStr . "WHERE a.student_id in (SELECT TOP $numofRecords c_student_id \ FROM active_student_record b WITH (NOLOCK) \ INNER JOIN student_contributor c WITH (NOLOCK) on c.contributor_id = b.c_st)
For PostgreSQL, convert to:
$sqlStr = $sqlStr . "WHERE a.student_id in (SELECT TOP $numofRecords c_student_id \ FROM active_student_record b INNER JOIN student_contributor c \ on c.contributor_id = b.c_student_contr_id WHERE b_current_1 is true \ LIMIT $numofRecords)"
| App developer |
| Handle dynamic SQL queries and Perl variables. | Dynamic SQL queries are SQL statements that are built at application runtime. These queries are constructed dynamically when the application is running, depending on certain conditions, so the full text of the query isn’t known until runtime. An example is a financial analytics application that analyzes the top 10 shares on a daily basis, and these shares change every day. The SQL tables are created based on top performers, and the values aren’t known until runtime.Let’s say that the inline SQL queries for this example are passed to a wrapper function to get the results set in a variable, and then a variable uses a condition to determine whether the table exists:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html)Here’s an example of variable handling, followed by the SQL Server and PostgreSQL queries for this use case.
my $tableexists = db_read( arg 1, $sql_qry, undef, 'writer'); my $table_already_exists = $tableexists->[0]{table_exists}; if ($table_already_exists){ # do some thing } else { # do something else }
SQL Server:
my $sql_qry = "SELECT OBJECT_ID('$backendTable', 'U') table_exists", undef, 'writer')";
PostgreSQL:
my $sql_qry = "SELECT TO_REGCLASS('$backendTable', 'U') table_exists", undef, 'writer')";
The following example uses a** **Perl variable in inline SQL, which runs a `SELECT` statement with a `JOIN` to fetch the primary key of the table and position of the key column.SQL Server:
my $sql_qry = "SELECT column_name', character_maxi mum_length \ FROM INFORMATION_SCHEMA.COLUMNS \ WHERE TABLE_SCHEMA='$example_schemaInfo' \ AND TABLE_NAME='$example_table' \ AND DATA_TYPE IN ('varchar','nvarchar');";
PostgreSQL:
my $sql_qry = "SELECT c1.column_name, c1.ordinal_position \ FROM information_schema.key_column_usage AS c LEFT \ JOIN information_schema.table_constraints AS t1 \ ON t1.constraint_name = c1.constraint_name \ WHERE t1.table_name = $example_schemaInfo'.'$example_table’ \ AND t1.constraint_type = 'PRIMARY KEY' ;";
| App developer |
### Make additional changes to your Perl-based or Python-based application to support PostgreSQL
| Task | Description | Skills required |
| --- | --- | --- |
| Convert additional SQL Server constructs to PostgreSQL. | The following changes apply to all applications, regardless of programming language.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.html) | App developer |
### Improve performance
| Task | Description | Skills required |
| --- | --- | --- |
| Take advantage of AWS services to make performance enhancements. | When you migrate to the AWS Cloud, you can refine your application and database design to take advantage of AWS services. For example, if the queries from your Python application, which is connected to an Aurora PostgreSQL-Compatible database server, is taking more time than your original Microsoft SQL Server queries, you could consider creating a feed of historical data directly to an Amazon Simple Storage Service (Amazon S3) bucket from the Aurora server, and use Amazon Athena-based SQL queries to generate reports and analytic data queries for your user dashboards. | App developer, Cloud architect |
## Related resources
+ [Perl](https://www.perl.org/)
+ [Perl DBI Module](https://metacpan.org/pod/DBI)
+ [Python](https://www.python.org/)
+ [psycopg2](https://pypi.org/project/psycopg2/)
+ [SQLAlchemy](https://www.sqlalchemy.org/)
+ [Bulk Copy - PostgreSQL](https://www.postgresql.org/docs/9.2/sql-copy.html)
+ [Bulk Copy - Microsoft SQL Server](https://docs.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-ver15)
+ [PostgreSQL](https://www.postgresql.org/)
+ [Working with Amazon Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html)
## Additional information
Both Microsoft SQL Server and Aurora PostgreSQL-Compatible are ANSI SQL-complaint. However, you should still be aware of any incompatibilities in syntax, column data types, native database-specific functions, bulk inserts, and case sensitivity when you migrate your Python or Perl application from SQL Server to PostgreSQL.
The following sections provide more information about possible inconsistencies.
**Data type comparison**
Data type changes from SQL Server to PostgreSQL can lead to significant differences in the resulting data that applications operate on. For a comparison of data types, see the table on the [Sqlines website](https://www.sqlines.com/sql-server-to-postgresql).
**Native or built-in SQL functions**
The behavior of some functions differs between SQL Server and PostgreSQL databases. The following table provides a comparison.
|
|
| Microsoft SQL Server | Description | PostgreSQL |
| --- |--- |--- |
| `CAST` | Converts a value from one data type to another. | PostgreSQL `type :: operator` |
| `GETDATE()` | Returns the current database system date and time, in a `YYYY-MM-DD hh:mm:ss.mmm` format. | `CLOCK_TIMESTAMP` |
| `DATEADD` | Adds a time/date interval to a date. | `INTERVAL` expression |
| `CONVERT` | Converts a value to a specific data format. | `TO_CHAR` |
| `DATEDIFF` | Returns the difference between two dates. | `DATE_PART` |
| `TOP` | Limits the number of rows in a `SELECT` results set. | `LIMIT/FETCH` |
** Anonymous blocks**
A structured SQL query is organized into sections such as declaration, executables, and exception handling. The following table compares the Microsoft SQL Server and PostgreSQL versions of a simple anonymous block. For complex anonymous blocks, we recommend that you call a custom database function within your application.
|
|
| Microsoft SQL Server | PostgreSQL |
| --- |--- |
|
my $sql_qry1= my $sql_qry2 = my $sqlqry = "BEGIN TRAN $sql_qry1 $sql_qry2 if @\@error !=0 ROLLBACK TRAN else COMIT TRAN";
|
my $sql_qry1= my $sql_qry2 = my $sql_qry = " DO \$\$ BEGIN $header_sql $content_sql END \$\$";
|
**Other differences**
+ **Bulk inserts of rows:** The PostgreSQL equivalent of the [Microsoft SQL Server bcp utility](https://docs.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-ver15) is [COPY](https://www.postgresql.org/docs/9.2/sql-copy.html).
+ **Case sensitivity:** Column names are case-sensitive in PostgreSQL, so you have to convert your SQL Server column names to lowercase or uppercase. This becomes a factor when you extract or compare data, or place column names in results sets or variables. The following example identifies columns where values might be stored in uppercase or lowercase.
```
my $sql_qry = "SELECT $record_id FROM $exampleTable WHERE LOWER($record_name) = \'failed transaction\'";
```
+ **Concatenation: **SQL Server uses `+` as an operator for string concatenation, whereas PostgreSQL uses `||`.
+ **Validation: **You should test and validate inline SQL queries and functions before you use them in application code for PostgreSQL.
+ **ORM Library inclusion : **You can also look for including or replace existing database connection library with Python ORM libraries such as [SQLAlchemy](https://www.sqlalchemy.org/) and [PynomoDB](https://pynamodb.readthedocs.io/en/latest/quickstart.html). This will help to easily query and manipulate data from a database using an object-oriented paradigm.
# Migration patterns by workload
**Topics**
+ [IBM](migration-migration-patterns-by-workload-ibm-pattern-list.md)
+ [Microsoft](migration-migration-patterns-by-workload-microsoft-pattern-list.md)
+ [N/A](migration-migration-patterns-by-workload-notapplicable-pattern-list.md)
+ [Open source](migration-migration-patterns-by-workload-open-source-pattern-list.md)
+ [Oracle](migration-migration-patterns-by-workload-oracle-pattern-list.md)
+ [SAP](migration-migration-patterns-by-workload-sap-pattern-list.md)
# IBM
**Topics**
+ [Migrate a Db2 database from Amazon EC2 to Aurora MySQL-Compatible by using AWS DMS](migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.md)
+ [Migrate Db2 for LUW to Amazon EC2 by using log shipping to reduce outage time](migrate-db2-for-luw-to-amazon-ec2-by-using-log-shipping-to-reduce-outage-time.md)
+ [Migrate Db2 for LUW to Amazon EC2 with high availability disaster recovery](migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery.md)
+ [Migrate from IBM Db2 on Amazon EC2 to Aurora PostgreSQL-Compatible using AWS DMS and AWS SCT](migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.md)
+ [Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2](migrate-from-ibm-websphere-application-server-to-apache-tomcat-on-amazon-ec2.md)
# Microsoft
**Topics**
+ [Accelerate the discovery and migration of Microsoft workloads to AWS](accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.md)
+ [Change Python and Perl applications to support database migration from Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible Edition](change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.md)
+ [Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel and Python](create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel-and-python.md)
+ [Export a Microsoft SQL Server database to Amazon S3 by using AWS DMS](export-a-microsoft-sql-server-database-to-amazon-s3-by-using-aws-dms.md)
+ [Implement SHA1 hashing for PII data when migrating from SQL Server to PostgreSQL](implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.md)
+ [Ingest and migrate EC2 Windows instances into an AWS Managed Services account](ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.md)
+ [Migrate a messaging queue from Microsoft Azure Service Bus to Amazon SQS](migrate-a-messaging-queue-from-microsoft-azure-service-bus-to-amazon-sqs.md)
+ [Migrate a Microsoft SQL Server database from Amazon EC2 to Amazon DocumentDB by using AWS DMS](migrate-a-microsoft-sql-server-database-from-amazon-ec2-to-amazon-documentdb-by-using-aws-dms.md)
+ [Migrate a Microsoft SQL Server database to Aurora MySQL by using AWS DMS and AWS SCT](migrate-a-microsoft-sql-server-database-to-aurora-mysql-by-using-aws-dms-and-aws-sct.md)
+ [Migrate a .NET application from Microsoft Azure App Service to AWS Elastic Beanstalk](migrate-a-net-application-from-microsoft-azure-app-service-to-aws-elastic-beanstalk.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon EC2](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-ec2.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using linked servers](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-linked-servers.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using native backup and restore methods](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-native-backup-and-restore-methods.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS DMS](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-dms.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS SCT data extraction agents](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-sct-data-extraction-agents.md)
+ [Migrate an on-premises Microsoft SQL Server database to Microsoft SQL Server on Amazon EC2 running Linux](migrate-an-on-premises-microsoft-sql-server-database-to-microsoft-sql-server-on-amazon-ec2-running-linux.md)
+ [Migrate data from Microsoft Azure Blob to Amazon S3 by using Rclone](migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone.md)
+ [Migrate IIS-hosted applications to Amazon EC2 by using appcmd.exe](migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.md)
+ [Migrate an on-premises Microsoft SQL Server database to Amazon EC2 using Application Migration Service](migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.md)
+ [Migrate Windows SSL certificates to an Application Load Balancer using ACM](migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.md)
+ [Rehost on-premises workloads in the AWS Cloud: migration checklist](rehost-on-premises-workloads-in-the-aws-cloud-migration-checklist.md)
+ [Resolve connection errors after migrating Microsoft SQL Server to the AWS Cloud](resolve-connection-errors-after-migrating-microsoft-sql-server-to-the-aws-cloud.md)
+ [Set up Multi-AZ infrastructure for a SQL Server Always On FCI by using Amazon FSx](set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.md)
# N/A
**Topics**
+ [Create an approval process for firewall requests during a rehost migration to AWS](create-an-approval-process-for-firewall-requests-during-a-rehost-migration-to-aws.md)
# Open source
**Topics**
+ [Create application users and roles in Aurora PostgreSQL-Compatible](create-application-users-and-roles-in-aurora-postgresql-compatible.md)
+ [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL with AWS SCT and AWS DMS using AWS CLI and CloudFormation](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.md)
+ [Migrate an on-premises MariaDB database to Amazon RDS for MariaDB using native tools](migrate-an-on-premises-mariadb-database-to-amazon-rds-for-mariadb-using-native-tools.md)
+ [Migrate an on-premises MySQL database to Amazon RDS for MySQL](migrate-an-on-premises-mysql-database-to-amazon-rds-for-mysql.md)
+ [Migrate an on-premises MySQL database to Aurora MySQL](migrate-an-on-premises-mysql-database-to-aurora-mysql.md)
+ [Migrate an on-premises PostgreSQL database to Aurora PostgreSQL](migrate-an-on-premises-postgresql-database-to-aurora-postgresql.md)
+ [Migrate a Couchbase Server database to Amazon EC2](migrate-couchbase-server-ec2.md)
+ [Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2 with Auto Scaling](migrate-from-ibm-websphere-application-server-to-apache-tomcat-on-amazon-ec2-with-auto-scaling.md)
+ [Migrate from PostgreSQL on Amazon EC2 to Amazon RDS for PostgreSQL using pglogical](migrate-from-postgresql-on-amazon-ec2-to-amazon-rds-for-postgresql-using-pglogical.md)
+ [Migrate on-premises Java applications to AWS using AWS App2Container](migrate-on-premises-java-applications-to-aws-using-aws-app2container.md)
+ [Migrate on-premises MySQL databases to Aurora MySQL using Percona XtraBackup, Amazon EFS, and Amazon S3](migrate-on-premises-mysql-databases-to-aurora-mysql-using-percona-xtrabackup-amazon-efs-and-amazon-s3.md)
+ [Migrate Oracle external tables to Amazon Aurora PostgreSQL-Compatible](migrate-oracle-external-tables-to-amazon-aurora-postgresql-compatible.md)
+ [Restart the AWS Replication Agent automatically without disabling SELinux after rebooting a RHEL source server](restart-the-aws-replication-agent-automatically-without-disabling-selinux-after-rebooting-a-rhel-source-server.md)
+ [Transport PostgreSQL databases between two Amazon RDS DB instances using pg\$1transport](transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.md)
# Oracle
**Topics**
+ [Convert VARCHAR2(1) data type for Oracle to Boolean data type for Amazon Aurora PostgreSQL](convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.md)
+ [Emulate Oracle DR by using a PostgreSQL-compatible Aurora global database](emulate-oracle-dr-by-using-a-postgresql-compatible-aurora-global-database.md)
+ [Incrementally migrate from Amazon RDS for Oracle to Amazon RDS for PostgreSQL using Oracle SQL Developer and AWS SCT](incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.md)
+ [Load BLOB files into TEXT by using file encoding in Aurora PostgreSQL-Compatible](load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.md)
+ [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL in SSL mode by using AWS DMS](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.md)
+ [Migrate an Amazon RDS for Oracle database to another AWS account and AWS Region using AWS DMS for ongoing replication](migrate-an-amazon-rds-for-oracle-database-to-another-aws-account-and-aws-region-using-aws-dms-for-ongoing-replication.md)
+ [Migrate an on-premises Oracle database to Amazon EC2 by using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.md)
+ [Migrate an on-premises Oracle database to Amazon OpenSearch Service using Logstash](migrate-an-on-premises-oracle-database-to-amazon-opensearch-service-using-logstash.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-an-on-premises-oracle-database-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for Oracle](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-using-oracle-data-pump.md)
+ [Migrate an on-premises Oracle database to Amazon RDS for PostgreSQL by using an Oracle bystander and AWS DMS](migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.md)
+ [Migrate an on-premises Oracle database to Oracle on Amazon EC2](migrate-an-on-premises-oracle-database-to-oracle-on-amazon-ec2.md)
+ [Migrate an Oracle database from Amazon EC2 to Amazon RDS for MariaDB using AWS DMS and AWS SCT](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-mariadb-using-aws-dms-and-aws-sct.md)
+ [Migrate an Oracle database from Amazon EC2 to Amazon RDS for Oracle using AWS DMS](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-oracle-using-aws-dms.md)
+ [Migrate an Oracle database to Amazon RDS for Oracle by using Oracle GoldenGate flat file adapters](migrate-an-oracle-database-to-amazon-rds-for-oracle-by-using-oracle-goldengate-flat-file-adapters.md)
+ [Migrate an Oracle Database to Amazon Redshift using AWS DMS and AWS SCT](migrate-an-oracle-database-to-amazon-redshift-using-aws-dms-and-aws-sct.md)
+ [Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT](migrate-an-oracle-database-to-aurora-postgresql-using-aws-dms-and-aws-sct.md)
+ [Migrate an Oracle JD Edwards EnterpriseOne database to AWS by using Oracle Data Pump and AWS DMS](migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.md)
+ [Migrate an Oracle partitioned table to PostgreSQL by using AWS DMS](migrate-an-oracle-partitioned-table-to-postgresql-by-using-aws-dms.md)
+ [Migrate an Oracle PeopleSoft database to AWS by using AWS DMS](migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.md)
+ [Migrate data from an on-premises Oracle database to Aurora PostgreSQL](migrate-data-from-an-on-premises-oracle-database-to-aurora-postgresql.md)
+ [Migrate from Amazon RDS for Oracle to Amazon RDS for MySQL](migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-mysql.md)
+ [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using materialized views and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-materialized-views-and-aws-dms.md)
+ [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using SharePlex and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-shareplex-and-aws-dms.md)
+ [Migrate from Oracle Database to Amazon RDS for PostgreSQL by using Oracle GoldenGate](migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.md)
+ [Migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-from-oracle-on-amazon-ec2-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md)
+ [Migrate from Oracle WebLogic to Apache Tomcat (TomEE) on Amazon ECS](migrate-from-oracle-weblogic-to-apache-tomcat-tomee-on-amazon-ecs.md)
+ [Migrate function-based indexes from Oracle to PostgreSQL](migrate-function-based-indexes-from-oracle-to-postgresql.md)
+ [Migrate legacy applications from Oracle Pro\$1C to ECPG](migrate-legacy-applications-from-oracle-pro-c-to-ecpg.md)
+ [Migrate Oracle CLOB values to individual rows in PostgreSQL on AWS](migrate-oracle-clob-values-to-individual-rows-in-postgresql-on-aws.md)
+ [Migrate Oracle Database error codes to an Amazon Aurora PostgreSQL-Compatible database](migrate-oracle-database-error-codes-to-an-amazon-aurora-postgresql-compatible-database.md)
+ [Migrate Oracle native functions to PostgreSQL using extensions](migrate-oracle-native-functions-to-postgresql-using-extensions.md)
+ [Migrate Oracle PeopleSoft to Amazon RDS Custom](migrate-oracle-peoplesoft-to-amazon-rds-custom.md)
+ [Migrate Oracle ROWID functionality to PostgreSQL on AWS](migrate-oracle-rowid-functionality-to-postgresql-on-aws.md)
+ [Migrate Oracle SERIALLY\$1REUSABLE pragma packages into PostgreSQL](migrate-oracle-serially-reusable-pragma-packages-into-postgresql.md)
+ [Migrate virtual generated columns from Oracle to PostgreSQL](migrate-virtual-generated-columns-from-oracle-to-postgresql.md)
+ [Set up Oracle UTL\$1FILE functionality on Aurora PostgreSQL-Compatible](set-up-oracle-utl_file-functionality-on-aurora-postgresql-compatible.md)
+ [Validate database objects after migrating from Oracle to Amazon Aurora PostgreSQL](validate-database-objects-after-migrating-from-oracle-to-amazon-aurora-postgresql.md)
# SAP
**Topics**
+ [Migrate from SAP ASE to Amazon RDS for SQL Server using AWS DMS](migrate-from-sap-ase-to-amazon-rds-for-sql-server-using-aws-dms.md)
+ [Migrate SAP ASE on Amazon EC2 to Amazon Aurora PostgreSQL-Compatible using AWS SCT and AWS DMS](migrate-sap-ase-on-amazon-ec2-to-amazon-aurora-postgresql-compatible-using-aws-sct-and-aws-dms.md)
# More patterns
**Topics**
+ [Access AWS services from IBM z/OS by installing the AWS CLI](access-aws-services-from-ibm-z-os-by-installing-aws-cli.md)
+ [Assess query performance for migrating SQL Server databases to MongoDB Atlas on AWS](assess-query-performance-for-migrating-sql-server-databases-to-mongodb-atlas-on-aws.md)
+ [Automate cross-Region failover and failback by using DR Orchestrator Framework](automate-cross-region-failover-and-failback-by-using-dr-orchestrator-framework.md)
+ [Automate database tasks in SQL Server Express on Amazon EC2 by using AWS Lambda and Task Scheduler](automate-database-tasks-in-sql-server-express-edition-running-on-amazon-ec2.md)
+ [Build an advanced mainframe file viewer in the AWS Cloud](build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.md)
+ [Connect to Application Migration Service data and control planes over a private network](connect-to-application-migration-service-data-and-control-planes-over-a-private-network.md)
+ [Containerize mainframe workloads that have been modernized by Blu Age](containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.md)
+ [Convert JSON Oracle queries into PostgreSQL database SQL](convert-json-oracle-queries-into-postgresql-database-sql.md)
+ [Convert the Teradata NORMALIZE temporal feature to Amazon Redshift SQL](convert-the-teradata-normalize-temporal-feature-to-amazon-redshift-sql.md)
+ [Convert the Teradata RESET WHEN feature to Amazon Redshift SQL](convert-the-teradata-reset-when-feature-to-amazon-redshift-sql.md)
+ [Copy Amazon DynamoDB tables across accounts using AWS Backup](copy-amazon-dynamodb-tables-across-accounts-using-aws-backup.md)
+ [Deploy a Cassandra cluster on Amazon EC2 with private static IPs to avoid rebalancing](deploy-a-cassandra-cluster-on-amazon-ec2-with-private-static-ips-to-avoid-rebalancing.md)
+ [Deploy multiple-stack applications using AWS CDK with TypeScript](deploy-multiple-stack-applications-using-aws-cdk-with-typescript.md)
+ [Deploy SQL Server failover cluster instances on Amazon EC2 and Amazon FSx by using Terraform](deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.md)
+ [Emulate Oracle PL/SQL associative arrays in Amazon Aurora PostgreSQL and Amazon RDS for PostgreSQL](emulate-oracle-plsql-associative-arrays-in-aurora-and-rds-postgresql.md)
+ [Estimate the Amazon RDS engine size for an Oracle database by using AWR reports](estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.md)
+ [Generate data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.md)
+ [Handle anonymous blocks in Dynamic SQL statements in Aurora PostgreSQL](handle-anonymous-blocks-in-dynamic-sql-statements-in-aurora-postgresql.md)
+ [Identify duplicate container images automatically when migrating to an Amazon ECR repository](identify-duplicate-container-images-automatically-when-migrating-to-ecr-repository.md)
+ [Set up a Microsoft SQL Server failover cluster on Amazon EC2 by using FSx for Windows File Server](microsoft-sql-failover-cluster-on-amazon-ec2.md)
+ [Migrate Apache Cassandra workloads to Amazon Keyspaces by using AWS Glue](migrate-apache-cassandra-workloads-to-amazon-keyspaces-by-using-aws-glue.md)
+ [Migrate your container workloads from Azure Red Hat OpenShift (ARO) to Red Hat OpenShift Service on AWS (ROSA)](migrate-container-workloads-from-aro-to-rosa.md)
+ [Migrate from Oracle 8i or 9i to Amazon RDS for Oracle using SharePlex and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-oracle-using-shareplex-and-aws-dms.md)
+ [Migrate Microsoft SQL Server Always On availability group using AWS Application Migration Service](migrate-microsoft-sql-server-always-on-group-using-mgn.md)
+ [Migrate Oracle functions and procedures that have more than 100 arguments to PostgreSQL](migrate-oracle-functions-and-procedures-that-have-more-than-100-arguments-to-postgresql.md)
+ [Migrate SAP HANA to AWS using SAP HSR with the same hostname](migrate-sap-hana-to-aws-using-sap-hsr-with-the-same-hostname.md)
+ [Migrate SQL Server to AWS using distributed availability groups](migrate-sql-server-to-aws-using-distributed-availability-groups.md)
+ [Modernize and deploy mainframe applications using AWS Transform and Terraform](modernize-mainframe-app-transform-terraform.md)
+ [Modernize mainframe online printing workloads on AWS by using Micro Focus Enterprise Server and LRS VPSX/MFI](modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.md)
+ [Modernize mainframe output management on AWS by using Rocket Enterprise Server and LRS PageCenterX](modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.md)
+ [Modify HTTP headers when you migrate from F5 to an Application Load Balancer on AWS](modify-http-headers-when-you-migrate-from-f5-to-an-application-load-balancer-on-aws.md)
+ [Manage Multi-AZ failover for EMR clusters by using Application Recovery Controller](multi-az-failover-spark-emr-clusters-arc.md)
+ [Analyze object dependencies for partial database migrations from Oracle to PostgreSQL](multilevel-object-analysis-for-database-migration-from-oracle-to-postgresql.md)
+ [Set up a CI/CD pipeline for database migration by using Terraform](set-up-ci-cd-pipeline-for-db-migration-with-terraform.md)
+ [Set up disaster recovery for Oracle JD Edwards EnterpriseOne with AWS Elastic Disaster Recovery](set-up-disaster-recovery-for-oracle-jd-edwards-enterpriseone-with-aws-elastic-disaster-recovery.md)
+ [Simplify private certificate management by using AWS Private CA and AWS RAM](simplify-private-certificate-management-by-using-aws-private-ca-and-aws-ram.md)
+ [Transfer large-scale Db2 z/OS data to Amazon S3 in CSV files](transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.md)
+ [Transform Easytrieve to modern languages by using AWS Transform custom](transform-easytrieve-modern-languages.md)
# Modernization
**Topics**
+ [Automatically archive items to Amazon S3 using DynamoDB TTL](automatically-archive-items-to-amazon-s3-using-dynamodb-ttl.md)
+ [Build a multi-tenant serverless architecture in Amazon OpenSearch Service](build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.md)
+ [Deploy multiple-stack applications using AWS CDK with TypeScript](deploy-multiple-stack-applications-using-aws-cdk-with-typescript.md)
+ [Automate deployment of nested applications using AWS SAM](automate-deployment-of-nested-applications-using-aws-sam.md)
+ [Implement SaaS tenant isolation for Amazon S3 by using an AWS Lambda token vending machine](implement-saas-tenant-isolation-for-amazon-s3-by-using-an-aws-lambda-token-vending-machine.md)
+ [Implement the serverless saga pattern by using AWS Step Functions](implement-the-serverless-saga-pattern-by-using-aws-step-functions.md)
+ [Manage on-premises container applications by setting up Amazon ECS Anywhere with the AWS CDK](manage-on-premises-container-applications-by-setting-up-amazon-ecs-anywhere-with-the-aws-cdk.md)
+ [Modernize ASP.NET Web Forms applications on AWS](modernize-asp-net-web-forms-applications-on-aws.md)
+ [Tenant onboarding in SaaS architecture for the silo model using C\$1 and AWS CDK](tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.md)
+ [Decompose monoliths into microservices by using CQRS and event sourcing](decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.md)
+ [More patterns](modernization-more-patterns-pattern-list.md)
# Automatically archive items to Amazon S3 using DynamoDB TTL
*Tabby Ward, Amazon Web Services*
## Summary
This pattern provides steps to remove older data from an Amazon DynamoDB table and archive it to an Amazon Simple Storage Service (Amazon S3) bucket on Amazon Web Services (AWS) without having to manage a fleet of servers.
This pattern uses Amazon DynamoDB Time to Live (TTL) to automatically delete old items and Amazon DynamoDB Streams to capture the TTL-expired items. It then connects DynamoDB Streams to AWS Lambda, which runs the code without provisioning or managing any servers.
When new items are added to the DynamoDB stream, the Lambda function is initiated and writes the data to an Amazon Data Firehose delivery stream. Firehose provides a simple, fully managed solution to load the data as an archive into Amazon S3.
DynamoDB is often used to store time series data, such as webpage click-stream data or Internet of Things (IoT) data from sensors and connected devices. Rather than deleting less frequently accessed items, many customers want to archive them for auditing purposes. TTL simplifies this archiving by automatically deleting items based on the timestamp attribute.
Items deleted by TTL can be identified in DynamoDB Streams, which captures a time-ordered sequence of item-level modifications and stores the sequence in a log for up to 24 hours. This data can be consumed by a Lambda function and archived in an Amazon S3 bucket to reduce the storage cost. To further reduce the costs, [Amazon S3 lifecycle rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) can be created to automatically transition the data (as soon as it gets created) to lowest-cost [storage classes](https://aws.amazon.com/s3/storage-classes/).
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ [AWS Command Line Interface (AWS CLI) 1.7 or later](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html), installed and configured on macOS, Linux, or Windows.
+ [Python 3.7](https://www.python.org/downloads/release/python-370/) or later.
+ [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html), installed and configured. If Boto3 is not already installed, run the `python -m pip install boto3` command to install it.
## Architecture
**Technology stack **
+ Amazon DynamoDB
+ Amazon DynamoDB Streams
+ Amazon Data Firehose
+ AWS Lambda
+ Amazon S3
![\[Four-step process from DynamoDB to the S3 bucket.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9dbc833f-cf3c-4574-8f09-d0b81134fe41/images/50d9da65-5398-4a99-bc8f-58afc80e9d7b.png)
1. Items are deleted by TTL.
1. The DynamoDB stream trigger invokes the Lambda stream processor function.
1. The Lambda function puts records in the Firehose delivery stream in batch format.
1. Data records are archived in the S3 bucket.
## Tools
+ [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) – The AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) – Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale.
+ [Amazon DynamoDB Time to Live (TTL)](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html) – Amazon DynamoDB TTL helps you define a per-item timestamp to determine when an item is no longer required.
+ [Amazon DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Types_Amazon_DynamoDB_Streams.html) – Amazon DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours.
+ [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) – Amazon Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) – AWS Lambda runs code without the need to provision or manage servers. You pay only for the compute time you consume.
+ [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) – Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.
**Code**
The code for this pattern is available in the GitHub [Archive items to S3 using DynamoDB TTL](https://github.com/aws-samples/automatically-archive-items-to-s3-using-dynamodb-ttl) repository.
## Epics
### Set up a DynamoDB table, TTL , and a DynamoDB stream
| Task | Description | Skills required |
| --- | --- | --- |
| Create a DynamoDB table. | Use the AWS CLI to create a table in DynamoDB called `Reservation`. Choose random read capacity unit (RCU) and write capacity unit (WCU), and give your table two attributes: `ReservationID` and `ReservationDate`.
`ReservationDate` is an epoch timestamp that will be used to turn on TTL. | Cloud architect, App developer |
| Turn on DynamoDB TTL. | Use the AWS CLI to turn on DynamoDB TTL for the `ReservationDate` attribute.
| Cloud architect, App developer |
| Turn on a DynamoDB stream. | Use the AWS CLI to turn on a DynamoDB stream for the `Reservation` table by using the `NEW_AND_OLD_IMAGES` stream type.
This stream will contain records for new items, updated items, deleted items, and items that are deleted by TTL. The records for items that are deleted by TTL contain an additional metadata attribute to distinguish them from items that were deleted manually. The `userIdentity` field for TTL deletions indicates that the DynamoDB service performed the delete action. In this pattern, only the items deleted by TTL are archived, but you could archive only the records where `eventName` is `REMOVE` and `userIdentity` contains `principalId` equal to `dynamodb.amazonaws.com`. | Cloud architect, App developer |
### Create and configure an S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | Use the AWS CLI to create a destination S3 bucket in your AWS Region, replacing `us-east-1` with your Region and amzn-s3-demo-destination-bucket with the name of your bucket.
Make sure that your S3 bucket's name is globally unique, because the namespace is shared by all AWS accounts. | Cloud architect, App developer |
| Create a 30-day lifecycle policy for the S3 bucket. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-archive-items-to-amazon-s3-using-dynamodb-ttl.html) | Cloud architect, App developer |
### Create a Firehose delivery stream
| Task | Description | Skills required |
| --- | --- | --- |
| Create and configure a Firehose delivery stream. | Download and edit the `CreateFireHoseToS3.py` code example from the GitHub repository. This code is written in Python and shows you how to create a Firehose delivery stream and an AWS Identity and Access Management (IAM) role. The IAM role will have a policy that can be used by Firehose to write to the destination S3 bucket.To run the script, use the following command and command line arguments.Argument 1= ``, which is the Amazon Resource Name (ARN) for the bucket that you created earlierArgument 2= Your Firehose name (This pilot is using `firehose_to_s3_stream`.)Argument 3= Your IAM role name (This pilot is using `firehose_to_s3`.)
If the specified IAM role does not exist, the script will create an assume role with a trusted relationship policy, as well as a policy that grants sufficient Amazon S3 permission. For examples of these policies, see the *Additional information* section. | Cloud architect, App developer |
| Verify the Firehose delivery stream. | Describe the Firehose delivery stream by using the AWS CLI to verify that the delivery stream was successfully created.
| Cloud architect, App developer |
### Create a Lambda function to process the Firehose delivery stream
| Task | Description | Skills required |
| --- | --- | --- |
| Create a trust policy for the Lambda function. | Create a trust policy file with the following information.
This gives your function permission to access AWS resources. | Cloud architect, App developer |
| Create an execution role for the Lambda function. | To create the execution role, run the following code.
aws iam create-role --role-name lambda-ex --assume-role-policy-document file://TrustPolicy.json
| Cloud architect, App developer |
| Add permission to the role. | To add permission to the role, use the `attach-policy-to-role` command.
aws iam attach-role-policy --role-name lambda-ex --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole aws iam attach-role-policy --role-name lambda-ex --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaDynamoDBExecutionRole aws iam attach-role-policy --role-name lambda-ex --policy-arn arn:aws:iam::aws:policy/AmazonKinesisFirehoseFullAccess aws iam attach-role-policy --role-name lambda-ex --policy-arn arn:aws:iam::aws:policy/IAMFullAccess
| Cloud architect, App developer |
| Create a Lambda function. | Compress the `LambdaStreamProcessor.py` file from the code repository by running the following command.
zip function.zip LambdaStreamProcessor.py
When you create the Lambda function, you will need the Lambda execution role ARN. To get the ARN, run the following code.
aws iam get-role \ --role-name lambda-ex
To create the Lambda function, run the following code.
# Review the environment variables and replace them with your values.
| Cloud architect, App developer |
| Configure the Lambda function trigger. | Use the AWS CLI to configure the trigger (DynamoDB Streams), which invokes the Lambda function. The batch size of 400 is to avoid running into Lambda concurrency issues.
| Cloud architect, App developer |
### Test the functionality
| Task | Description | Skills required |
| --- | --- | --- |
| Add items with expired timestamps to the Reservation table. | To test the functionality, add items with expired epoch timestamps to the `Reservation` table. TTL will automatically delete items based on the timestamp. The Lambda function is initiated upon DynamoDB Stream activities, and it filters the event to identify `REMOVE` activity or deleted items. It then puts records in the Firehose delivery stream in batch format.The Firehose delivery stream transfers items to a destination S3 bucket with the `firehosetos3example/year=current year/month=current month/ day=current day/hour=current hour/` prefix.To optimize data retrieval, configure Amazon S3 with the `Prefix` and `ErrorOutputPrefix` that are detailed in the *Additional information* section. | Cloud architect |
### Clean up the resources
| Task | Description | Skills required |
| --- | --- | --- |
| Delete all resources. | Delete all the resources to ensure that you aren't charged for any services that you aren't using. | Cloud architect, App developer |
## Related resources
+ [Managing your storage lifecycle](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)
+ [Amazon S3 Storage Classes](https://aws.amazon.com/s3/storage-classes/)
+ [AWS SDK for Python (Boto3) documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
## Additional information
**Create and configure a Firehose delivery stream – Policy examples**
*Firehose trusted relationship policy example document*
```
firehose_assume_role = {
'Version': '2012-10-17',
'Statement': [
{
'Sid': '',
'Effect': 'Allow',
'Principal': {
'Service': 'firehose.amazonaws.com'
},
'Action': 'sts:AssumeRole'
}
]
}
```
*S3 permissions policy example*
```
s3_access = {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"{your s3_bucket ARN}/*",
"{Your s3 bucket ARN}"
]
}
]
}
```
**Test the functionality – Amazon S3 configuration**
The Amazon S3 configuration with the following `Prefix` and `ErrorOutputPrefix` is chosen to optimize data retrieval.
*Prefix *
```
firehosetos3example/year=! {timestamp: yyyy}/month=! {timestamp:MM}/day=! {timestamp:dd}/hour=!{timestamp:HH}/
```
Firehose first creates a base folder called `firehosetos3example` directly under the S3 bucket. It then evaluates the expressions `!{timestamp:yyyy}`, `!{timestamp:MM}`, `!{timestamp:dd}`, and `!{timestamp:HH}` to year, month, day, and hour using the Java [DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) format.
For example, an approximate arrival timestamp of 1604683577 in Unix epoch time evaluates to `year=2020`, `month=11`, `day=06`, and `hour=05`. Therefore, the location in Amazon S3, where data records are delivered, evaluates to `firehosetos3example/year=2020/month=11/day=06/hour=05/`.
*ErrorOutputPrefix*
```
firehosetos3erroroutputbase/!{firehose:random-string}/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/
```
The `ErrorOutputPrefix` results in a base folder called `firehosetos3erroroutputbase` directly under the S3 bucket. The expression `!{firehose:random-string}` evaluates to an 11-character random string such as `ztWxkdg3Thg`. The location for an Amazon S3 object where failed records are delivered could evaluate to `firehosetos3erroroutputbase/ztWxkdg3Thg/processing-failed/2020/11/06/`.
# Build a multi-tenant serverless architecture in Amazon OpenSearch Service
*Tabby Ward and Nisha Gambhir, Amazon Web Services*
## Summary
Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale Elasticsearch, which is a popular open-source search and analytics engine. OpenSearch Service provides free-text search as well as near real-time ingestion and dashboarding for streaming data such as logs and metrics.
Software as a service (SaaS) providers frequently use OpenSearch Service to address a broad range of use cases, such as gaining customer insights in a scalable and secure way while reducing complexity and downtime.
Using OpenSearch Service in a multi-tenant environment introduces a series of considerations that affect partitioning, isolation, deployment, and management of your SaaS solution. SaaS providers have to consider how to effectively scale their Elasticsearch clusters with continually shifting workloads. They also need to consider how tiering and noisy neighbor conditions could impact their partitioning model.
This pattern reviews the models that are used to represent and isolate tenant data with Elasticsearch constructs. In addition, the pattern focuses on a simple serverless reference architecture as an example to demonstrate indexing and searching using OpenSearch Service in a multi-tenant environment. It implements the pool data partitioning model, which shares the same index among all tenants while maintaining a tenant's data isolation. This pattern uses the following AWS services: Amazon API Gateway, AWS Lambda, Amazon Simple Storage Service (Amazon S3), and OpenSearch Service.
For more information about the pool model and other data partitioning models, see the [Additional information](#build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service-additional) section.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ [AWS Command Line Interface (AWS CLI) version 2.x](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html), installed and configured on macOS, Linux, or Windows
+ [Python version 3.9](https://www.python.org/downloads/release/python-3921/)
+ [pip3](https://pip.pypa.io/en/stable/) – The Python source code is provided as a .zip file to be deployed in a Lambda function. If you want to use the code locally or customize it, follow these steps to develop and recompile the source code:
1. Generate the `requirements.txt` file by running the the following command in the same directory as the Python scripts: `pip3 freeze > requirements.txt`
1. Install the dependencies: `pip3 install -r requirements.txt`
**Limitations**
+ This code runs in Python, and doesn’t currently support other programming languages.
+ The sample application doesn’t include AWS cross-Region or disaster recovery (DR) support.
+ This pattern is intended for demonstration purposes only. It is not intended to be used in a production environment.
## Architecture
The following diagram illustrates the high-level architecture of this pattern. The architecture includes the following:
+ Lambda to index and query the content
+ OpenSearch Service to perform search
+ API Gateway to provide an API interaction with the user
+ Amazon S3 to store raw (non-indexed) data
+ Amazon CloudWatch to monitor logs
+ AWS Identity and Access Management (IAM) to create tenant roles and policies
![\[High-level multi-tenant serverless architecture.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/1a8501e7-0776-4aca-aed3-28e3ada1d15d.png)
**Automation and scale**
For simplicity, the pattern uses AWS CLI to provision the infrastructure and to deploy the sample code. You can create an CloudFormation template or AWS Cloud Development Kit (AWS CDK) scripts to automate the pattern.
## Tools
**AWS services**
+ [AWS CLI](https://aws.amazon.com/cli/) is a unified tool for managing AWS services and resources by using commands in your command-line shell.
+ [Lambda](https://aws.amazon.com/lambda/) is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second.
+ [API Gateway](https://aws.amazon.com/api-gateway/) is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale.
+ [Amazon S3](https://aws.amazon.com/s3/) is an object storage service that lets you store and retrieve any amount of information at any time, from anywhere on the web.
+ [OpenSearch Service](https://aws.amazon.com/opensearch-service/) is a fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost-effectively at scale.
**Code**
The attachment provides sample files for this pattern. These include:
+ `index_lambda_package.zip` – The Lambda function for indexing data in OpenSearch Service by using the pool model.
+ `search_lambda_package.zip` – The Lambda function for searching for data in OpenSearch Service.
+ `Tenant-1-data` – Sample raw (non-indexed) data for Tenant-1.
+ `Tenant-2-data` – Sample raw (non-indexed) data for Tenant-2.
**Important**
The stories in this pattern include AWS CLI command examples that are formatted for Unix, Linux, and macOS. For Windows, replace the backslash (\$1) Unix continuation character at the end of each line with a caret (^).
**Note**
In AWS CLI commands, replace all values within the angle brackets (<>) with correct values.
## Epics
### Create and configure an S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | Create an S3 bucket in your AWS Region. This bucket will hold the non-indexed tenant data for the sample application. Make sure that the S3 bucket's name is globally unique, because the namespace is shared by all AWS accounts.To create an S3 bucket, you can use the AWS CLI [create-bucket](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/create-bucket.html) command as follows:
aws s3api create-bucket \ --bucket \ --region
where `tenantrawdata` is the S3 bucket name. (You can use any unique name that follows [the bucket naming guidelines](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html).) | Cloud architect, Cloud administrator |
### Create and configure an Elasticsearch cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Create an OpenSearch Service domain. | Run the AWS CLI [create-elasticsearch-domain](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/es/create-elasticsearch-domain.html) command to create an OpenSearch Service domain:
The instance count is set to 1 because the domain is for testing purposes. You need to enable fine-grained access control by using the `advanced-security-options` parameter, because the details cannot be changed after the domain has been created. This command creates a master user name (`KibanaUser`) and a password that you can use to log in to the Kibana console.Because the domain is part of a virtual private cloud (VPC), you have to make sure that you can reach the Elasticsearch instance by specifying the access policy to use.For more information, see [Launching your Amazon OpenSearch Service domains within a VPC](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html) in the AWS documentation. | Cloud architect, Cloud administrator |
| Set up a bastion host. | Set up a Amazon Elastic Compute Cloud (Amazon EC2) Windows instance as a bastion host to access the Kibana console. The Elasticsearch security group must allow traffic from the Amazon EC2 security group. For instructions, see the blog post [Controlling Network Access to EC2 Instances Using a Bastion Server](https://aws.amazon.com/blogs/security/controlling-network-access-to-ec2-instances-using-a-bastion-server/).When the bastion host has been set up, and you have the security group that is associated with the instance available, use the AWS CLI [authorize-security-group-ingress](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/authorize-security-group-ingress.html) command to add permission to the Elasticsearch security group to allow port 443 from the Amazon EC2 (bastion host) security group.
| Cloud architect, Cloud administrator |
### Create and configure the Lambda index function
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Lambda execution role. | Run the AWS CLI [create-role](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-role.html) command to grant the Lambda index function access to AWS services and resources:
aws iam create-role \ --role-name index-lambda-role \ --assume-role-policy-document file://lambda_assume_role.json
where `lambda_assume_role.json` is a JSON document that grants `AssumeRole` permissions to the Lambda function, as follows:
| Cloud architect, Cloud administrator |
| Attach managed policies to the Lambda role. | Run the AWS CLI [attach-role-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html) command to attach managed policies to the role created in the previous step. These two policies give the role permissions to create an elastic network interface and to write logs to CloudWatch Logs.
aws iam attach-role-policy \ --role-name index-lambda-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam attach-role-policy \ --role-name index-lambda-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
| Cloud architect, Cloud administrator |
| Create a policy to give the Lambda index function permission to read the S3 objects. | Run the AWS CLI [create-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-policy.html) command to to give the Lambda index function `s3:GetObject` permission to read the objects in the S3 bucket:
aws iam create-policy \ --policy-name s3-permission-policy \ --policy-document file://s3-policy.json
The file `s3-policy.json` is a JSON document shown below that grants `s3:GetObject` permissions to allow read access to S3 objects. If you used a different name when you created the S3 bucket, provide the correct bucket name in the `Resource `section:
| Cloud architect, Cloud administrator |
| Attach the Amazon S3 permission policy to the Lambda execution role. | Run the AWS CLI [attach-role-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html) command to attach the Amazon S3 permission policy you created in the previous step to the Lambda execution role:
aws iam attach-role-policy \ --role-name index-lambda-role \ --policy-arn
where `PolicyARN` is the Amazon Resource Name (ARN) of the Amazon S3 permission policy. You can get this value from the output of the previous command. | Cloud architect, Cloud administrator |
| Create the Lambda index function. | Run the AWS CLI [create-function](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/create-function.html) command to create the Lambda index function, which will access OpenSearch Service:
| Cloud architect, Cloud administrator |
| Allow Amazon S3 to call the Lambda index function. | Run the AWS CLI [add-permission](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/add-permission.html) command to give Amazon S3 the permission to call the Lambda index function:
| Cloud architect, Cloud administrator |
| Add a Lambda trigger for the Amazon S3 event. | Run the AWS CLI [put-bucket-notification-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-bucket-notification-configuration.html) command to send notifications to the Lambda index function when the Amazon S3 `ObjectCreated` event is detected. The index function runs whenever an object is uploaded to the S3 bucket.
The file `s3-trigger.json` is a JSON document in the current folder that adds the resource policy to the Lambda function when the Amazon S3 `ObjectCreated` event occurs. | Cloud architect, Cloud administrator |
### Create and configure the Lambda search function
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Lambda execution role. | Run the AWS CLI [create-role](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-role.html) command to grant the Lambda search function access to AWS services and resources:
aws iam create-role \ --role-name search-lambda-role \ --assume-role-policy-document file://lambda_assume_role.json
where `lambda_assume_role.json` is a JSON document in the current folder that grants `AssumeRole` permissions to the Lambda function, as follows:
| Cloud architect, Cloud administrator |
| Attach managed policies to the Lambda role. | Run the AWS CLI [attach-role-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html) command to attach managed policies to the role created in the previous step. These two policies give the role permissions to create an elastic network interface and to write logs to CloudWatch Logs.
aws iam attach-role-policy \ --role-name search-lambda-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam attach-role-policy \ --role-name search-lambda-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
| Cloud architect, Cloud administrator |
| Create the Lambda search function. | Run the AWS CLI [create-function](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/create-function.html) command to create the Lambda search function, which will access OpenSearch Service:
| Cloud architect, Cloud administrator |
### Create and configure tenant roles
| Task | Description | Skills required |
| --- | --- | --- |
| Create tenant IAM roles. | Run the AWS CLI [create-role](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-role.html) command to create two tenant roles that will be used to test the search functionality:
aws iam create-role \ --role-name Tenant-1-role \ --assume-role-policy-document file://assume-role-policy.json
aws iam create-role \ --role-name Tenant-2-role \ --assume-role-policy-document file://assume-role-policy.json
The file `assume-role-policy.json` is a JSON document in the current folder that grants `AssumeRole` permissions to the Lambda execution role:
| Cloud architect, Cloud administrator |
| Create a tenant IAM policy. | Run the AWS CLI [create-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-policy.html) command to create a tenant policy that grants access to Elasticsearch operations:
aws iam create-policy \ --policy-name tenant-policy \ --policy-document file://policy.json
The file `policy.json` is a JSON document in the current folder that grants permissions on Elasticsearch:
| Cloud architect, Cloud administrator |
| Attach the tenant IAM policy to the tenant roles. | Run the AWS CLI [attach-role-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html) command to attach the tenant IAM policy to the two tenant roles you created in the earlier step:
aws iam attach-role-policy \ --policy-arn arn:aws:iam::account-id:policy/tenant-policy \ --role-name Tenant-1-role
aws iam attach-role-policy \ --policy-arn arn:aws:iam::account-id:policy/tenant-policy \ --role-name Tenant-2-role
The policy ARN is from the output of the previous step. | Cloud architect, Cloud administrator |
| Create an IAM policy to give Lambda permissions to assume role. | Run the AWS CLI [create-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-policy.html) command to create a policy for Lambda to assume the tenant role:
aws iam create-policy \ --policy-name assume-tenant-role-policy \ --policy-document file://lambda_policy.json
The file `lambda_policy.json` is a JSON document in the current folder that grants permissions to `AssumeRole`:
For `Resource`, you can use a wildcard character to avoid creating a new policy for each tenant. | Cloud architect, Cloud administrator |
| Create an IAM policy to give the Lambda index role permission to access Amazon S3. | Run the AWS CLI [create-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-policy.html) command to give the Lambda index role permission to access the objects in the S3 bucket:
aws iam create-policy \ --policy-name s3-permission-policy \ --policy-document file://s3_lambda_policy.json
The file `s3_lambda_policy.json` is the following JSON policy document in the current folder:
| Cloud architect, Cloud administrator |
| Attach the policy to the Lambda execution role. | Run the AWS CLI [attach-role-policy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html) command to attach the policy created in the previous step to the Lambda index and search execution roles you created earlier:
aws iam attach-role-policy \ --policy-arn arn:aws:iam::account-id:policy/assume-tenant-role-policy \ --role-name index-lambda-role
aws iam attach-role-policy \ --policy-arn arn:aws:iam::account-id:policy/assume-tenant-role-policy \ --role-name search-lambda-role
aws iam attach-role-policy \ --policy-arn arn:aws:iam::account-id:policy/s3-permission-policy \ --role-name index-lambda-role
The policy ARN is from the output of the previous step. | Cloud architect, Cloud administrator |
### Create and configure a search API
| Task | Description | Skills required |
| --- | --- | --- |
| Create a REST API in API Gateway. | Run the AWS CLI [create-rest-api](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/apigateway/create-rest-api.html) command to create a REST API resource:
For the endpoint configuration type, you can specify `EDGE` instead of `REGIONAL` to use edge locations instead of a particular AWS Region.Note the value of the `id` field from the command output. This is the API ID that you will use in subsequent commands. | Cloud architect, Cloud administrator |
| Create a resource for the search API. | The search API resource starts the Lambda search function with the resource name `search`. (You don’t have to create an API for the Lambda index function, because it runs automatically when objects are uploaded to the S3 bucket.)[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html) | Cloud architect, Cloud administrator |
| Create a GET method for the search API. | Run the AWS CLI [put-method](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/apigateway/put-method.html) command to create a `GET `method for the search API:
For `resource-id`, specify the ID from the output of the `create-resource` command. | Cloud architect, Cloud administrator |
| Create a method response for the search API. | Run the AWS CLI [put-method-response](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/apigateway/put-method-response.html) command to add a method response for the search API:
For `resource-id`, specify the ID from the output of the earlier `create-resource` command. | Cloud architect, Cloud administrator |
| Set up a proxy Lambda integration for the search API. | Run the AWS CLI [put-integration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/apigateway/put-integration.html) command to set up an integration with the Lambda search function:
aws apigateway put-integration \ --rest-api-id \ --resource-id \ --http-method GET \ --type AWS_PROXY \ --integration-http-method GET \ --uri arn:aws:apigateway:region:lambda:path/2015-03-31/functions/arn:aws:lambda:::function:/invocations
For `resource-id`, specify the ID from the earlier `create-resource` command. | Cloud architect, Cloud administrator |
| Grant API Gateway permission to call the Lambda search function. | Run the AWS CLI [add-permission](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/add-permission.html) command to give API Gateway permission to use the search function:
Change the `source-arn` path if you used a different API resource name instead of `search`. | Cloud architect, Cloud administrator |
| Deploy the search API. | Run the AWS CLI [create-deployment](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/apigateway/create-deployment.html) command to create a stage resource named `dev`:
aws apigateway create-deployment \ --rest-api-id \ --stage-name dev
If you update the API, you can use the same AWS CLI command to redeploy it to the same stage. | Cloud architect, Cloud administrator |
### Create and configure Kibana roles
| Task | Description | Skills required |
| --- | --- | --- |
| Log in to the Kibana console. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html) | Cloud architect, Cloud administrator |
| Create and configure Kibana roles. | To provide data isolation and to make sure that one tenant cannot retrieve the data of another tenant, you need to use document security, which allows tenants to access only documents that contain their tenant ID.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html) | Cloud architect, Cloud administrator |
| Map users to roles. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html)We recommend that you automate the creation of the tenant and Kibana roles at the time of tenant onboarding. | Cloud architect, Cloud administrator |
| Create the tenant-data index. | In the navigation pane, under **Management**, choose **Dev Tools**, and then run the following command. This command creates the `tenant-data` index to define the mapping for the `TenantId` property.
| Cloud architect, Cloud administrator |
### Create VPC endpoints for Amazon S3 and AWS STS
| Task | Description | Skills required |
| --- | --- | --- |
| Create a VPC endpoint for Amazon S3. | Run the AWS CLI [create-vpc-endpoint](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/create-vpc-endpoint.html) command to create a VPC endpoint for Amazon S3. The endpoint enables the Lambda index function in the VPC to access Amazon S3.
For `vpc-id`, specify the VPC that you’re using for the Lambda index function. For `service-name`, use the correct URL for the Amazon S3 endpoint. For `route-table-ids`, specify the route table that’s associated with the VPC endpoint. | Cloud architect, Cloud administrator |
| Create a VPC endpoint for AWS STS. | Run the AWS CLI [create-vpc-endpoint](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/create-vpc-endpoint.html) command to create a VPC endpoint for AWS Security Token Service (AWS STS). The endpoint enables the Lambda index and search functions in the VPC to access AWS STS. The functions use AWS STS when they assume the IAM role.
For `vpc-id`, specify the VPC that you’re using for the Lambda index and search functions. For `subnet-id`, provide the subnet in which this endpoint should be created. For `security-group-id`, specify the security group to associate this endpoint with. (It could be the same as the security group Lambda uses.) | Cloud architect, Cloud administrator |
### Test multi-tenancy and data isolation
| Task | Description | Skills required |
| --- | --- | --- |
| Update the Python files for the index and search functions. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html)You can get the Elasticsearch endpoint from the **Overview **tab of the OpenSearch Service console. It has the format `.es.amazonaws.com`. | Cloud architect, App developer |
| Update the Lambda code. | Use the AWS CLI [update-function-code](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/update-function-code.html) command to update the Lambda code with the changes you made to the Python files:
| Cloud architect, App developer |
| Upload raw data to the S3 bucket. | Use the AWS CLI [cp](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) command to upload data for the Tenant-1 and Tenant-2 objects to the `tenantrawdata` bucket (specify the name of the S3 bucket you created for this purpose):
The S3 bucket is set up to run the Lambda index function whenever data is uploaded so that the document is indexed in Elasticsearch. | Cloud architect, Cloud administrator |
| Search data from the Kibana console. | On the Kibana console, run the following query:
GET tenant-data/_search
This query displays all the documents indexed in Elasticsearch. In this case, you should see two separate documents for Tenant-1 and Tenant-2. | Cloud architect, Cloud administrator |
| Test the search API from API Gateway. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service.html)For screen illustrations, see the [Additional information](#build-a-multi-tenant-serverless-architecture-in-amazon-opensearch-service-additional) section. | Cloud architect, App developer |
| Clean up resources. | Clean up all the resources you created to prevent additional charges to your account. | AWS DevOps, Cloud architect, Cloud administrator |
## Related resources
+ [AWS SDK for Python (Boto)](https://aws.amazon.com/sdk-for-python/)
+ [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/)
+ [API Gateway documentation](https://docs.aws.amazon.com/apigateway/)
+ [Amazon S3 documentation](https://docs.aws.amazon.com/s3/)
+ [Amazon OpenSearch Service documentation](https://docs.aws.amazon.com/elasticsearch-service/)
+ [Fine-grained access control in Amazon OpenSearch Service](https://docs.amazonaws.cn/en_us/elasticsearch-service/latest/developerguide/fgac.html)
+ [Creating a search application with Amazon OpenSearch Service](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/search-example.html)
+ [Launching your Amazon OpenSearch Service domains within a VPC](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html)
## Additional information
**Data partitioning models**
There are three common data partitioning models used in multi-tenant systems: silo, pool, and hybrid. The model you choose depends on the compliance, noisy neighbor, operations, and isolation needs of your environment.
*Silo model*
In the silo model, each tenant’s data is stored in a distinct storage area where there is no commingling of tenant data. You can use two approaches to implement the silo model with OpenSearch Service: domain per tenant and index per tenant.
+ **Domain per tenant** – You can use a separate OpenSearch Service domain (synonymous with an Elasticsearch cluster) per tenant. Placing each tenant in its own domain provides all the benefits associated with having data in a standalone construct. However, this approach introduces management and agility challenges. Its distributed nature makes it harder to aggregate and assess the operational health and activity of tenants. This is a costly option that requires each OpenSearch Service domain to have three master nodes and two data nodes for production workloads at the minimum.
![\[Domain per tenant silo model for multi-tenant serverless architectures.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/c2195f82-e5ed-40bb-b76a-3b0210bf1254.png)
+ **Index per tenant** – You can place tenant data in separate indexes within an OpenSearch Service cluster. With this approach, you use a tenant identifier when you create and name the index, by prepending the tenant identifier to the index name. The index per tenant approach helps you achieve your silo goals without introducing a completely separate cluster for each tenant. However, you might encounter memory pressure if the number of indexes grows, because this approach requires more shards, and the master node has to handle more allocation and rebalancing.
![\[Index per tenant silo model for multi-tenant serverless architectures.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/354a9463-25bb-422b-84de-d4875a7c8ea2.png)
**Isolation in the silo model** – In the silo model, you use IAM policies to isolate the domains or indexes that hold each tenant’s data. These policies prevent one tenant from accessing another tenant’s data. To implement your silo isolation model, you can create a resource-based policy that controls access to your tenant resource. This is often a domain access policy that specifies which actions a principal can perform on the domain’s sub-resources, including Elasticsearch indexes and APIs. With IAM identity-based polices, you can specify *allowed* or *denied* actions on the domain, indexes, or APIs within OpenSearch Service. The `Action` element of an IAM policy describes the specific action or actions that are allowed or denied by the policy, and the `Principal `element specifies the affected accounts, users, or roles.
The following sample policy grants Tenant-1 full access (as specified by `es:*`) to the sub-resources on the `tenant-1` domain only. The trailing `/*` in the `Resource` element indicates that this policy applies to the domain’s sub-resources, not to the domain itself. When this policy is in effect, tenants are not allowed to create a new domain or modify settings on an existing domain.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:::user/Tenant-1"
},
"Action": "es:*",
"Resource": "arn:aws:es:::domain/tenant-1/*"
}
]
}
```
To implement the tenant per index silo model, you would need to modify this sample policy to further restrict Tenant-1 to the specified index or indexes, by specifying the index name. The following sample policy restricts Tenant-1 to the `tenant-index-1` index.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/Tenant-1"
},
"Action": "es:*",
"Resource": "arn:aws:es:::domain/test-domain/tenant-index-1/*"
}
]
}
```
*Pool model*
In the pool model, all tenant data is stored in an index within the same domain. The tenant identifier is included in the data (document) and used as the partition key, so you can determine which data belongs to which tenant. This model reduces the management overhead. Operating and managing the pooled index is easier and more efficient than managing multiple indexes. However, because tenant data is commingled within the same index, you lose the natural tenant isolation that the silo model provides. This approach might also degrade performance because of the noisy neighbor effect.
![\[Pool model for multi-tenant serverless architectures.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/c2c3bb0f-6ccd-47a7-ab67-e7f3f8c7f289.png)
**Tenant isolation in the pool model** – In general, tenant isolation is challenging to implement in the pool model. The IAM mechanism used with the silo model doesn’t allow you to describe isolation based on the tenant ID stored in your document.
An alternative approach is to use the [fine-grained access control](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/fgac.html) (FGAC) support provided by the Open Distro for Elasticsearch. FGAC allows you to control permissions at an index, document, or field level. With each request, FGAC evaluates the user credentials and either authenticates the user or denies access. If FGAC authenticates the user, it fetches all roles mapped to that user and uses the complete set of permissions to determine how to handle the request.
To achieve the required isolation in the pooled model, you can use [document-level security](https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/document-level-security/), which lets you restrict a role to a subset of documents in an index. The following sample role restricts queries to Tenant-1. By applying this role to Tenant-1, you can achieve the necessary isolation.
```
{
"bool": {
"must": {
"match": {
"tenantId": "Tenant-1"
}
}
}
}
```
*Hybrid model*
The hybrid model uses a combination of the silo and pool models in the same environment to offer unique experiences to each tenant tier (such as free, standard, and premium tiers). Each tier follows the same security profile that was used in the pool model.
![\[Hybrid model for multi-tenant serverless architectures.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/e7def98a-38ef-435a-9881-7e95ae4d4940.png)
**Tenant isolation in the hybrid model** – In the hybrid model, you follow the same security profile as in the pool model, where using the FGAC security model at the document level provided tenant isolation. Although this strategy simplifies cluster management and offers agility, it complicates other aspects of the architecture. For example, your code requires additional complexity to determine which model is associated with each tenant. You also have to ensure that single-tenant queries don’t saturate the entire domain and degrade the experience for other tenants.
**Testing in API Gateway**
*Test window for Tenant-1 query*
![\[Test window for Tenant-1 query.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/a6757d3f-977a-4ecc-90cb-83ab7f1c3588.png)
*Test window for Tenant-2 query*
![\[Test window for Tenant-2 query.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/750196bb-03f6-4b6e-92cd-eb7141602547/images/31bfd656-33ca-4750-b6e6-da4d703c2071.png)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/750196bb-03f6-4b6e-92cd-eb7141602547/attachments/attachment.zip)
# Deploy multiple-stack applications using AWS CDK with TypeScript
*Dr. Rahul Sharad Gaikwad, Amazon Web Services*
## Summary
This pattern provides a step-by-step approach for application deployment on Amazon Web Services (AWS) using AWS Cloud Development Kit (AWS CDK) with TypeScript. As an example, the pattern deploys a serverless real-time analytics application.
The pattern builds and deploys nested stack applications. The parent AWS CloudFormation stack calls the child, or nested, stacks. Each child stack builds and deploys the AWS resources that are defined in the CloudFormation stack. AWS CDK Toolkit, the command line interface (CLI) command `cdk`, is the primary interface for the CloudFormation stacks.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Existing virtual private cloud (VPC) and subnets
+ AWS CDK Toolkit installed and configured
+ A user with administrator permissions and a set of access keys.
+ Node.js
+ AWS Command Line Interface (AWS CLI)
**Limitations **
+ Because AWS CDK uses AWS CloudFormation, AWS CDK applications are subject to CloudFormation service quotas. For more information, see [AWS CloudFormation quotas](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html).
**Product versions**
This pattern has been built and tested using the following tools and versions.
+ AWS CDK Toolkit 1.83.0
+ Node.js 14.13.0
+ npm 7.0.14
The pattern should work with any version of AWS CDK or npm. Note that Node.js versions 13.0.0 through 13.6.0 are not compatible with the AWS CDK.
## Architecture
**Target technology stack **
+ AWS Amplify Console
+ Amazon API Gateway
+ AWS CDK
+ Amazon CloudFront
+ Amazon Cognito
+ Amazon DynamoDB
+ Amazon Data Firehose
+ Amazon Kinesis Data Streams
+ AWS Lambda
+ Amazon Simple Storage Service (Amazon S3)
**Target architecture**
The following diagram shows multiple-stack application deployment using AWS CDK with TypeScript.
![\[Stack architecture in the VPC, with a parent stack and two child stacks that contain resources.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0ac29a11-1362-4084-92ed-6b85205763ca/images/8f92e86a-aa3d-4f8a-9b11-b92c52a7226c.png)
The following diagram shows the architecture of the example serverless real-time application.
![\[Application architecture in the Region.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0ac29a11-1362-4084-92ed-6b85205763ca/images/2df00faf-f871-4aec-9655-19ba2eb14cf8.png)
## Tools
**Tools**
+ [AWS Amplify Console](https://docs.aws.amazon.com/amplify/latest/userguide/welcome.html) is the control center for fullstack web and mobile application deployments in AWS. Amplify Console hosting provides a git-based workflow for hosting fullstack serverless web apps with continuous deployment. The Admin UI is a visual interface for frontend web and mobile developers to create and manage app backends outside the AWS console.
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale.
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) is a command line cloud development kit that helps you interact with your AWS CDK app. The `cdk` CLI command is the primary tool for interacting with your AWS CDK app. It runs your app, interrogates the application model you defined, and produces and deploys the AWS CloudFormation templates generated by the AWS CDK.
+ [Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html) is a web service that speeds up distribution of static and dynamic web content, such as .html, .css, .js, and image files. CloudFront delivers your content through a worldwide network of data centers called edge locations for lower latency and improved performance.
+ [Amazon Cognito](https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html) provides authentication, authorization, and user management for your web and mobile apps. Your users can sign in directly or through a third party.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
+ [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) is a fully managed service for delivering real-time [streaming data](https://aws.amazon.com/streaming-data/) to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers.
+ [Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) is a service for collecting and processing large streams of data records in real time.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that supports running code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume—there is no charge when your code is not running.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Code**
The code for this pattern is attached.
## Epics
### Install AWS CDK Toolkit
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS CDK Toolkit. | To install AWS CDK Toolkit globally, run the following command.`npm install -g aws-cdk` | DevOps |
| Verify the version. | To verify the AWS CDK Toolkit version, run the following command. `cdk --version` | DevOps |
### Set up AWS credentials
| Task | Description | Skills required |
| --- | --- | --- |
| Set up credentials. | To set up credentials, run the `aws configure` command and follow the prompts.
$aws configure AWS Access Key ID [None]: AWS Secret Access Key [None]: your_secret_access_key Default region name [None]: Default output format [None]:
| DevOps |
### Download the project code
| Task | Description | Skills required |
| --- | --- | --- |
| Download the attached project code. | For more information about the directory and file structure, see the *Additional information* section. | DevOps |
### Bootstrap the AWS CDK environment
| Task | Description | Skills required |
| --- | --- | --- |
| Bootstrap the environment. | To deploy the AWS CloudFormation template to the account and AWS Region that you want to use, run the following command.`cdk bootstrap /`For more information, see the [AWS documentation](https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html). | DevOps |
### Build and deploy the project
| Task | Description | Skills required |
| --- | --- | --- |
| Build the project. | To build the project code, run the `npm run build` command. | DevOps |
| Deploy the project. | To deploy the project code, run the `cdk deploy` command. | |
### Verify outputs
| Task | Description | Skills required |
| --- | --- | --- |
| Verify stack creation. | On the AWS Management Console, choose **CloudFormation**. In the stacks for the project, verify that a parent stack and two child stacks have been created. | DevOps |
### Test the application
| Task | Description | Skills required |
| --- | --- | --- |
| Send data to Kinesis Data Streams. | Configure your AWS Account to send data to Kinesis Data Streams using Amazon Kinesis Data Generator (KDG). For more information, see [Amazon Kinesis Data Generator](https://awslabs.github.io/amazon-kinesis-data-generator/web/help.html). | DevOps |
| Create an Amazon Cognito user. | To create an Amazon Cognito user, download the cognito-setup.json CloudFormation template from the *Create an Amazon Cognito User* section on the [Kinesis Data Generator help page](https://awslabs.github.io/amazon-kinesis-data-generator/web/help.html). Initiate the template, and then enter your Amazon Cognito **Username** and **Password**.The **Outputs** tab lists the Kinesis Data Generator URL. | DevOps |
| Log in to Kinesis Data Generator | To log in to KDG, use the Amazon Cognito credentials that you provided and the Kinesis Data Generator URL. | DevOps |
| Test the application. | In KDG, in **Record template**, **Template 1**, paste the test code from the *Additional information* section, and choose **Send data**. | DevOps |
| Test API Gateway. | After the data has been ingested, test API Gateway by using the `GET` method to retrieve data. | DevOps |
## Related resources
**References**
+ [AWS Cloud Development Kit](https://aws.amazon.com/cdk/)
+ [AWS CDK on GitHub](https://github.com/aws/aws-cdk)
+ [Working with nested stacks](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-nested-stacks.html)
+ [AWS sample example - Serverless real-time analytics](https://github.com/aws-samples/serverless-realtime-analytics)
## Additional information
**Directory and file details**
This pattern sets up the following three stacks.
+ `parent-cdk-stack.ts` – This stack acts as the parent stack and calls the two child applications as nested stacks.
+ `real-time-analytics-poc-stack.ts` – This nested stack contains the infrastructure and application code.
+ `real-time-analytics-web-stack.ts` – This nested stack contains only the static web application code.
*Important files and their functionality*
+ `bin/real-time-analytics-poc.ts` – Entry point of the AWS CDK application. It loads all stacks defined under `lib/`.
+ `lib/real-time-analytics-poc-stack.ts` – Definition of the AWS CDK application’s stack (`real-time-analytics-poc`).
+ `lib/real-time-analytics-web-stack.ts` – Definition of the AWS CDK application’s stack (`real-time-analytics-web-stack`).
+ `lib/parent-cdk-stack.ts` – Definition of the AWS CDK application’s stack (`parent-cdk`).
+ `package.json` – npm module manifest, which includes the application name, version, and dependencies.
+ `package-lock.json` – Maintained by npm.
+ `cdk.json` – Toolkit for running the application.
+ `tsconfig.json` – The project’s TypeScript configuration.
+ `.gitignore` – List of files that Git should exclude from source control.
+ `node_modules` – Maintained by npm; includes the project’s dependencies.
The following section of code in the parent stack calls child applications as a nested AWS CDK stacks.
```
import * as cdk from '@aws-cdk/core';
import { Construct, Stack, StackProps } from '@aws-cdk/core';
import { RealTimeAnalyticsPocStack } from './real-time-analytics-poc-stack';
import { RealTimeAnalyticsWebStack } from './real-time-analytics-web-stack';
export class CdkParentStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
new RealTimeAnalyticsPocStack(this, 'RealTimeAnalyticsPocStack');
new RealTimeAnalyticsWebStack(this, 'RealTimeAnalyticsWebStack');
}
}
```
**Code for testing**
```
session={{date.now('YYYYMMDD')}}|sequence={{date.now('x')}}|reception={{date.now('x')}}|instrument={{random.number(9)}}|l={{random.number(20)}}|price_0={{random.number({"min":10000, "max":30000})}}|price_1={{random.number({"min":10000, "max":30000})}}|price_2={{random.number({"min":10000, "max":30000})}}|price_3={{random.number({"min":10000, "max":30000})}}|price_4={{random.number({"min":10000, "max":30000})}}|price_5={{random.number({"min":10000, "max":30000})}}|price_6={{random.number({"min":10000, "max":30000})}}|price_7={{random.number({"min":10000, "max":30000})}}|price_8={{random.number({"min":10000, "max":30000})}}|
```
**Testing API Gateway**
On the API Gateway console, test API Gateway by using the `GET` method.
![\[API Gateway Console with GET chosen under OPTIONS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0ac29a11-1362-4084-92ed-6b85205763ca/images/452e5b8f-6d61-401d-8484-e5a436cb6f1b.png)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/0ac29a11-1362-4084-92ed-6b85205763ca/attachments/attachment.zip)
# Automate deployment of nested applications using AWS SAM
*Dr. Rahul Sharad Gaikwad, Ishwar Chauthaiwale, Dmitry Gulin, and Tabby Ward, Amazon Web Services*
## Summary
On Amazon Web Services (AWS), AWS Serverless Application Model (AWS SAM) is an open-source framework that provides shorthand syntax to express functions, APIs, databases, and event source mappings. With just a few lines for each resource, you can define the application you want and model it by using YAML. During deployment, SAM transforms and expands the SAM syntax into AWS CloudFormation syntax that you can use to build serverless applications faster.
AWS SAM simplifies the development, deployment, and management of serverless applications on the AWS platform. It provides a standardized framework, faster deployment, local testing capabilities, resource management, seamless Integration with Development Tools, and a supportive community. These features make it a valuable tool for building serverless applications efficiently and effectively.
This pattern uses AWS SAM templates to automate the deployment of nested applications. A nested application is an application within another application. Parent applications call their child applications. These are loosely coupled components of a serverless architecture.
Using nested applications, you can rapidly build highly sophisticated serverless architectures by reusing services or components that are independently authored and maintained but are composed using AWS SAM and the Serverless Application Repository. Nested applications help you to build applications that are more powerful, avoid duplicated work, and ensure consistency and best practices across your teams and organizations. To demonstrate nested applications, the pattern deploys an [example AWS serverless shopping cart application](https://github.com/aws-samples/aws-sam-nested-stack-sample).
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ An existing virtual private cloud (VPC) and subnets
+ An integrated development environment such as Visual Studio Code (for more information, see [Tools to Build on AWS](https://aws.amazon.com/getting-started/tools-sdks/#IDE_and_IDE_Toolkits))
+ Python wheel library installed using pip install wheel, if it’s not already installed
**Limitations **
+ The maximum number of applications that can be nested in a serverless application is 200.
+ The maximum number of parameters for a nested application can have 60.
**Product versions**
+ This solution is built on AWS SAM command line interface (AWS SAM CLI) version 1.21.1, but this architecture should work with later AWS SAM CLI versions.
## Architecture
**Target technology stack **
+ Amazon API Gateway
+ AWS SAM
+ Amazon Cognito
+ Amazon DynamoDB
+ AWS Lambda
+ Amazon Simple Queue Service (Amazon SQS) queue
**Target architecture**
The following diagram shows how user requests are made to the shopping services by calling APIs. The user's request, including all necessary information, is sent to Amazon API Gateway and the Amazon Cognito authorizer, which performs authentication and authorization mechanisms for the APIs.
When an item is added, deleted, or updated in DynamoDB, an event is put onto DynamoDB Streams, which in turn initiates a Lambda function. To avoid immediate deletion of old items as part of a synchronous workflow, messages are put onto an SQS queue, which initiates a worker function to delete the messages.
![\[POST and PUT operations from API Gateway to Lambda functions to DynamoDB and Product Service.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/218adecc-b5b8-4193-9012-b5d584e2e128/images/5b454bae-5fd4-405d-a37d-6bafc3fcf889.png)
In this solution setup, AWS SAM CLI serves as the interface for AWS CloudFormation stacks. AWS SAM templates automatically deploy nested applications. The parent SAM template calls the child templates, and the parent CloudFormation stack deploys the child stacks. Each child stack builds the AWS resources that are defined in the AWS SAM CloudFormation templates.
![\[Four-step process using AWS SAM CLI with a parent and three child CloudFormation stacks.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/218adecc-b5b8-4193-9012-b5d584e2e128/images/5828026e-72ad-4a3f-a5f2-bffac0f13e42.png)
1. Build and deploy the stacks.
1. The Auth CloudFormation stack contains Amazon Cognito.
1. The Product CloudFormation stack contains an Lambda function and Amazon API Gateway
1. The Shopping CloudFormation stack contains a Lambda function, Amazon API Gateway, the SQS queue, and the Amazon DynamoDB database.
## Tools
**Tools**
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.
+ [Amazon Cognito](https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html) provides authentication, authorization, and user management for web and mobile apps.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [AWS Serverless Application Model (AWS SAM)](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) is an open-source framework that helps you build serverless applications in the AWS Cloud.
+ [Amazon Simple Queue Service (Amazon SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
**Code **
The code for this pattern is available in the GitHub [AWS SAM Nested Stack Sample](https://github.com/aws-samples/aws-sam-nested-stack-sample) repository.
## Epics
### Install AWS SAM CLI
| Task | Description | Skills required |
| --- | --- | --- |
| Install AWS SAM CLI. | To install AWS SAM CLI, see the instructions in the [AWS SAM documentation](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html). | DevOps engineer |
| Set up AWS credentials. | To set AWS credentials so that the AWS SAM CLI can make calls to AWS services on your behalf, run the `aws configure` command and follow the prompts.
$aws configure AWS Access Key ID [None]: AWS Secret Access Key [None]: your_secret_access_key Default region name [None]: Default output format [None]:
For more information on setting up your credentials, see [Authentication and access credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html). | DevOps engineer |
### Initialize the AWS SAM project
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the AWS SAM code repository. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-deployment-of-nested-applications-using-aws-sam.html) | DevOps engineer |
| Deploy templates to initialize the project. | To initialize the project, run the `SAM init` command. When prompted to choose a template source, choose `Custom Template Location`. | DevOps engineer |
### Compile and build the SAM template code
| Task | Description | Skills required |
| --- | --- | --- |
| Review the AWS SAM application templates. | Review the templates for the nested applications. This example uses the following nested application templates:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-deployment-of-nested-applications-using-aws-sam.html) | DevOps engineer |
| Review the parent template. | Review the template that will invoke the nested application templates. In this example, the parent template is `template.yml`. All separate applications are nested in the single parent template `template.yml`. | DevOps engineer |
| Compile and build the AWS SAM template code. | Using the AWS SAM CLI, run the following command.
sam build
| DevOps engineer |
### Deploy the AWS SAM template
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the applications. | To launch the SAM template code that creates the nested application CloudFormation stacks and deploys code in the AWS environment, run the following command.
sam deploy --guided --stack-name shopping-cart-nested-stack --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND
The command will prompt with a few questions. Answer all questions with `y`. | DevOps engineer |
### Verify the deployment
| Task | Description | Skills required |
| --- | --- | --- |
| Verify the stacks. | To review the AWS CloudFormation stacks and AWS resources that were defined in the AWS SAM templates, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-deployment-of-nested-applications-using-aws-sam.html) | DevOps engineer |
## Related resources
**References**
+ [AWS Serverless Application Model (AWS SAM)](https://aws.amazon.com/serverless/sam/#:~:text=The%20AWS%20Serverless%20Application%20Model,and%20model%20it%20using%20YAML.)
+ [AWS SAM on GitHub](https://github.com/aws/serverless-application-model)
+ [Serverless Shopping Cart Microservice](https://github.com/aws-samples/aws-serverless-shopping-cart) (AWS example application)
**Tutorials and videos **
+ [Build a Serverless App](https://youtu.be/Hv3YrP8G4ag)
+ [AWS Online Tech Talks: Serverless Application Building and Deployments with AWS SAM](https://youtu.be/1NU7vyJw9LU)
## Additional information
After all the code is in place, the example has the following directory structure:
+ [sam\$1stacks](https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html) – This folder contains the `shared.py` layer. A layer is a file archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in a deployment package.
+ *product-mock-service* – This folder contains all product-related Lambda functions and files.
+ *shopping-cart-service* – This folder contains all shopping-related Lambda functions and files.
# Implement SaaS tenant isolation for Amazon S3 by using an AWS Lambda token vending machine
*Tabby Ward, Thomas Davis, and Sravan Periyathambi, Amazon Web Services*
## Summary
Multitenant SaaS applications must implement systems to ensure that tenant isolation is maintained. When you store tenant data on the same AWS resource—such as when multiple tenants store data in the same Amazon Simple Storage Service (Amazon S3) bucket—you must ensure that cross-tenant access cannot occur. Token vending machines (TVMs) are one way to provide tenant data isolation. These machines provide a mechanism for obtaining tokens while abstracting the complexity of how these tokens are generated. Developers can use a TVM without having detailed knowledge of how it produces tokens.
This pattern implements a TVM by using AWS Lambda. The TVM generates a token that consists of temporary security token service (STS) credentials that limit access to a single SaaS tenant's data in an S3 bucket.
TVMs, and the code that’s provided with this pattern, are typically used with claims that are derived from JSON Web Tokens (JWTs) to associate requests for AWS resources with a tenant-scoped AWS Identity and Access Management (IAM) policy. You can use the code in this pattern as a basis to implement a SaaS application that generates scoped, temporary STS credentials based on the claims provided in a JWT token.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ AWS Command Line Interface (AWS CLI) [version 1.19.0 or later](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html), installed and configured on macOS, Linux, or Windows. Alternatively, you can use AWS CLI [version 2.1 or later](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
**Limitations **
+ This code runs in Java and doesn’t currently support other programming languages.
+ The sample application doesn’t include AWS cross-Region or disaster recovery (DR) support.
+ This pattern demonstrates how a Lambda TVM for a SaaS application can provide scoped tenant access. This pattern is not intended to be used in production environments without additional security testing as a part of your specific application or use case.
## Architecture
**Target technology stack **
+ AWS Lambda
+ Amazon S3
+ IAM
+ AWS Security Token Service (AWS STS)
**Target architecture **
![\[Generating a token to gain temporary STS credentials to access data in an S3 bucket.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/97a34c8e-d04e-40b6-acbf-1baa176d22a9/images/14d0508a-703b-4229-85e6-c5094de7fe01.png)
## Tools
**AWS services**
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [AWS Security Token Service (AWS STS)](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) helps you request temporary, limited-privilege credentials for users.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Code**
The source code for this pattern is available as an attachment and includes the following files:
+ `s3UploadSample.jar` provides the source code for a Lambda function that uploads a JSON document to an S3 bucket.
+ `tvm-layer.zip` provides a reusable Java library that supplies a token (STS temporary credentials) for the Lambda function to access the S3 bucket and upload the JSON document.
+ `token-vending-machine-sample-app.zip` provides the source code used to create these artifacts and compilation instructions.
To use these files, follow the instructions in the next section.
## Epics
### Determine variable values
| Task | Description | Skills required |
| --- | --- | --- |
| Determine variable values. | The implementation of this pattern includes several variable names that must be used consistently. Determine the values that should be used for each variable, and provide that value when requested in subsequent steps.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-saas-tenant-isolation-for-amazon-s3-by-using-an-aws-lambda-token-vending-machine.html) | Cloud administrator |
### Create an S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket for the sample application. | Use the following AWS CLI command to create an S3 bucket. Provide the ``** **value in the code snippet:
aws s3api create-bucket --bucket
The Lambda sample application uploads JSON files to this bucket. | Cloud administrator |
### Create the IAM TVM role and policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create a TVM role. | Use one of the following AWS CLI commands to create an IAM role. Provide the ``** **value in the command.For macOS or Linux shells:
The Lambda sample application assumes this role when the application is invoked. The capability to assume the application role with a scoped policy gives the code broader permissions to access the S3 bucket. | Cloud administrator |
| Create an inline TVM role policy. | Use one of the following AWS CLI commands to create an IAM policy. Provide the ``,** **``, and `` values in the command.For macOS or Linux shells:
This policy is attached to the TVM role. It gives the code the capability to assume the application role, which has broader permissions to access the S3 bucket. | Cloud administrator |
| Attach the managed Lambda policy. | Use the following AWS CLI command to attach the `AWSLambdaBasicExecutionRole` IAM policy. Provide the `` value in the command:
aws iam attach-role-policy \ --role-name \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
For the Windows command line:
aws iam attach-role-policy ^ --role-name ^ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
This managed policy is attached to the TVM role to permit Lambda to send logs to Amazon CloudWatch. | Cloud administrator |
### Create the IAM application role and policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create the application role. | Use one of the following AWS CLI commands to create an IAM role. Provide the ``, ``, and `` values in the command.For macOS or Linux shells:
The Lambda sample application assumes this role with a scoped policy to get tenant-based access to an S3 bucket. | Cloud administrator |
| Create an inline application role policy. | Use one of the following AWS CLI mmands to create an IAM policy. Provide the `` and ``** **values in the command.For macOS or Linux shells:
This policy is attached to the application role. It provides broad access to objects in the S3 bucket. When the sample application assumes the role, these permissions are scoped to a specific tenant with the TVM's dynamically generated policy. | Cloud administrator |
### Create the Lambda sample application with TVM
| Task | Description | Skills required |
| --- | --- | --- |
| Download the compiled source files. | Download the `s3UploadSample.jar` and `tvm-layer.zip`** **files, which are included as attachments. The source code used to create these artifacts and compilation instuctions are provided in `token-vending-machine-sample-app.zip`. | Cloud administrator |
| Create the Lambda layer. | Use the following AWS CLI command to create a Lambda layer, which makes the TVM accessible to Lambda. If you aren’t running this command from the location where you downloaded` tvm-layer.zip`, provide the correct path to `tvm-layer.zip` in the `--zip-file` parameter.
This command creates a Lambda layer that contains the reusable TVM library. | Cloud administrator, App developer |
| Create the Lambda function. | Use the following AWS CLI command to create a Lambda function. Provide the ``, ``, ``, ``, ``, and `` values in the command. If you aren’t running this command from the location where you downloaded `s3UploadSample.jar`, provide the correct path to `s3UploadSample.jar` in the `--zip-file` parameter.
This command creates a Lambda function with the sample application code and the TVM layer attached. It also sets two environment variables: `S3_BUCKET` and `ROLE`. The sample application uses these variables to determine the role to assume and the S3 bucket to upload JSON documents to. | Cloud administrator, App developer |
### Test the sample application and TVM
| Task | Description | Skills required |
| --- | --- | --- |
| Invoke the Lambda sample application. | Use one of the following AWS CLI commands to start the Lambda sample application with its expected payload. Provide the `` and `` values in the command.For macOS and Linux shells:
This command calls the Lambda function and returns the result in a `response.json` document. On many Unix-based systems, you can change `response.json` to `/dev/stdout` to output the results directly to your shell without creating another file. Changing the `` value in subsequent invocations of this Lambda function alters the location of the JSON document and the permissions the token provides. | Cloud administrator, App developer |
| View the S3 bucket to see created objects. | Browse to the S3 bucket ( ``) that you created earlier. This bucket contains an S3 object prefix with the value of ``. Under that prefix, you will find a JSON document named with a UUID. Invoking the sample application multiple times adds more JSON documents. | Cloud administrator |
| View the logs for the sample application in CloudWatch Logs. | View the logs that are associated with the Lambda function named `` in CloudWatch Logs. For instructions, see [Sending Lambda function logs to CloudWatch Logs](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html) in the Lambda documentation. You can view the tenant-scoped policy generated by the TVM in these logs. This tenant-scoped policy gives permissions for the sample application to the Amazon S3 **PutObject**, **GetObject**, **DeleteObject**, and **ListBucket** APIs, but only for the object prefix that’s associated with ``. In subsequent invocations of the sample application, if you change ``, the TVM updates the scoped policy to correspond to the tenant provided in the invocation payload. This dynamically generated policy shows how tenant-scoped access can be maintained with a TVM in SaaS applications. The TVM functionality is provided in a Lambda layer so that it can be attached to other Lambda functions used by an application without having to replicate the code.For an illustration of the dynamically generated policy, see the [Additional information](#implement-saas-tenant-isolation-for-amazon-s3-by-using-an-aws-lambda-token-vending-machine-additional) section. | Cloud administrator |
## Related resources
+ [Isolating Tenants with Dynamically Generated IAM Policies](https://aws.amazon.com/blogs/apn/isolating-saas-tenants-with-dynamically-generated-iam-policies/) (blog post)
+ [Applying Dynamically Generated Isolation Policies in SaaS Environments](https://aws.amazon.com/blogs/apn/applying-dynamically-generated-isolation-policies-in-saas-environments/) (blog post)
+ [SaaS on AWS](https://aws.amazon.com/saas/)
## Additional information
The following log shows the dynamically generated policy produced by the TVM code in this pattern. In this screenshot, the `` is `DOC-EXAMPLE-BUCKET` and the `` is `test-tenant-1`. The STS credentials returned by this scoped policy are unable to perform any actions on objects in the S3 bucket except for objects that are associated with the object key prefix `test-tenant-1`.
![\[Log showing a dynamically generated policy produced by the TVM code.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/97a34c8e-d04e-40b6-acbf-1baa176d22a9/images/d4776ebe-fb8f-41ac-b8c5-b4f97a821c8c.png)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/97a34c8e-d04e-40b6-acbf-1baa176d22a9/attachments/attachment.zip)
# Implement the serverless saga pattern by using AWS Step Functions
*Tabby Ward, Joe Kern, and Rohan Mehta, Amazon Web Services*
## Summary
In a microservices architecture, the main goal is to build decoupled and independent components to promote agility, flexibility, and faster time to market for your applications. As a result of decoupling, each microservice component has its own data persistence layer. In a distributed architecture, business transactions can span multiple microservices. Because these microservices cannot use a single atomicity, consistency, isolation, durability (ACID) transaction, you might end up with partial transactions. In this case, some control logic is needed to undo the transactions that have already been processed. The distributed saga pattern is typically used for this purpose.
The saga pattern is a failure management pattern that helps establish consistency in distributed applications and coordinates transactions between multiple microservices to maintain data consistency. When you use the saga pattern, every service that performs a transaction publishes an event that triggers subsequent services to perform the next transaction in the chain. This continues until the last transaction in the chain is complete. If a business transaction fails, saga orchestrates a series of compensating transactions that undo the changes that were made by the preceding transactions.
This pattern demonstrates how to automate the setup and deployment of a sample application (which handles travel reservations) with serverless technologies such as AWS Step Functions, AWS Lambda, and Amazon DynamoDB. The sample application also uses Amazon API Gateway and Amazon Simple Notification Service (Amazon SNS) to implement a saga execution coordinator. The pattern can be deployed with an infrastructure as code (IaC) framework such as the AWS Cloud Development Kit (AWS CDK), the AWS Serverless Application Model (AWS SAM), or Terraform.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ Permissions to create an AWS CloudFormation stack. For more information, see [Controlling access](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-template.html) in the CloudFormation documentation.
+ IaC framework of your choice (AWS CDK, AWS SAM, or Terraform) configured with your AWS account so that you can use the framework CLI to deploy the application.
+ NodeJS, used to build the application and run it locally.
+ A code editor of your choice (such as Visual Studio Code, Sublime, or Atom).
**Product versions**
+ [NodeJS version 14](https://nodejs.org/en/download/)
+ [AWS CDK version 2.37.1](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install)
+ [AWS SAM version 1.71.0](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html)
+ [Terraform version 1.3.7](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
**Limitations**
Event sourcing is a natural way to implement the saga orchestration pattern in a microservices architecture where all components are loosely coupled and don’t have direct knowledge of one another. If your transaction involves a small number of steps (three to five), the saga pattern might be a great fit. However complexity increases with the number of microservices and the number of steps.
Testing and debugging can become difficult when you’re using this design, because you have to have all services running in order to simulate the transaction pattern.
## Architecture
**Target architecture **
The proposed architecture uses AWS Step Functions to build a saga pattern to book flights, book car rentals, and process payments for a vacation.
The following workflow diagram illustrates the typical flow of the travel reservation system. The workflow consists of reserving air travel ("ReserveFlight"), reserving a car ("ReserveCarRental"), processing payments ("ProcessPayment"), confirming flight reservations ("ConfirmFlight"), and confirming car rentals ("ConfirmCarRental") followed by a success notification when these steps are complete. However, if the system encounters any errors in running any of these transactions, it starts to fail backward. For example, an error with payment processing ("ProcessPayment") triggers a refund ("RefundPayment"), which then triggers a cancellation of the rental car and flight ("CancelRentalReservation" and "CancelFlightReservation"), which ends the entire transaction with a failure message.
This pattern deploys separate Lambda functions for each task that is highlighted in the diagram as well as three DynamoDB tables for flights, car rentals, and payments. Each Lambda function creates, updates, or deletes the rows in the respective DynamoDB tables, depending on whether a transaction is confirmed or rolled back. The pattern uses Amazon SNS to send text (SMS) messages to subscribers, notifying them of failed or successful transactions.
![\[Workflow for a travel reservation system based on the saga pattern.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fec0789c-d9b1-4d80-b179-dd9a7ecbec07/images/daad3e8e-6e6b-41c2-95c1-ca79d53ead64.png)
**Automation and scale**
You can create the configuration for this architecture by using one of the IaC frameworks. Use one of the following links for your preferred IaC.
+ [Deploy with AWS CDK](https://serverlessland.com/workflows/saga-pattern-cdk)
+ [Deploy with AWS SAM](https://serverlessland.com/workflows/saga-pattern-sam)
+ [Deploy with Terraform](https://serverlessland.com/workflows/saga-pattern-tf)
## Tools
**AWS services**
+ [AWS Step Functions](https://aws.amazon.com/step-functions/) is a serverless orchestration service that lets you combine AWS Lambda functions and other AWS services to build business-critical applications. Through the Step Functions graphical console, you see your application’s workflow as a series of event-driven steps.
+ [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. You can use DynamoDB to create a database table that can store and retrieve any amount of data, and serve any level of request traffic.
+ [AWS Lambda](https://aws.amazon.com/lambda/) is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second.
+ [Amazon API Gateway](https://aws.amazon.com/api-gateway/) is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale.
+ [Amazon Simple Notification Service (Amazon SNS)](https://aws.amazon.com/sns/) is a managed service that provides message delivery from publishers to subscribers.
+ [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/) is a software development framework for defining your cloud application resources by using familiar programming languages such as TypeScript, JavaScript, Python, Java, and C\$1/.Net.
+ [AWS Serverless Application Model (AWS SAM)](https://aws.amazon.com/serverless/sam/) is an open-source framework for building serverless applications. It provides shorthand syntax to express functions, APIs, databases, and event source mappings.
**Code**
The code for a sample application that demonstrates the saga pattern, including the IaC template (AWS CDK, AWS SAM, or Terraform), the Lambda functions, and the DynamoDB tables can be found in the following links. Follow the instructions in the first epic to install these.
+ [Deploy with AWS CDK](https://serverlessland.com/workflows/saga-pattern-cdk)
+ [Deploy with AWS SAM](https://serverlessland.com/workflows/saga-pattern-sam)
+ [Deploy with Terraform](https://serverlessland.com/workflows/saga-pattern-tf)
## Epics
### Install packages, compile, and build
| Task | Description | Skills required |
| --- | --- | --- |
| Install the NPM packages. | Create a new directory, navigate to that directory in a terminal, and clone the GitHub repository of your choice from the *Code* section earlier in this pattern.In the root folder that has the `package.json` file, run the following command to download and install all Node Package Manager (NPM) packages:
npm install
| Developer, Cloud architect |
| Compile scripts. | In the root folder, run the following command to instruct the TypeScript transpiler to create all necessary JavaScript files:
npm run build
| Developer, Cloud architect |
| Watch for changes and recompile. | In the root folder, run the following command in a separate terminal window to watch for code changes, and compile the code when it detects a change:
npm run watch
| Developer, Cloud architect |
| Run unit tests (AWS CDK only). | If you’re using the AWS CDK, in the root folder, run the following command to perform the Jest unit tests:
npm run test
| Developer, Cloud architect |
### Deploy resources to the target AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the demo stack to AWS. | The application is AWS Region-agnostic. If you use a profile, you must declare the Region explicitly in either the [AWS Command Line Interface (AWS CLI) profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) or through [AWS CLI environment variables.](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)In the root folder, run the following command to create a deployment assembly and to deploy it to the default AWS account and Region.AWS CDK:
cdk bootstrap cdk deploy
AWS SAM:
sam build sam deploy --guided
Terraform:
terraform init terraform apply
This step might take several minutes to complete. This command uses the default credentials that were configured for the AWS CLI.Note the API Gateway URL that is displayed on the console after deployment is complete. You will need this information to test the saga execution flow. | Developer, Cloud architect |
| Compare the deployed stack with the current state. | In the root folder, run the following command to compare the deployed stack with the current state after making changes to the source code:AWS CDK:
cdk diff
AWS SAM:
sam deploy
Terraform:
terraform plan
| Developer, Cloud architect |
### Test the execution flow
| Task | Description | Skills required |
| --- | --- | --- |
| Test the saga execution flow. | Navigate to the API Gateway URL that you noted in the earlier step, when you deployed the stack. This URL triggers the state machine to start. For more information about how to manipulate the flow of the state machine by passing different URL parameters, see the [Additional information](#implement-the-serverless-saga-pattern-by-using-aws-step-functions-additional) section.To view the results, sign in to the AWS Management Console and navigate to the Step Functions console. Here, you can see every step of the saga state machine. You can also view the DynamoDB table to see the records inserted, updated, or deleted. If you refresh the screen frequently, you can watch the transaction status change from `pending` to `confirmed`. You can subscribe to the SNS topic by updating the code in the `stateMachine.ts` file with your cell phone number to receive SMS messages upon successful or failed reservations. For more information, see *Amazon SNS* in the [Additional information](#implement-the-serverless-saga-pattern-by-using-aws-step-functions-additional) section. | Developer, Cloud architect |
### Clean up
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up resources. | To clean up the resources deployed for this application, you can use one of the following commands.AWS CDK:
cdk destroy
AWS SAM:
sam delete
Terraform:
terraform destroy
| App developer, Cloud architect |
## Related resources
**Technical papers**
+ [Implementing Microservices on AWS](https://docs.aws.amazon.com/pdfs/whitepapers/latest/microservices-on-aws/microservices-on-aws.pdf)
+ [Serverless Application Lens](https://docs.aws.amazon.com/wellarchitected/latest/serverless-applications-lens/welcome.html)
**AWS service documentation**
+ [Getting started with the AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html)
+ [Getting started with AWS SAM](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started.html)
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/)
+ [Amazon DynamoDB](https://docs.aws.amazon.com/dynamodb/)
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/)
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/)
+ [Amazon SNS](https://docs.aws.amazon.com/sns/)
**Tutorials**
+ [Hands-on Workshops for Serverless Computing](https://aws.amazon.com/serverless-workshops/)
## Additional information
**Code**
For testing purposes, this pattern deploys API Gateway and a test Lambda function that triggers the Step Functions state machine. With Step Functions, you can control the functionality of the travel reservation system by passing a `run_type` parameter to mimic failures in "ReserveFlight," "ReserveCarRental," "ProcessPayment," "ConfirmFlight," and "ConfirmCarRental."
The `saga` Lambda function (`sagaLambda.ts`) takes input from the query parameters in the API Gateway URL, creates the following JSON object, and passes it to Step Functions for execution:
```
let input = {
"trip_id": tripID, // value taken from query parameter, default is AWS request ID
"depart_city": "Detroit",
"depart_time": "2021-07-07T06:00:00.000Z",
"arrive_city": "Frankfurt",
"arrive_time": "2021-07-09T08:00:00.000Z",
"rental": "BMW",
"rental_from": "2021-07-09T00:00:00.000Z",
"rental_to": "2021-07-17T00:00:00.000Z",
"run_type": runType // value taken from query parameter, default is "success"
};
```
You can experiment with different flows of the Step Functions state machine by passing the following URL parameters:
+ **Successful Execution** ─ https://\$1api gateway url\$1
+ **Reserve Flight Fail** ─ https://\$1api gateway url\$1?**runType=failFlightsReservation**
+ **Confirm Flight Fail** ─ https://\$1api gateway url\$1?**runType=failFlightsConfirmation**
+ **Reserve Car Rental Fail** ─ https://\$1api gateway url\$1?**runType=failCarRentalReservation**
+ **Confirm Car Rental Fail** ─ https://\$1api gateway url\$1?**runType=failCarRentalConfirmation**
+ **Process Payment Fail** ─ https://\$1api gateway url\$1?**runType=failPayment**
+ **Pass a Trip ID** ─ https://\$1api gateway url\$1?**tripID=**\$1by default, trip ID will be the AWS request ID\$1
**IaC templates**
The linked repositories include IaC templates that you can use to create the entire sample travel reservation application.
+ [Deploy with AWS CDK](https://serverlessland.com/workflows/saga-pattern-cdk)
+ [Deploy with AWS SAM](https://serverlessland.com/workflows/saga-pattern-sam)
+ [Deploy with Terraform](https://serverlessland.com/workflows/saga-pattern-tf)
**DynamoDB tables**
Here are the data models for the flights, car rentals, and payments tables.
```
Flight Data Model:
var params = {
TableName: process.env.TABLE_NAME,
Item: {
'pk' : {S: event.trip_id},
'sk' : {S: flightReservationID},
'trip_id' : {S: event.trip_id},
'id': {S: flightReservationID},
'depart_city' : {S: event.depart_city},
'depart_time': {S: event.depart_time},
'arrive_city': {S: event.arrive_city},
'arrive_time': {S: event.arrive_time},
'transaction_status': {S: 'pending'}
}
};
Car Rental Data Model:
var params = {
TableName: process.env.TABLE_NAME,
Item: {
'pk' : {S: event.trip_id},
'sk' : {S: carRentalReservationID},
'trip_id' : {S: event.trip_id},
'id': {S: carRentalReservationID},
'rental': {S: event.rental},
'rental_from': {S: event.rental_from},
'rental_to': {S: event.rental_to},
'transaction_status': {S: 'pending'}
}
};
Payment Data Model:
var params = {
TableName: process.env.TABLE_NAME,
Item: {
'pk' : {S: event.trip_id},
'sk' : {S: paymentID},
'trip_id' : {S: event.trip_id},
'id': {S: paymentID},
'amount': {S: "750.00"}, // hard coded for simplicity as implementing any monetary transaction functionality is beyond the scope of this pattern
'currency': {S: "USD"},
'transaction_status': {S: "confirmed"}
}
};
```
**Lambda functions**
The following functions will be created to support the state machine flow and execution in Step Functions:
+ **Reserve Flights**: Inserts a record into the DynamoDB Flights table with a `transaction_status` of `pending`, to book a flight.
+ **Confirm Flight**: Updates the record in the DynamoDB Flights table, to set `transaction_status` to `confirmed`, to confirm the flight.
+ **Cancel Flights Reservation**: Deletes the record from the DynamoDB Flights table, to cancel the pending flight.
+ **Reserve Car Rentals**: Inserts a record into the DynamoDB CarRentals table with a `transaction_status` of `pending`, to book a car rental.
+ **Confirm Car Rentals**: Updates the record in the DynamoDB CarRentals table, to set `transaction_status` to `confirmed`, to confirm the car rental.
+ **Cancel Car Rentals Reservation:** Deletes the record from the DynamoDB CarRentals table, to cancel the pending car rental.
+ **Process Payment**: Inserts a record into the DynamoDB Payment table for the payment.
+ **Cancel Payment**: Deletes the record from the DynamoDB Payments table for the payment.
**Amazon SNS**
The sample application creates the following topic and subscription for sending SMS messages and notifying the customer about successful or failed reservations. If you want to receive text messages while testing the sample application, update the SMS subscription with your valid phone number in the state machine definition file.
AWS CDK snippet (add the phone number in the second line of the following code):
```
const topic = new sns.Topic(this, 'Topic');
topic.addSubscription(new subscriptions.SmsSubscription('+11111111111'));
const snsNotificationFailure = new tasks.SnsPublish(this ,'SendingSMSFailure', {
topic:topic,
integrationPattern: sfn.IntegrationPattern.REQUEST_RESPONSE,
message: sfn.TaskInput.fromText('Your Travel Reservation Failed'),
});
const snsNotificationSuccess = new tasks.SnsPublish(this ,'SendingSMSSuccess', {
topic:topic,
integrationPattern: sfn.IntegrationPattern.REQUEST_RESPONSE,
message: sfn.TaskInput.fromText('Your Travel Reservation is Successful'),
});
```
AWS SAM snippet (replace the `+1111111111` strings with your valid phone number):
```
StateMachineTopic11111111111:
Type: 'AWS::SNS::Subscription'
Properties:
Protocol: sms
TopicArn:
Ref: StateMachineTopic
Endpoint: '+11111111111'
Metadata:
'aws:sam:path': SamServerlessSagaStack/StateMachine/Topic/+11111111111/Resource
```
Terraform snippet (replace the `+111111111` string with your valid phone number):
```
resource "aws_sns_topic_subscription" "sms-target" {
topic_arn = aws_sns_topic.topic.arn
protocol = "sms"
endpoint = "+11111111111"
}
```
**Successful reservations**
The following flow illustrates a successful reservation with "ReserveFlight," "ReserveCarRental," and "ProcessPayment" followed by "ConfirmFlight" and "ConfirmCarRental." The customer is notified about the successful booking through SMS messages that are sent to the subscriber of the SNS topic.
![\[Example of a successful reservation implemented by Step Functions by using the saga pattern.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fec0789c-d9b1-4d80-b179-dd9a7ecbec07/images/f58c894e-7721-4bc7-8f7d-29f23faa5dc1.png)
**Failed reservations**
This flow is an example of failure in the saga pattern. If, after booking flights and car rentals, "ProcessPayment" fails, steps are canceled in reverse order. The reservations are released, and the customer is notified of the failure through SMS messages that are sent to the subscriber of the SNS topic.
![\[Example of a failed reservation implemented by Step Functions by using the saga pattern.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/fec0789c-d9b1-4d80-b179-dd9a7ecbec07/images/7c64d326-be27-42c3-b03f-d677efedb9a7.png)
# Manage on-premises container applications by setting up Amazon ECS Anywhere with the AWS CDK
*Dr. Rahul Sharad Gaikwad, Amazon Web Services*
## Summary
[Amazon ECS Anywhere](https://aws.amazon.com/ecs/anywhere/) is an extension of the Amazon Elastic Container Service (Amazon ECS). You can use ECS Anywhere to deploy native Amazon ECS tasks in an on-premises or customer-managed environment. This feature helps reduce costs and mitigate complex local container orchestration and operations. You can use ECS Anywhere to deploy and run container applications in both on-premises and cloud environments. It removes the need for your team to learn multiple domains and skill sets, or to manage complex software on their own.
This pattern demonstrates the steps to set up ECS Anywhere by using [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/) stacks.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ AWS Command Line Interface (AWS CLI), installed and configured. (See [Installing, updating, and uninstalling the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) in the AWS CLI documentation.)
+ AWS CDK Toolkit, installed and configured. (See [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html) in the AWS CDK documentation, and follow the instructions to install version 2 globally.)
+ Node package manager (npm), installed and configured for the AWS CDK in TypeScript. (See [Downloading and installing Node.js and npm ](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm)in the npm documentation.)
**Limitations **
+ For limitations and considerations, see [External instances (Amazon ECS Anywhere)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html#ecs-anywhere-considerations) in the Amazon ECS documentation.
**Product versions**
+ AWS CDK Toolkit version 2
+ npm version 7.20.3 or later
+ Node.js version 16.6.1 or later
## Architecture
**Target technology stack **
+ AWS CloudFormation
+ AWS CDK
+ Amazon ECS Anywhere
+ AWS Identity and Access Management (IAM)
**Target architecture **
The following diagram illustrates a high-level system architecture of ECS Anywhere setup using the AWS CDK with TypeScript, as implemented by this pattern.
1. When you deploy the AWS CDK stack, it creates a CloudFormation stack on AWS.
1. The CloudFormation stack provisions an Amazon ECS cluster and related AWS resources.
1. To register an external instance with an Amazon ECS cluster, you must install AWS Systems Manager Agent (SSM Agent) on your virtual machine (VM) and register the VM as an AWS Systems Manager managed instance.
1. You must also install the Amazon ECS container agent and Docker on your VM to register it as an external instance with the Amazon ECS cluster.
1. When the external instance is registered and configured with the Amazon ECS cluster, it can run multiple containers on your VM, which is registered as an external instance.
![\[ECS Anywhere setup using the AWS CDK with TypeScript.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/3ed63c00-40e7-4831-bb9d-63049c3490aa/images/ff7dc774-830d-4b9f-8262-7314afe7a033.png)
**Automation and scale**
The [GitHub repository](https://github.com/aws-samples/amazon-ecs-anywhere-cdk-samples/) that is provided with this pattern uses the AWS CDK as an infrastructure as code (IaC) tool to create the configuration for this architecture. AWS CDK helps you orchestrate resources and set up ECS Anywhere.
## Tools
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/latest/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
**Code**
The source code for this pattern is available on GitHub, in the [Amazon ECS Anywhere CDK Samples](https://github.com/aws-samples/amazon-ecs-anywhere-cdk-samples) repository. To clone and use the repository, follow the instructions in the next section.
## Epics
### Verify AWS CDK configuration
| Task | Description | Skills required |
| --- | --- | --- |
| Verify the AWS CDK version. | Verify the version of the AWS CDK Toolkit by running the following command:
cdk --version
This pattern requires AWS CDK version 2. If you have an earlier version of the AWS CDK, follow the instructions in the [AWS CDK documentation](https://docs.aws.amazon.com/cdk/v2/guide/cli.html) to update it. | DevOps engineer |
| Set up AWS credentials. | To set up credentials, run the `aws configure` command and follow the prompts:
$aws configure AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [None]: Default output format [None]:
| DevOps engineer |
### Bootstrap the AWS CDK environment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the AWS CDK code repository. | Clone the GitHub code repository for this pattern by using the command:
| DevOps engineer |
| Bootstrap the environment. | To deploy the AWS CloudFormation template to the account and AWS Region that you want to use, run the following command:
cdk bootstrap /
For more information, see [Bootstrapping](https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html) in the AWS CDK documentation. | DevOps engineer |
### Build and deploy the project
| Task | Description | Skills required |
| --- | --- | --- |
| Install package dependencies and compile TypeScript files. | Install the package dependencies and compile the TypeScript files by running the following commands:
$cd amazon-ecs-anywhere-cdk-samples $npm install $npm fund
These commands install all the packages from the sample repository. If you get any errors about missing packages, use one of the following commands:
$npm ci
—or—
$npm install -g @aws-cdk/
For more information, see [npm ci](https://docs.npmjs.com/cli/v7/commands/npm-ci) and [npm install](https://docs.npmjs.com/cli/v7/commands/npm-install) in the npm documentation. | DevOps engineer |
| Build the project. | To build the project code, run the command:
npm run build
For more information about building and deploying the project, see [Your first AWS CDK app](https://docs.aws.amazon.com/cdk/latest/guide/hello_world.html#:~:text=the%20third%20parameter.-,Synthesize%20an%20AWS%20CloudFormation%20template,-Synthesize%20an%20AWS) in the AWS CDK documentation. | DevOps engineer |
| Deploy the project. | To deploy the project code, run the command:
cdk deploy
| DevOps engineer |
| Verify stack creation and output. | Open the AWS CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/),** **and choose the `EcsAnywhereStack` stack. The **Outputs** tab shows the commands to run on your external VM. | DevOps engineer |
### Set up an on-premises machine
| Task | Description | Skills required |
| --- | --- | --- |
| Set up your VM by using Vagrant. | For demonstration purposes, you can use [HashiCorp Vagrant](https://www.vagrantup.com/) to create a VM. Vagrant is an open-source utility for building and maintaining portable virtual software development environments. Create a Vagrant VM by running the `vagrant up` command from the root directory where Vagrantfile is placed. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/cli/up). | DevOps engineer |
| Register your VM as an external instance. | 1. Log in to the Vagrant VM by using the `vagrant ssh` command. For more information, see the [Vagrant documentation](https://www.vagrantup.com/docs/cli/ssh).2. Create an activation code and ID that you can use to register your VM with AWS Systems Manager and to activate your external instance. The output from this command includes `ActivationId` and `ActivationCode` values:
aws ssm create-activation --iam-role EcsAnywhereInstanceRole | tee ssm-activation.json
3. Export the activation ID and code values:
export ACTIVATION_ID= export ACTIVATION_CODE=
4. Download the installation script to your on-premises server or VM:
For more information about setting up and registering your VM, see [Registering an external instance to a cluster](https://docs.amazonaws.cn/en_us/AmazonECS/latest/developerguide/ecs-anywhere-registration.html) in the Amazon ECS documentation. | DevOps engineer |
| Verify the status of ECS Anywhere and the external VM. | To verify whether your virtual box is connected to the Amazon ECS control plane and running, use the following commands:
| DevOps engineer |
### Clean up
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up and delete resources. | After you walk through this pattern, you should remove the resources you created to avoid incurring any further charges. To clean up, run the command:
cdk destroy
| DevOps engineer |
## Related resources
+ [Amazon ECS Anywhere Documentation](https://aws.amazon.com/ecs/anywhere/)
+ [Amazon ECS Anywhere Demo](https://www.youtube.com/watch?v=-eud6yUXsJM)
+ [Amazon ECS Anywhere Workshop Samples](https://github.com/aws-samples/aws-ecs-anywhere-workshop-samples)
# Modernize ASP.NET Web Forms applications on AWS
*Vijai Anand Ramalingam and Sreelaxmi Pai, Amazon Web Services*
## Summary
This pattern describes the steps for modernizing a legacy, monolith ASP.NET Web Forms application by porting it to ASP.NET Core on AWS.
Porting ASP.NET Web Forms applications to ASP.NET Core helps you take advantage of the performance, cost savings, and robust ecosystem of Linux. However, it can be a significant manual effort. In this pattern, the legacy application is modernized incrementally by using a phased approach, and then containerized in the AWS Cloud.
Consider a legacy, monolith application for a shopping cart. Let’s assume that it was created as an ASP.NET Web Forms application and consists of .aspx pages with a code-behind (`aspx.cs`) file. The modernization process consists of these steps:
1. Break the monolith into microservices by using the appropriate decomposition patterns. For more information, see the guide [Decomposing monoliths into microservices](https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-decomposing-monoliths/) on the AWS Prescriptive Guidance website.
1. Port your legacy ASP.NET Web Forms (.NET Framework) application to ASP.NET Core in .NET 5 or later. In this pattern, you use Porting Assistant for .NET to scan your ASP.NET Web Forms application and identify incompatibilities with ASP.NET Core. This reduces the manual porting effort.
1. Redevelop the Web Forms UI layer by using React. This pattern doesn’t cover UI redevelopment. For instructions, see [Create a New React App](https://reactjs.org/docs/create-a-new-react-app.html) in the React documentation.
1. Redevelop the Web Forms code-behind file (business interface) as an ASP.NET Core web API. This pattern uses NDepend reports to help identify required files and dependencies.
1. Upgrade shared/common projects, such as Business Logic and Data Access, in your legacy application to .NET 5 or later by using Porting Assistant for .NET.
1. Add AWS services to complement your application. For example, you can use [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) to monitor, store, and access your application’s logs, and [AWS Systems Manager](https://aws.amazon.com/systems-manager/) to store your application settings.
1. Containerize the modernized ASP.NET Core application. This pattern creates a Docker file that targets Linux in Visual Studio and uses Docker Desktop to test it locally. This step assumes that your legacy application is already running on an on-premises or Amazon Elastic Compute Cloud (Amazon EC2) Windows instance. For more information, see the pattern [Run an ASP.NET Core web API Docker container on an Amazon EC2 Linux instance](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html).
1. Deploy the modernized ASP.NET core application to Amazon Elastic Container Service (Amazon ECS). This pattern doesn’t cover the deployment step. For instructions, see the [Amazon ECS Workshop](https://ecsworkshop.com/).
**Note**
This pattern doesn’t cover UI development, database modernization, or container deployment steps.
## Prerequisites and limitations
**Prerequisites **
+ [Visual Studio](https://visualstudio.microsoft.com/downloads/) or [Visual Studio Code](https://code.visualstudio.com/download), downloaded and installed.
+ Access to an AWS account using the AWS Management Console and the AWS Command Line Interface (AWS CLI) version 2. (See the [instructions for configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).)
+ The AWS Toolkit for Visual Studio (see [setup instructions](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/setup.html)).
+ Docker Desktop, [downloaded](https://www.docker.com/products/docker-desktop) and installed.
+ .NET SDK, [downloaded](https://download.visualstudio.microsoft.com/download/pr/4263dc3b-dc67-4f11-8d46-cc0ae86a232e/66782bbd04c53651f730b2e30a873f18/dotnet-sdk-5.0.203-win-x64.exe) and installed.
+ NDepend tool, [downloaded](https://www.ndepend.com/download) and installed. To install the NDepend extension for Visual Studio, run `NDepend.VisualStudioExtension.Installer` ([see instructions](https://www.ndepend.com/docs/getting-started-with-ndepend#Part1)). You can select Visual Studio 2019 or 2022, depending on your requirements.
+ Porting Assistant for .NET, [downloaded](https://aws.amazon.com/porting-assistant-dotnet/) and installed.
## Architecture
**Modernizing the shopping cart application**
The following diagram illustrates the modernization process for a legacy ASP.NET shopping cart application.
![\[Modernizing a legacy shopping cart application\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36cda8e6-f2cb-4f1a-b37f-fa3045cc5ba1/images/4367e259-9bb3-4eb6-a54d-1c1e2dece7d4.png)
**Target architecture**
The following diagram illustrates the architecture of the modernized shopping cart application on AWS. ASP.NET Core web APIs are deployed to an Amazon ECS cluster. Logging and configuration services are provided by Amazon CloudWatch Logs and AWS Systems Manager.
![\[Target architecture for ASP.NET Web Forms application on AWS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36cda8e6-f2cb-4f1a-b37f-fa3045cc5ba1/images/ed6d65ec-0dc9-43ab-ac07-1f172e089399.png)
## Tools
**AWS services**
+ [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) – Amazon Elastic Container Service (Amazon ECS) is a highly scalable, fast container management service for running, stopping, and managing containers on a cluster. You can run your tasks and services on a serverless infrastructure that is managed by AWS Fargate. Alternatively, for more control over your infrastructure, you can run your tasks and services on a cluster of EC2 instances that you manage.
+ [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) – Amazon CloudWatch Logs centralizes the logs from all your systems, applications, and AWS services that you use. You can view and monitor the logs, search them for specific error codes or patterns, filter them based on specific fields, or archive them securely for future analysis.
+ [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) ─ AWS Systems Manager is an AWS service that you can use to view and control your infrastructure on AWS. Using the Systems Manager console, you can view operational data from multiple AWS services and automate operational tasks across your AWS resources. Systems Manager helps you maintain security and compliance by scanning your managed instances and reporting (or taking corrective action) on any policy violations it detects.
**Tools**
+ [Visual Studio](https://visualstudio.microsoft.com/) or [Visual Studio Code](https://code.visualstudio.com/) – Tools for building .NET applications, web APIs, and other programs.
+ [AWS Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/welcome.html) – An extension for Visual Studio that helps develop, debug, and deploy .NET applications that use AWS services.
+ [Docker Desktop](https://www.docker.com/products/docker-desktop) – A tool that simplifies building and deploying containerized applications.
+ [NDepend](https://www.ndepend.com/features/) – An analyzer that monitors .NET code for dependencies, quality issues, and code changes.
+ [Porting Assistant for .NET](https://aws.amazon.com/porting-assistant-dotnet/) – An analysis tool that scans .NET code to identify incompatibilities with .NET Core and to estimate the migration effort.
## Epics
### Port your legacy application to .NET 5 or later version
| Task | Description | Skills required |
| --- | --- | --- |
| Upgrade your.NET Framework legacy application to .NET 5. | You can use Porting Assistant for .NET to convert your legacy ASP.NET Web Forms application to .NET 5 or later. Follow the instructions in the [Porting Assistant for .NET documentation](https://docs.aws.amazon.com/portingassistant/latest/userguide/porting-assistant-getting-started.html). | App developer |
| Generate NDepend reports. | When you modernize your ASP.NET Web Forms application by decomposing it into microservices, you might not need all the .cs files from the legacy application. You can use NDepend to generate a report for any code-behind (.cs) file, to get all the callers and callees. This report helps you identify and use only the required files in your microservices.After you install NDepend (see the [Prerequisites ](#modernize-asp-net-web-forms-applications-on-aws-prereqs)section), open the solution (.sln file) for your legacy application in Visual Studio and follow these steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-asp-net-web-forms-applications-on-aws.html)This process generates a report for the code-behind file that lists all callers and callees. For more information about the dependency graph, see the [NDepend documentation](https://www.ndepend.com/docs/visual-studio-dependency-graph). | App developer |
| Create a new .NET 5 solution. | To create a new .NET 5 (or later) structure for your modernized ASP.NET Core web APIs:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-asp-net-web-forms-applications-on-aws.html)For more information about creating projects and solutions, see the [Visual Studio documentation](https://docs.microsoft.com/en-us/visualstudio/ide/creating-solutions-and-projects).As you build the solution and verify functionality, you might identify several additional files to be added to the solution, in addition to the files that NDepend identified. | App developer |
### Update your application code
| Task | Description | Skills required |
| --- | --- | --- |
| Implement web APIs with ASP.NET Core. | Let’s assume that one of the microservices that you identified in your legacy monolith shopping cart application is *Products*. You created a new ASP.NET Core web API project for *Products* in the previous epic. In this step, you identify and modernize all the web forms (.aspx pages) that are related to *Products*. Let’s assume that *Products* consists of four web forms, as illustrated earlier in the [Architecture](#modernize-asp-net-web-forms-applications-on-aws-architecture) section:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-asp-net-web-forms-applications-on-aws.html)You should analyze each web form, identify all the requests that are sent to the database to perform some logic, and get responses. You can implement each request as a web API endpoint. Given its web forms, *Products* can have the following possible endpoints:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-asp-net-web-forms-applications-on-aws.html)As mentioned previously, you can also reuse all the other projects that you upgraded to .NET 5, including Business Logic, Data Access, and shared/common projects. | App developer |
| Configure Amazon CloudWatch Logs. | You can use [Amazon CloudWatch Logs](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) to monitor, store, and access your application’s logs. You can log data into Amazon CloudWatch Logs by using an AWS SDK. You can also integrate .NET applications with CloudWatch Logs by using popular .NET logging frameworks such as [NLog](https://www.nuget.org/packages/AWS.Logger.NLog/), [Log4Net](https://www.nuget.org/packages/AWS.Logger.Log4net/), and [ASP.NET Core logging framework](https://www.nuget.org/packages/AWS.Logger.AspNetCore/).For more information about this step, see the blog post [Amazon CloudWatch Logs and .NET Logging Frameworks](https://aws.amazon.com/blogs/developer/amazon-cloudwatch-logs-and-net-logging-frameworks/). | App developer |
| Configure AWS Systems Manager Parameter Store. | You can use [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html) to store application settings such as connection strings separately from your application’s code. The NuGet package [Amazon.Extensions.Configuration.SystemsManager](https://www.nuget.org/packages/Amazon.Extensions.Configuration.SystemsManager/) simplifies how your application loads these settings from the AWS Systems Manager Parameter Store into the .NET Core configuration system. For more information about this step, see the blog post [.NET Core configuration provider for AWS Systems Manager](https://aws.amazon.com/blogs/developer/net-core-configuration-provider-for-aws-systems-manager/). | App developer |
### Add authentication and authorization
| Task | Description | Skills required |
| --- | --- | --- |
| Use a shared cookie for authentication. | Modernizing a legacy monolith application is an iterative process and requires the monolith and its modernized version to co-exist. You can use a shared cookie to achieve seamless authentication between the two versions. The legacy ASP.NET application continues to validate user credentials and issues the cookie while the modernized ASP.NET Core application validates the cookie. For instructions and sample code, see the [sample GitHub project](https://github.com/aws-samples/dotnet-share-auth-cookie-between-monolith-and-modernized-apps). | App developer |
### Build and run the container locally
| Task | Description | Skills required |
| --- | --- | --- |
| Create a Docker image by using Visual Studio. | In this step, you create a Docker file by using the Visual Studio for .NET Core web API.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-asp-net-web-forms-applications-on-aws.html)Visual Studio creates a Docker file for your project. For a sample Docker file, see [Visual Studio Container Tools for Docker](https://docs.microsoft.com/en-us/visualstudio/containers/overview) on the Microsoft website. | App developer |
| Build and run the container by using Docker Desktop. | Now you can build, create and run the container in Docker Desktop.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-asp-net-web-forms-applications-on-aws.html) | App developer |
## Related resources
+ [Run an ASP.NET Core web API Docker container on an Amazon EC2 Linux instance](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html) (AWS Prescriptive Guidance)
+ [Amazon ECS Workshop](https://ecsworkshop.com/)
+ [Perform ECS blue/green deployments through CodeDeploy using AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/blue-green.html) (AWS CloudFormation documentation)
+ [Getting started with NDepend](https://www.ndepend.com/docs/getting-started-with-ndepend) (NDepend documentation)
+ [Porting Assistant for .NET](https://aws.amazon.com/porting-assistant-dotnet/)
## Additional information
The following tables provide examples of sample projects for a legacy shopping cart application and the equivalent projects in your modernized ASP.NET Core application.
**Legacy solution:**
|
|
| Project name | Project template | Target framework |
| --- |--- |--- |
| Business Interface | Class Library | .NET Framework |
| BusinessLogic | Class Library | .NET Framework |
| WebApplication | ASP.NET Framework Web Application | .NET Framework |
| UnitTests | NUnit Test Project | .NET Framework |
| Shared ->Common | Class Library | .NET Framework |
| Shared ->Framework | Class Library | .NET Framework |
**New solution:**
|
|
| Project name | Project template | Target framework |
| --- |--- |--- |
| BusinessLogic | Class Library | .NET 5.0 |
| | ASP.NET Core Web API | .NET 5.0 |
| .UnitTests | NUnit 3 Test Project | .NET 5.0 |
| Shared ->Common | Class Library | .NET 5.0 |
| Shared ->Framework | Class Library | .NET 5.0 |
# Tenant onboarding in SaaS architecture for the silo model using C\$1 and AWS CDK
*Tabby Ward, Susmitha Reddy Gankidi, and Vijai Anand Ramalingam, Amazon Web Services*
## Summary
Software as a service (SaaS) applications can be built with a variety of different architectural models. The *silo model* refers to an architecture where tenants are provided dedicated resources.
SaaS applications rely on a frictionless model for introducing new tenants into their environment. This often requires the orchestration of a number of components to successfully provision and configure all the elements needed to create a new tenant. This process, in SaaS architecture, is referred to as tenant on-boarding. On-boarding should be fully automated for every SaaS environment by utilizing infrastructure as code in your on-boarding process.
This pattern guides you through an example of creating a tenant and provisioning a basic infrastructure for the tenant on Amazon Web Services (AWS). The pattern uses C\$1 and the AWS Cloud Development Kit (AWS CDK).
Because this pattern creates a billing alarm, we recommend deploying the stack in the US East (N. Virginia), or us-east-1, AWS Region. For more information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html).
## Prerequisites and limitations
**Prerequisites**** **
+ An active [AWS account](https://aws.amazon.com/account/).
+ An AWS Identity and Access Management (IAM) principal with sufficient IAM access to create AWS resources for this pattern. For more information, see [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
+ [Install Amazon Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to perform AWS CDK deployment.
+ [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/) downloaded and installed or [Visual Studio Code](https://code.visualstudio.com/download) downloaded and installed.
+ [AWS Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/setup.html) set up.
+ [.NET Core 3.1 or later](https://dotnet.microsoft.com/download/dotnet-core/3.1) (required for C\$1 AWS CDK applications)
+ [Amazon.Lambda.Tools](https://github.com/aws/aws-extensions-for-dotnet-cli#aws-lambda-amazonlambdatools) installed.
**Limitations**** **
+ AWS CDK uses [AWS CloudFormation](https://aws.amazon.com/cloudformation/), so AWS CDK applications are subject to CloudFormation service quotas. For more information, see [AWS CloudFormation quotas](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html).
+ The tenant CloudFormation stack is created with a CloudFormation service role `infra-cloudformation-role` with wildcard characters on actions (`sns`\$1 and `sqs*`) but with resources locked down to the `tenant-cluster` prefix. For a production use case, evaluate this setting and provide only required access to this service role. The `InfrastructureProvision` Lambda function also uses a wildcard character (`cloudformation*`) to provision the CloudFormation stack but with resources locked down to the `tenant-cluster` prefix.
+ This example code's docker build uses `--platform=linux/amd64` to force `linux/amd64` based images. This is to ensure that the final image artifacts will be suitable for Lambda, which by default uses x86-64 architecture. If you need to change the target Lambda architecture, be sure to change both the Dockerfiles and the AWS CDK codes. For more information, see the blog post [Migrating AWS Lambda functions to Arm-based AWS Graviton2 processors](https://aws.amazon.com/blogs/compute/migrating-aws-lambda-functions-to-arm-based-aws-graviton2-processors/).
+ The stack deletion process will not clean up CloudWatch Logs (log groups and logs) generated by the stack. You must manually clean up the logs through the AWS Management Console Amazon CloudWatch console or the through the API.
This pattern is set up as an example. For production use, evaluate the following setups and make changes based on your business requirements:
+ The [AWS Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) bucket in this example does not have versioning enabled for simplicity. Evaluate and update the setup as needed.
+ This example sets up [Amazon API Gateway](https://aws.amazon.com/api-gateway/) REST API endpoints without authentication, authorization, or throttling for simplicity. For production use, we recommend integrating the system with the business security infrastructure. Evaluate this setting and add required security settings as needed.
+ For this tenant infrastructure example, [Amazon Simple Notification Service (Amazon SNS)](https://aws.amazon.com/sns/) and [Amazon Simple Queue Service (Amazon SQS)](https://aws.amazon.com/sqs/) have only minimum setups. The [AWS Key Management Service (AWS KMS)](https://aws.amazon.com/kms/) for each tenant opens to [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) and Amazon SNS services in the account to consume based on the [AWS KMS key policy](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-key-management.html#compatibility-with-aws-services). The setup is only an example placeholder. Adjust the setups as needed based on your business use case.
+ The entire setup, which includes but isn’t limited to API endpoints and backend tenant provisioning and deletion by using AWS CloudFormation, covers only the basic happy path case. Evaluate and update the setup with the necessary retry logic, additional error handling logic, and security logic based on your business needs.
+ The example code is tested with up-to-date [cdk-nag](https://github.com/cdklabs/cdk-nag) to check for policies at the time of this writing. New policies might be enforced in the future. These new policies might require you to manually modify the stack based on the recommendations before the stack can be deployed. Review the existing code to ensure that it aligns with your business requirements.
+ The code relies on the AWS CDK to generate a random suffix instead of relying on static assigned physical names for most created resources. This setup is to ensure that these resources are unique and do not conflict with other stacks. For more information, see the [AWS CDK documentation](https://docs.aws.amazon.com/cdk/v2/guide/resources.html#resources_physical_names). Adjust this based on your business requirements.
+ This example code packages .NET Lambda artifacts into Docker based images and runs with the Lambda provided [Container image runtime](https://docs.aws.amazon.com/lambda/latest/dg/csharp-image.html). The container image runtime has advantages for standard transfer and store mechanisms (container registries) and more accurate local test environments (through the container image). You can switch the project to use [Lambda provided .NET runtimes](https://docs.aws.amazon.com/lambda/latest/dg/lambda-csharp.html) to reduce the build time of the Docker images, but you will then need to set up transfer and store mechanisms and ensure that the local setup matches the Lambda setup. Adjust the code to align with users' business requirements.
**Product versions**
+ AWS CDK version 2.45.0 or later
+ Visual Studio 2022
## Architecture
**Technology stack**
+ Amazon API Gateway
+ AWS CloudFormation
+ Amazon CloudWatch
+ Amazon DynamoDB
+ AWS Identity and Access Management (IAM)
+ AWS KMS
+ AWS Lambda
+ Amazon S3
+ Amazon SNS
+ Amazon SQS
**Architecture**
The following diagram shows the tenant stack creation flow. For more information about the control-plane and tenant technology stacks, see the *Additional information* section.
![\[Workflow to create a tenant and provision a basic infrastructure for the tenant on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5baef800-fe39-4eb8-b11d-2c23eb3175fc/images/0b579484-b87c-4acb-8c60-8c33c18370e3.png)
**Tenant stack creation flow**
1. User sends a POST API request with new tenant payload (tenant name, tenant description) in JSON to a REST API hosted by Amazon API Gateway. The API Gateway processes the request and forwards it to the backend Lambda Tenant On-boarding function. In this example, there is no authorization or authentication. In a production setup, this API should be integrated with the SaaS infrastructure security system.
1. The Tenant On-boarding function verifies the request. Then it attempts to store the tenant record, which includes the tenant name, generated tenant universally unique identifier (UUID), and tenant description, into the Amazon DynamoDB Tenant On-boarding table.
1. After DynamoDB stores the record, a DynamoDB stream initiates the downstream Lambda Tenant Infrastructure function.
1. The Tenant Infrastructure Lambda function acts based the on received DynamoDB stream. If the stream is for the INSERT event, the function uses the stream's NewImage section (latest update record, Tenant Name field) to invoke CloudFormation to create a new tenant infrastructure using the template that is stored in the S3 bucket. The CloudFormation template requires the Tenant Name parameter.
1. AWS CloudFormation creates the tenant infrastructure based on the CloudFormation template and input parameters.
1. Each tenant infrastructure setup has a CloudWatch alarm, a billing alarm, and an alarm event.
1. The alarm event becomes a message to an SNS topic, which is encrypted by the tenant's AWS KMS key.
1. The SNS topic forwards the received alarm message to the SQS queue, which is encrypted by the tenant's AWS KMS for encryption key.
Other systems can be integrated with Amazon SQS to perform actions based on messages in queue. In this example, to keep the code generic, incoming messages remain in queue and require manual deletion.
**Tenant stack deletion flow**
1. User sends a DELETE API request with new tenant payload (tenant name, tenant description) in JSON to the REST API hosted by Amazon API Gateway, which will process the request and forward to Tenant On-boarding function. In this example, there is no authorization or authentication. In a production setup, this API will be integrated with the SaaS infrastructure security system.
1. The Tenant On-boarding function will verify the request and then attempt to delete the tenant record (tenant name) from the Tenant On-boarding table.
1. After DynamoDB deletes the record successfully (the record exists in the table and is deleted), a DynamoDB stream initiates the downstream Lambda Tenant Infrastructure function.
1. The Tenant Infrastructure Lambda function acts based on the received DynamoDB stream record. If the stream is for the REMOVE event, the function uses the record's OldImage section (record information and Tenant Name field, before the latest change, which is delete) to initiate deletion of an existing stack based on that record information.
1. AWS CloudFormation deletes the target tenant stack according to the input.
## Tools
**AWS services**
+ [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html) helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.
+ [AWS Cloud Development Kit (AWS CDK)](https://docs.aws.amazon.com/cdk/v2/guide/home.html) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.
+ [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/v2/guide/cli.html) is a command line cloud development kit that helps you interact with your AWS Cloud Development Kit (AWS CDK) app.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Key Management Service (AWS KMS) ](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html)helps you create and control cryptographic keys to help protect your data.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Amazon Simple Notification Service (Amazon SNS)](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
+ [Amazon Simple Queue Service (Amazon SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
+ [AWS Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/welcome.html) is a plugin for the Visual Studio integrated development environment (IDE). The Toolkit for Visual Studio supports developing, debugging, and deploying .NET applications that use AWS services.
**Other tools**
+ [Visual Studio](https://docs.microsoft.com/en-us/visualstudio/ide/whats-new-visual-studio-2022?view=vs-2022) is an IDE that includes compilers, code completion tools, graphical designers, and other features that support software development.
**Code**
The code for this pattern is in the [Tenant onboarding in SaaS Architecture for Silo Model APG Example](https://github.com/aws-samples/tenant-onboarding-in-saas-architecture-for-silo-model-apg-example) repository.
## Epics
### Set up AWS CDK
| Task | Description | Skills required |
| --- | --- | --- |
| Verify Node.js installation. | To verify that Node.js is installed on your local machine, run the following command.
node --version
| AWS administrator, AWS DevOps |
| Install AWS CDK Toolkit. | To install AWS CDK Toolkit on your local machine, run the following command.
npm install -g aws-cdk
If npm is not installed, you can install it from the [Node.js site](https://nodejs.org/en/download/package-manager/). | AWS administrator, AWS DevOps |
| Verify the AWS CDK Toolkit version. | To verify that the AWS CDK Toolkit version is installed correctly on your machine, run the following command.
cdk --version
| AWS administrator, AWS DevOps |
### Review the code for the tenant onboarding control plane
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the repository. | Clone the [repository](https://github.com/aws-samples/tenant-onboarding-in-saas-architecture-for-silo-model-apg-example), and navigate to the `\tenant-onboarding-in-saas-architecture-for-silo-model-apg-example` folder.In Visual Studio 2022, open the `\src\TenantOnboardingInfra.sln` solution. Open the `TenantOnboardingInfraStack.cs` file and review the code.The following resources are created as part of this stack:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html) | AWS administrator, AWS DevOps |
| Review the CloudFormation template. | In the `\tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\template` folder, open `infra.yaml`, and review the CloudFormation template. This template will be hydrated with the tenant name retrieved from the tenant onboarding DynamoDB table.The template provisions the tenant-specific infrastructure. In this example, it provisions the AWS KMS key, Amazon SNS , Amazon SQS, and the CloudWatch alarm. | App developer, AWS DevOps |
| Review the tenant onboarding function. | Open `Function.cs`, and review the code for the tenant onboarding function, which is created with the Visual Studio AWS Lambda Project (.NET Core- C\$1) template with the .NET 6 (Container Image) blueprint.Open the `Dockerfile`, and review the code. The `Dockerfile` is a text file that consists of instructions for building the Lambda container image.Note that the following NuGet packages are added as dependencies to the `TenantOnboardingFunction` project:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html) | App developer, AWS DevOps |
| Review the Tenant InfraProvisioning function. | Navigate to `\tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\src\InfraProvisioningFunction`.Open `Function.cs`, and review the code for the tenant infrastructure provisioning function, which is created with the Visual Studio AWS Lambda Project (.NET Core- C\$1) template with the .NET 6 (Container Image) blueprint.Open the `Dockerfile`, and review the code. Note that the following NuGet packages are added as dependencies to the `InfraProvisioningFunction` project:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html) | App developer, AWS DevOps |
### Deploy the AWS resources
| Task | Description | Skills required |
| --- | --- | --- |
| Build the solution. | To build the solution, perform the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html)Make sure that you update the `Amazon.CDK.Lib NuGet` package to the latest version in `\tenant-onboarding-in-saas-architecture-for-silo-model-apg-example\src\TenantOnboardingInfra` project before you build the solution. | App developer |
| Bootstrap the AWS CDK environment. | Open the Windows command prompt and navigate to the AWS CDK app root folder where the `cdk.json` file is available (`\tenant-onboarding-in-saas-architecture-for-silo-model-apg-example`). Run the following command for bootstrapping.
cdk bootstrap
If you have created an AWS profile for the credentials, use the command with your profile.
cdk bootstrap --profile
| AWS administrator, AWS DevOps |
| List the AWS CDK stacks. | To list all the stacks to be created as part of this project, run the following command.
cdk ls cdk ls --profile
If you have created an AWS profile for the credentials, use the command with your profile.
cdk ls --profile
| AWS administrator, AWS DevOps |
| Review which AWS resources will be created. | To review all the AWS resources that will be created as part of this project, run the following command.
cdk diff
If you have created an AWS profile for the credentials, use the command with your profile.
cdk diff --profile
| AWS administrator, AWS DevOps |
| Deploy all the AWS resources by using AWS CDK. | To deploy all the AWS resources run the following command.
cdk deploy --all --require-approval never
If you have created an AWS profile for the credentials, use the command with your profile.
cdk deploy --all --require-approval never --profile
After the deployment is complete, copy the API URL from the outputs section in the command prompt, which is shown in the following example.
| AWS administrator, AWS DevOps |
### Verify the functionality
| Task | Description | Skills required |
| --- | --- | --- |
| Create a new tenant. | To create the new tenant, send the following curl request.
curl -X POST tenant -d '{"Name":"Tenant123", "Description":"Stack for Tenant123"}'
Change the place holder `` to the actual value from AWS CDK, as shown in the following example.
curl -X POST https://j2qmp8ds21i1i.execute-api.us-west-2.amazonaws.com/prod/tenant -d '{"Name":"Tenant123", "Description":"test12"}'
The following example shows the output.
{"message": "A new tenant added - 5/4/2022 7:11:30 AM"}
| App developer, AWS administrator, AWS DevOps |
| Verify the newly created tenant details in DynamoDB. | To verify the newly created tenant details in DynamoDB, perform the following steps.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html) | App developer, AWS administrator, AWS DevOps |
| Verify the stack creation for the new tenant. | Verify that the new stack was successfully created and provisioned with infrastructure for the newly created tenant according to the CloudFormation template.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html) | App developer, AWS administrator, AWS DevOps |
| Delete the tenant stack. | To delete the tenant stack, send the following curl request.
curl -X DELETE tenant/
Change the place holder `` to the actual value from AWS CDK, and change `` to the actual value from the previous tenant creation step, as shown in the following example.
| App developer, AWS DevOps, AWS administrator |
| Verify the stack deletion for the existing tenant. | To verify that the existing tenant stack got deleted, perform the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html) | App developer, AWS administrator, AWS DevOps |
### Clean up
| Task | Description | Skills required |
| --- | --- | --- |
| Destroy the environment. | Before the stack clean up, ensure the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/tenant-onboarding-in-saas-architecture-for-the-silo-model-using-c-and-aws-cdk.html)After testing is done, AWS CDK can be used to destroy all the stacks and related resources by running the following command.
cdk destroy --all;
If you created an AWS profile for the credentials, use the profile.Confirm the stack deletion prompt to delete the stack. | AWS administrator, AWS DevOps |
| Clean up Amazon CloudWatch Logs. | The stack deletion process will not clean up CloudWatch Logs (log groups and logs) that were generated by the stack. Manually clean up the CloudWatch resources by using the CloudWatch console or the API. | App developer, AWS DevOps, AWS administrator |
## Related resources
+ [AWS CDK .NET Workshop](https://cdkworkshop.com/40-dotnet.html)
+ [Working with the AWS CDK in C\$1](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-csharp.html)
+ [CDK .NET Reference](https://docs.aws.amazon.com/cdk/api/v2/dotnet/api/index.html)
## Additional information
**Control-plane technology stack**
The CDK code written in .NET is used to provision the control-plane infrastructure, which consists of the following resources:
1. **API Gateway**
Serves as the REST API entry point for the control-plane stack.
1. **Tenant on-boarding Lambda function**
This Lambda function is initiated by API Gateway using the m method.
A POST method API request results in (`tenant name`, `tenant description`) being inserted into the DynamoDB `Tenant Onboarding` table.
In this code example, the tenant name is also used as part of the tenant stack name and the names of resources within that stack. This is to make these resources easier to identify. This tenant name must be unique across the setup to avoid conflicts or errors. Detailed input validation setup is explained in the [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) documentation and the *Limitations* section.
The persistence process to the DynamoDB table will succeed only if the tenant name is not used in any other record in the table.
The tenant name in this case is the partition key for this table, because only the partition key can be used as a `PutItem` condition expression.
If the tenant name was never recorded before, the record will be saved into the table successfully.
However, if the tenant name is already used by an existing record in the table, the operation will fail and initiate a DynamoDB `ConditionalCheckFailedException` exception. The exception will be used to return a failure message (`HTTP BadRequest`) indicating that the tenant name already exists.
A `DELETE` method API request will remove the record for a specific tenant name from the `Tenant Onboardin`g table.
The DynamoDB record deletion in this example will succeed even if the record does not exist.
If the target record exists and is deleted, it will create a DynamoDB stream record. Otherwise, no downstream record will be created.
1. **Tenant on-boarding DynamoDB, with Amazon DynamoDB Streams enabled**
This records the tenant metadata information, and any record save or deletion will send a stream downstream to the `Tenant Infrastructure` Lambda function.
1. **Tenant infrastructure Lambda function**
This Lambda function is initiated by the DynamoDB stream record from the previous step. If the record is for an `INSERT` event, it invokes AWS CloudFormation to create a new tenant infrastructure with the CloudFormation template that is stored in an S3 bucket. If the record is for `REMOVE`, it initiates deletion of an existing stack based on the stream record's `Tenant Name` field.
1. **S3 bucket**
This is for storing the CloudFormation template.
1. **IAM roles for each Lambda function and a service role for CloudFormation**
Each Lambda function has its unique IAM role with [least-privilege permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) to achieve its task. For example, the `Tenant On-boarding` Lambda function has read/write access to DynamoDB, and the `Tenant Infrastructure` Lambda function can only read the DynamoDB stream.
A custom CloudFormation service role is created for tenant stack provisioning. This service role contains additional permissions for CloudFormation stack provisioning (for example, the AWS KMS key). This divides roles between Lambda and CloudFormation to avoid all permissions on a single role (Infrastructure Lambda role).
Permissions that allow powerful actions (such as creating and deleting CloudFormation stacks) are locked down and allowed only on resources that start with `tenantcluster-`. The exception is AWS KMS, because of its resource naming convention. The ingested tenant name from the API will be prepended with `tenantcluster-` along with other validation checks (alphanumeric with dash only, and limited to less than 30 characters to fit into most AWS resource naming). This ensures that the tenant name will not accidentally result in disruption of core infrastructure stacks or resources.
**Tenant technology stack**
A CloudFormation template is stored in the S3 bucket. The template provisions the tenant-specific AWS KMS key, a CloudWatch alarm, an SNS topic, an SQS queue, and an [SQS policy](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-identity-based-policies.html).
The AWS KMS key is used for data encryption by Amazon SNS and Amazon SQS for their messages. The security practices for [AwsSolutions-SNS2 and AwsSolutions-SQS2](https://github.com/cdklabs/cdk-nag/blob/main/RULES.md) recommend that you set up Amazon SNS and Amazon SQS with encryption. However, CloudWatch alarms don’t work with Amazon SNS when using an AWS managed key, so you must use a customer managed key in this case. For more information, see the [AWS Knowledge Center](https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-receive-sns-for-alarm-trigger/).
The SQS policy is used on the Amazon SQS queue to allow the created SNS topic to deliver the message to the queue. Without the SQS policy, the access will be denied. For more information, see the [Amazon SNS documentation](https://docs.aws.amazon.com/sns/latest/dg/subscribe-sqs-queue-to-sns-topic.html#SendMessageToSQS.sqs.permissions).
# Decompose monoliths into microservices by using CQRS and event sourcing
*Rodolfo Jr. Cerrada, Dmitry Gulin, and Tabby Ward, Amazon Web Services*
## Summary
This pattern combines two patterns, using both the command query responsibility separation (CQRS) pattern and the event sourcing pattern. The CQRS pattern separates responsibilities of the command and query models. The event sourcing pattern takes advantage of asynchronous event-driven communication to improve the overall user experience.
You can use CQRS and Amazon Web Services (AWS) services to maintain and scale each data model independently while refactoring your monolith application into microservices architecture. Then you can use the event sourcing pattern to synchronize data from the command database to the query database.
This pattern uses example code that includes a solution (\$1.sln) file that you can open using the latest version of Visual Studio. The example contains Reward API code to showcase how CQRS and event sourcing work in AWS serverless and traditional or on-premises applications.
To learn more about CQRS and event sourcing, see the [Additional information](#decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing-additional) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ Amazon CloudWatch
+ Amazon DynamoDB tables
+ Amazon DynamoDB Streams
+ AWS Identity and Access Management (IAM) access key and secret key; for more information, see the video in the *Related resources* section
+ AWS Lambda
+ Familiarity with Visual Studio
+ Familiarity with AWS Toolkit for Visual Studio; for more information, see the *AWS Toolkit for Visual Studio demo* video in the *Related resources* section
**Product versions**
+ [Visual Studio 2019 Community Edition](https://visualstudio.microsoft.com/downloads/).
+ [AWS Toolkit for Visual Studio 2019](https://aws.amazon.com/visualstudio/).
+ .NET Core 3.1. This component is an option in the Visual Studio installation. To include .NET Core during installation, select **NET Core cross-platform development**.
**Limitations**
+ The example code for a traditional on-premises application (ASP.NET Core Web API and data access objects) does not come with a database. However, it comes with the `CustomerData` in-memory object, which acts as a mock database. The code provided is enough for you to test the pattern.
## Architecture
**Source technology stack**
+ ASP.NET Core Web API project
+ IIS Web Server
+ Data access object
+ CRUD model
**Source architecture**
In the source architecture, the CRUD model contains both command and query interfaces in one application. For example code, see `CustomerDAO.cs` (attached).
![\[Connections between application, service interface, customer CRUD model, and database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/1cd3a84c-12c7-4306-99aa-23f2c53d3cd3.png)
**Target technology stack **
+ Amazon DynamoDB
+ Amazon DynamoDB Streams
+ AWS Lambda
+ (Optional) Amazon API Gateway
+ (Optional) Amazon Simple Notification Service (Amazon SNS)
**Target architecture **
In the target architecture, the command and query interfaces are separated. The architecture shown in the following diagram can be extended with API Gateway and Amazon SNS. For more information, see the [Additional information](#decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing-additional) section.
![\[Application connecting with serverless Customer Command and Customer Query microservices.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/1c665697-e3ac-4ef4-98d0-86c2cbf164c1.png)
1. Command Lambda functions perform write operations, such as create, update, or delete, on the database.
1. Query Lambda functions perform read operations, such as get or select, on the database.
1. This Lambda function processes the DynamoDB streams from the Command database and updates the Query database for the changes.
## Tools
**Tools**
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) – Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
+ [Amazon DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) – DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table. It then stores this information in a log for up to 24 hours. Encryption at rest encrypts the data in DynamoDB streams.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) – AWS Lambda is a compute service that supports running code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume—there is no charge when your code is not running.
+ [AWS Management Console](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/learn-whats-new.html) – The AWS Management Console is a web application that comprises a broad collection of service consoles for managing AWS services.
+ [Visual Studio 2019 Community Edition](https://visualstudio.microsoft.com/downloads/) – Visual Studio 2019 is an integrated development environment (IDE). The Community Edition is free for open-source contributors. In this pattern, you will use Visual Studio 2019 Community Edition to open, compile, and run example code. For viewing only, you can use any text editor or [Visual Studio Code](https://docs.aws.amazon.com/toolkit-for-vscode/latest/userguide/welcome.html).
+ [AWS Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/welcome.html) – The AWS Toolkit for Visual Studio is a plugin for the Visual Studio IDE. The AWS Toolkit for Visual Studio makes it easier for you to develop, debug, and deploy .NET applications that use AWS services.
**Code **
The example code is attached. For instructions on deploying the example code, see the *Epics* section.
## Epics
### Open and build the solution
| Task | Description | Skills required |
| --- | --- | --- |
| Open the solution. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer |
| Build the solution. | Open the context (right-click) menu for the solution, and then choose **Build Solution**. This will build and compile all the projects in the solution. It should compile successfully.Visual Studio Solution Explorer should show the directory structure.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer |
### Build the DynamoDB tables
| Task | Description | Skills required |
| --- | --- | --- |
| Provide credentials. | If you don't have an access key yet, see the video in the *Related resources* section.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer, Data engineer, DBA |
| Build the project. | To build the project, open the context (right-click) menu for the **AwS.APG.CQRSES.Build** project, and then choose **Build**. | App developer, Data engineer, DBA |
| Build and populate the tables. | To build the tables and populate them with seed data, open the context (right-click) menu for the **AwS.APG.CQRSES.Build** project, and then choose **Debug**,** Start New Instance**. | App developer, Data engineer, DBA |
| Verify the table construction and the data. | To verify, navigate to **AWS Explorer**, and expand **Amazon DynamoDB**. It should display the tables. Open each table to display the example data. | App developer, Data engineer, DBA |
### Run local tests
| Task | Description | Skills required |
| --- | --- | --- |
| Build the CQRS project. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer, Test engineer |
| Build the event-sourcing project. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer, Test engineer |
| Run the tests. | To run all tests, choose **View**, **Test Explorer**, and then choose **Run All Tests In View**. All tests should pass, which is indicated by a green check mark icon. | App developer, Test engineer |
### Publish the CQRS Lambda functions to AWS
| Task | Description | Skills required |
| --- | --- | --- |
| Publish the first Lambda function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer, DevOps engineer |
| Verify the function upload. | (Optional) You can verify that the function was successfully loaded by navigating to AWS Explorer and expanding **AWS Lambda**. To open the test window, choose the Lambda function (double-click). | App developer, DevOps engineer |
| Test the Lambda function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html)All CQRS Lambda projects are found under the `CQRS AWS Serverless\CQRS\Command Microservice` and` CQRS AWS Serverless\CQRS\Command Microservice` solution folders. For the solution directory and projects, see **Source code directory** in the [Additional information](#decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing-additional) section. | App developer, DevOps engineer |
| Publish the remaining functions. | Repeat the previous steps for the following projects:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer, DevOps engineer |
### Set up the Lambda function as an event listener
| Task | Description | Skills required |
| --- | --- | --- |
| Publish the Customer and Reward Lambda event handlers. | To publish each event handler, follow the steps in the preceding epic.The projects are under the `CQRS AWS Serverless\Event Source\Customer Event` and `CQRS AWS Serverless\Event Source\Reward Event` solution folders. For more information, see *Source code directory* in the [Additional information](#decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing-additional) section. | App developer |
| Attach the event-sourcing Lambda event listener. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html)After the listener is successfully attached to the DynamoDB table, it will be displayed on the Lambda designer page. | App developer |
| Publish and attach the EventSourceReward Lambda function. | To publish and attach the `EventSourceReward` Lambda function, repeat the steps in the previous two stories, selecting **cqrses-reward-cmd** from the **DynamoDB table** dropdown list. | App developer |
### Test and validate the DynamoDB streams and Lambda trigger
| Task | Description | Skills required |
| --- | --- | --- |
| Test the stream and the Lambda trigger. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer |
| Validate, using the DynamodDB reward query table. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer |
| Validate, using CloudWatch Logs. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/decompose-monoliths-into-microservices-by-using-cqrs-and-event-sourcing.html) | App developer |
| Validate the EventSourceCustomer trigger. | To validate the `EventSourceCustomer` trigger, repeat the steps in this epic, using the `EventSourceCustomer` trigger's respective customer table and CloudWatch logs. | App developer |
## Related resources
**References **
+ [Visual Studio 2019 Community Edition downloads](https://visualstudio.microsoft.com/downloads/)
+ [AWS Toolkit for Visual Studio download](https://aws.amazon.com/visualstudio/)
+ [AWS Toolkit for Visual Studio User Guide](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/welcome.html)
+ [Serverless on AWS](https://aws.amazon.com/serverless/)
+ [DynamoDB Use Cases and Design Patterns](https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/)
+ [Martin Fowler CQRS](https://martinfowler.com/bliki/CQRS.html)
+ [Martin Fowler Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html)
**Videos**
+ [AWS Toolkit for Visual Studio demo](https://www.youtube.com/watch?v=B190tcu1ERk)
+ [How do I create an access key ID for a new IAM user?](https://www.youtube.com/watch?v=665RYobRJDY)
## Additional information
**CQRS and event sourcing**
*CQRS*
The CQRS pattern separates a single conceptual operations model, such as a data access object single CRUD (create, read, update, delete) model, into command and query operations models. The command model refers to any operation, such as create, update, or delete, that changes the state. The query model refers to any operation that returns a value.
![\[Architecture with service interface, CRUD model, and database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/3f64756d-681e-4f0e-8034-746263d857b2.png)
1. The Customer CRUD model includes the following interfaces:
+ `Create Customer()`
+ `UpdateCustomer()`
+ `DeleteCustomer()`
+ `AddPoints()`
+ `RedeemPoints()`
+ `GetVIPCustomers()`
+ `GetCustomerList()`
+ `GetCustomerPoints()`
As your requirements become more complex, you can move from this single-model approach. CQRS uses a command model and a query model to separate the responsibility for writing and reading data. That way, the data can be independently maintained and managed. With a clear separation of responsibilities, enhancements to each model do not impact the other. This separation improves maintenance and performance, and it reduces the complexity of the application as it grows.
![\[The application separated into command and query models, sharing a single database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/12db023c-eb81-4c27-bbb9-b085b13176ae.png)
1. Interfaces in the Customer Command model:
+ `Create Customer()`
+ `UpdateCustomer()`
+ `DeleteCustomer()`
+ `AddPoints()`
+ `RedeemPoints()`
1. Interfaces in the Customer Query model:
+ `GetVIPCustomers()`
+ `GetCustomerList()`
+ `GetCustomerPoints()`
+ `GetMonthlyStatement()`
For example code, see *Source code directory*.
The CQRS pattern then decouples the database. This decoupling leads to the total independence of each service, which is the main ingredient of microservice architecture.
![\[Separate databases for command and query models.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/016dbfa8-3bd8-42ee-afa1-38a98986c7d5.png)
Using CQRS in the AWS Cloud, you can further optimize each service. For example, you can set different compute settings or choose between a serverless or a container-based microservice. You can replace your on-premises caching with Amazon ElastiCache. If you have an on-premises publish/subscribe messaging, you can replace it with Amazon Simple Notification Service (Amazon SNS). Additionally, you can take advantage of pay-as-you-go pricing and the wide array of AWS services that you pay only for what you use.
CQRS includes the following benefits:
+ Independent scaling – Each model can have its scaling strategy adjusted to meet the requirements and demand of the service. Similar to high-performance applications, separating read and write enables the model to scale independently to address each demand. You can also add or reduce compute resources to address the scalability demand of one model without affecting the other.
+ Independent maintenance – Separation of query and command models improves the maintainability of the models. You can make code changes and enhancements to one model without affecting the other.
+ Security – It's easier to apply the permissions and policies to separate models for read and write.
+ Optimized reads – You can define a schema that is optimized for queries. For example, you can define a schema for the aggregated data and a separate schema for the fact tables.
+ Integration – CQRS fits well with event-based programming models.
+ Managed complexity – The separation into query and command models is suited to complex domains.
When using CQRS, keep in mind the following caveats:
+ The CQRS pattern applies only to a specific portion of an application and not the whole application. If implemented on a domain that does not fit the pattern, it can reduce productivity, increase risk, and introduce complexity.
+ The pattern works best for frequently used models that have an imbalance read and write operations.
+ For read-heavy applications, such as large reports that take time to process, CQRS gives you the option to select the right database and create a schema to store your aggregated data. This improves the response time of reading and viewing the report by processing the report data only one time and dumping it in the aggregated table.
+ For the write-heavy applications, you can configure the database for write operations and allow the command microservice to scale independently when the demand for write increases. For examples, see the `AWS.APG.CQRSES.CommandRedeemRewardLambda` and `AWS.APG.CQRSES.CommandAddRewardLambda` microservices.
*Event sourcing*
The next step is to use event sourcing to synchronize the query database when a command is run. For example, consider the following events:
+ A customer reward point is added that requires the customer total or aggregated reward points in the query database to be updated.
+ A customer's last name is updated in the command database, which requires the surrogate customer information in the query database to be updated.
In the traditional CRUD model, you ensure consistency of data by locking the data until it finishes a transaction. In event sourcing, the data are synchronized through publishing a series of events that will be consumed by a subscriber to update its respective data.
The event-sourcing pattern ensures and records a full series of actions taken on the data and publishes it through a sequence of events. These events represent a set of changes to the data that subscribers of that event must process to keep their record updated. These events are consumed by the subscriber, synchronizing the data on the subscriber's database. In this case, that's the query database.
The following diagram shows event sourcing used with CQRS on AWS.
![\[Microservice architecture for the CQRS and event sourcing patterns using AWS serverless services.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/cc9bc84a-60b4-4459-9a5c-2334c69dbb4e.png)
1. Command Lambda functions perform write operations, such as create, update, or delete, on the database.
1. Query Lambda functions perform read operations, such as get or select, on the database.
1. This Lambda function processes the DynamoDB streams from the Command database and updates the Query database for the changes. You can also use this function also to publish a message to Amazon SNS so that its subscribers can process the data.
1. (Optional) The Lambda event subscriber processes the message published by Amazon SNS and updates the Query database.
1. (Optional) Amazon SNS sends email notification of the write operation.
On AWS, the query database can be synchronized by DynamoDB Streams. DynamoDB captures a time-ordered sequence of item-level modifications in a DynamobDB table in near-real time and durably stores the information within 24 hours.
Activating DynamoDB Streams enables the database to publish a sequence of events that makes the event sourcing pattern possible. The event sourcing pattern adds the event subscriber. The event subscriber application consumes the event and processes it depending on the subscriber's responsibility. In the previous diagram, the event subscriber pushes the changes to the Query DynamoDB database to keep the data synchronized. The use of Amazon SNS, the message broker, and the event subscriber application keeps the architecture decoupled.
Event sourcing includes the following benefits:
+ Consistency for transactional data
+ A reliable audit trail and history of the actions, which can be used to monitor actions taken in the data
+ Allows distributed applications such as microservices to synchronize their data across the environment
+ Reliable publication of events whenever the state changes
+ Reconstructing or replaying of past states
+ Loosely coupled entities that exchange events for migration from a monolithic application to microservices
+ Reduction of conflicts caused by concurrent updates; event sourcing avoids the requirement to update objects directly in the data store
+ Flexibility and extensibility from decoupling the task and the event
+ External system updates
+ Management of multiple tasks in a single event
When using event sourcing, keep in mind the following caveats:
+ Because there is some delay in updating data between the source subscriber databases, the only way to undo a change is to add a compensating event to the event store.
+ Implementing event sourcing has a learning curve since its different style of programming.
**Test data**
Use the following test data to test the Lambda function after successful deployment.
**CommandCreate Customer**
```
{ "Id":1501, "Firstname":"John", "Lastname":"Done", "CompanyName":"AnyCompany", "Address": "USA", "VIP":true }
```
**CommandUpdate Customer**
```
{ "Id":1501, "Firstname":"John", "Lastname":"Doe", "CompanyName":"Example Corp.", "Address": "Seattle, USA", "VIP":true }
```
**CommandDelete Customer**
Enter the customer ID as request data. For example, if the customer ID is 151, enter 151 as request data.
```
151
```
**QueryCustomerList**
This is blank. When it is invoked, it will return all customers.
**CommandAddReward**
This will add 40 points to customer with ID 1 (Richard).
```
{
"Id":10101,
"CustomerId":1,
"Points":40
}
```
**CommandRedeemReward**
This will deduct 15 points to customer with ID 1 (Richard).
```
{
"Id":10110,
"CustomerId":1,
"Points":15
}
```
**QueryReward**
Enter the ID of the customer. For example, enter 1 for Richard, 2 for Arnav, and 3 for Shirley.
```
2
```
**Source code directory**
Use the following table as a guide to the directory structure of the Visual Studio solution.
*CQRS On-Premises Code Sample solution directory*
![\[Solution directory with Command and Query services expanded.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/4811c2c0-643b-410f-bb87-0b86ec5e194c.png)
**Customer CRUD model**
CQRS On-Premises Code Sample\$1CRUD Model\$1AWS.APG.CQRSES.DAL project
**CQRS version of the Customer CRUD model**
+ Customer command: `CQRS On-Premises Code Sample\CQRS Model\Command Microservice\AWS.APG.CQRSES.Command`project
+ Customer query: `CQRS On-Premises Code Sample\CQRS Model\Query Microservice\AWS.APG.CQRSES.Query` project
**Command and Query microservices**
The Command microservice is under the solution folder `CQRS On-Premises Code Sample\CQRS Model\Command Microservice`:
+ `AWS.APG.CQRSES.CommandMicroservice` ASP.NET Core API project acts as the entry point where consumers interact with the service.
+ `AWS.APG.CQRSES.Command` .NET Core project is an object that hosts command-related objects and interfaces.
The query microservice is under the solution folder `CQRS On-Premises Code Sample\CQRS Model\Query Microservice`:
+ `AWS.APG.CQRSES.QueryMicroservice` ASP.NET Core API project acts as the entry point where consumers interact with the service.
+ `AWS.APG.CQRSES.Query` .NET Core project is an object that hosts query-related objects and interfaces.
*CQRS AWS Serverless code solution directory*
![\[Solution directory showing both microservices and the event source expanded.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9f1bc700-def4-4201-bb2d-f1fa27404f15/images/23f8655c-95ad-422c-b20a-e29dc145e995.png)
This code is the AWS version of the on-premises code using AWS serverless services.
In C\$1 .NET Core, each Lambda function is represented by one .NET Core project. In this pattern's example code, there is a separate project for each interface in the command and query models.
**CQRS using AWS services**
You can find the root solution directory for CQRS using AWS serverless services is in the `CQRS AWS Serverless\CQRS`folder. The example includes two models: Customer and Reward.
The command Lambda functions for Customer and Reward are under `CQRS\Command Microservice\Customer` and `CQRS\Command Microservice\Reward` folders. They contain the following Lambda projects:
+ Customer command: `CommandCreateLambda`, `CommandDeleteLambda`, and `CommandUpdateLambda`
+ Reward command: `CommandAddRewardLambda` and `CommandRedeemRewardLambda`
The query Lambda functions for Customer and Reward are found under the `CQRS\Query Microservice\Customer` and `CQRS\QueryMicroservice\Reward`folders. They contain the `QueryCustomerListLambda` and `QueryRewardLambda` Lambda projects.
**CQRS test project**
The test project is under the `CQRS\Tests` folder. This project contains a test script to automate testing the CQRS Lambda functions.
**Event sourcing using AWS services**
The following Lambda event handlers are initiated by the Customer and Reward DynamoDB streams to process and synchronize the data in query tables.
+ The `EventSourceCustomer` Lambda function is mapped to the Customer table (`cqrses-customer-cmd`) DynamoDB stream.
+ The `EventSourceReward` Lambda function is mapped to the Reward table (`cqrses-reward-cmd`) DynamoDB stream.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/9f1bc700-def4-4201-bb2d-f1fa27404f15/attachments/attachment.zip)
# More patterns
**Topics**
+ [Access container applications privately on Amazon EKS using AWS PrivateLink and a Network Load Balancer](access-container-applications-privately-on-amazon-eks-using-aws-privatelink-and-a-network-load-balancer.md)
+ [Automate adding or updating Windows registry entries using AWS Systems Manager](automate-adding-or-updating-windows-registry-entries-using-aws-systems-manager.md)
+ [Automate cross-Region failover and failback by using DR Orchestrator Framework](automate-cross-region-failover-and-failback-by-using-dr-orchestrator-framework.md)
+ [Automatically build and deploy a Java application to Amazon EKS using a CI/CD pipeline](automatically-build-and-deploy-a-java-application-to-amazon-eks-using-a-ci-cd-pipeline.md)
+ [Automatically build CI/CD pipelines and Amazon ECS clusters for microservices using AWS CDK](automatically-build-ci-cd-pipelines-and-amazon-ecs-clusters-for-microservices-using-aws-cdk.md)
+ [Back up and archive mainframe data to Amazon S3 using BMC AMI Cloud Data](back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.md)
+ [Build a Micro Focus Enterprise Server PAC with Amazon EC2 Auto Scaling and Systems Manager](build-a-micro-focus-enterprise-server-pac-with-amazon-ec2-auto-scaling-and-systems-manager.md)
+ [Build an enterprise data mesh with Amazon DataZone, AWS CDK, and AWS CloudFormation](build-enterprise-data-mesh-amazon-data-zone.md)
+ [Containerize mainframe workloads that have been modernized by Blu Age](containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.md)
+ [Convert and unpack EBCDIC data to ASCII on AWS by using Python](convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.md)
+ [Convert mainframe data files with complex record layouts using Micro Focus](convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.md)
+ [Create a portal for micro-frontends by using AWS Amplify, Angular, and Module Federation](create-amplify-micro-frontend-portal.md)
+ [Deploy containers by using Elastic Beanstalk](deploy-containers-by-using-elastic-beanstalk.md)
+ [Emulate Oracle DR by using a PostgreSQL-compatible Aurora global database](emulate-oracle-dr-by-using-a-postgresql-compatible-aurora-global-database.md)
+ [Generate data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.md)
+ [Generate Db2 z/OS data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.md)
+ [Identify duplicate container images automatically when migrating to an Amazon ECR repository](identify-duplicate-container-images-automatically-when-migrating-to-ecr-repository.md)
+ [Implement AI-powered Kubernetes diagnostics and troubleshooting with K8sGPT and Amazon Bedrock integration](implement-ai-powered-kubernetes-diagnostics-and-troubleshooting-with-k8sgpt-and-amazon-bedrock-integration.md)
+ [Implement Microsoft Entra ID-based authentication in an AWS Blu Age modernized mainframe application](implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application.md)
+ [Implement path-based API versioning by using custom domains in Amazon API Gateway](implement-path-based-api-versioning-by-using-custom-domains.md)
+ [Incrementally migrate from Amazon RDS for Oracle to Amazon RDS for PostgreSQL using Oracle SQL Developer and AWS SCT](incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.md)
+ [Integrate Stonebranch Universal Controller with AWS Mainframe Modernization](integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.md)
+ [Manage AWS Service Catalog products in multiple AWS accounts and AWS Regions](manage-aws-service-catalog-products-in-multiple-aws-accounts-and-aws-regions.md)
+ [Migrate an AWS member account from AWS Organizations to AWS Control Tower](migrate-an-aws-member-account-from-aws-organizations-to-aws-control-tower.md)
+ [Migrate and replicate VSAM files to Amazon RDS or Amazon MSK using Connect from Precisely](migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.md)
+ [Migrate from SAP ASE to Amazon RDS for SQL Server using AWS DMS](migrate-from-sap-ase-to-amazon-rds-for-sql-server-using-aws-dms.md)
+ [Migrate Oracle external tables to Amazon Aurora PostgreSQL-Compatible](migrate-oracle-external-tables-to-amazon-aurora-postgresql-compatible.md)
+ [Modernize the CardDemo mainframe application by using AWS Transform](modernize-carddemo-mainframe-app.md)
+ [Modernize and deploy mainframe applications using AWS Transform and Terraform](modernize-mainframe-app-transform-terraform.md)
+ [Modernize mainframe batch printing workloads on AWS by using Rocket Enterprise Server and LRS VPSX/MFI](modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.md)
+ [Modernize mainframe online printing workloads on AWS by using Micro Focus Enterprise Server and LRS VPSX/MFI](modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.md)
+ [Modernize mainframe output management on AWS by using Rocket Enterprise Server and LRS PageCenterX](modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.md)
+ [Move mainframe files directly to Amazon S3 using Transfer Family](move-mainframe-files-directly-to-amazon-s3-using-transfer-family.md)
+ [Optimize multi-account serverless deployments by using the AWS CDK and GitHub Actions workflows](optimize-multi-account-serverless-deployments.md)
+ [Optimize the performance of your AWS Blu Age modernized application](optimize-performance-aws-blu-age-modernized-application.md)
+ [Automate blue/green deployments of Amazon Aurora global databases by using IaC principles](p-automate-blue-green-deployments-aurora-global-databases-iac.md)
+ [Replicate mainframe databases to AWS by using Precisely Connect](replicate-mainframe-databases-to-aws-by-using-precisely-connect.md)
+ [Run Amazon ECS tasks on Amazon WorkSpaces with Amazon ECS Anywhere](run-amazon-ecs-tasks-on-amazon-workspaces-with-amazon-ecs-anywhere.md)
+ [Send telemetry data from AWS Lambda to OpenSearch for real-time analytics and visualization](send-telemetry-data-from-lambda-to-opensearch-for-analytics-visualization.md)
+ [Set up CloudFormation drift detection in a multi-Region, multi-account organization](set-up-aws-cloudformation-drift-detection-in-a-multi-region-multi-account-organization.md)
+ [Structure a Python project in hexagonal architecture using AWS Lambda](structure-a-python-project-in-hexagonal-architecture-using-aws-lambda.md)
+ [Test AWS infrastructure by using LocalStack and Terraform Tests](test-aws-infra-localstack-terraform.md)
+ [Transform Easytrieve to modern languages by using AWS Transform custom](transform-easytrieve-modern-languages.md)
+ [Upgrade SAP Pacemaker clusters from ENSA1 to ENSA2](upgrade-sap-pacemaker-clusters-from-ensa1-to-ensa2.md)
+ [Use Amazon Q Developer as a coding assistant to increase your productivity](use-q-developer-as-coding-assistant-to-increase-productivity.md)
+ [Validate Account Factory for Terraform (AFT) code locally](validate-account-factory-for-terraform-aft-code-locally.md)
# Mainframes
**Topics**
+ [Access AWS services from IBM z/OS by installing the AWS CLI](access-aws-services-from-ibm-z-os-by-installing-aws-cli.md)
+ [Back up and archive mainframe data to Amazon S3 using BMC AMI Cloud Data](back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.md)
+ [Build COBOL Db2 programs by using AWS Mainframe Modernization and AWS CodeBuild](build-cobol-db2-programs-mainframe-modernization-codebuild.md)
+ [Build a Micro Focus Enterprise Server PAC with Amazon EC2 Auto Scaling and Systems Manager](build-a-micro-focus-enterprise-server-pac-with-amazon-ec2-auto-scaling-and-systems-manager.md)
+ [Build an advanced mainframe file viewer in the AWS Cloud](build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.md)
+ [Containerize mainframe workloads that have been modernized by Blu Age](containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.md)
+ [Convert and unpack EBCDIC data to ASCII on AWS by using Python](convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.md)
+ [Convert mainframe files from EBCDIC format to character-delimited ASCII format in Amazon S3 using AWS Lambda](convert-mainframe-files-from-ebcdic-format-to-character-delimited-ascii-format-in-amazon-s3-using-aws-lambda.md)
+ [Convert mainframe data files with complex record layouts using Micro Focus](convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.md)
+ [Deploy an environment for containerized Blu Age applications by using Terraform](deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.md)
+ [Generate Db2 z/OS data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.md)
+ [Generate data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.md)
+ [Implement Microsoft Entra ID-based authentication in an AWS Blu Age modernized mainframe application](implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application.md)
+ [Integrate Stonebranch Universal Controller with AWS Mainframe Modernization](integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.md)
+ [Migrate and replicate VSAM files to Amazon RDS or Amazon MSK using Connect from Precisely](migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.md)
+ [Modernize the CardDemo mainframe application by using AWS Transform](modernize-carddemo-mainframe-app.md)
+ [Modernize and deploy mainframe applications using AWS Transform and Terraform](modernize-mainframe-app-transform-terraform.md)
+ [Modernize mainframe output management on AWS by using Rocket Enterprise Server and LRS PageCenterX](modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.md)
+ [Modernize mainframe batch printing workloads on AWS by using Rocket Enterprise Server and LRS VPSX/MFI](modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.md)
+ [Mainframe modernization: DevOps on AWS with Rocket Software Enterprise Suite](mainframe-modernization-devops-on-aws-with-micro-focus.md)
+ [Modernize mainframe online printing workloads on AWS by using Micro Focus Enterprise Server and LRS VPSX/MFI](modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.md)
+ [Move mainframe files directly to Amazon S3 using Transfer Family](move-mainframe-files-directly-to-amazon-s3-using-transfer-family.md)
+ [Optimize the performance of your AWS Blu Age modernized application](optimize-performance-aws-blu-age-modernized-application.md)
+ [Secure and streamline user access in a Db2 federation database on AWS by using trusted contexts](secure-and-streamline-user-access-in-a-db2-federation-database-on-aws-by-using-trusted-contexts.md)
+ [Transfer large-scale Db2 z/OS data to Amazon S3 in CSV files](transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.md)
+ [Transform Easytrieve to modern languages by using AWS Transform custom](transform-easytrieve-modern-languages.md)
+ [More patterns](mainframe-more-patterns-pattern-list.md)
# Access AWS services from IBM z/OS by installing the AWS CLI
*Souma Ghosh, Paulo Vitor Pereira, and Phil de Valence, Amazon Web Services*
## Summary
The [AWS Command Line Interface (AWS CLI)](https://aws.amazon.com/cli/) is an open source tool for managing multiple AWS services by using commands in a command line shell. With minimal configuration, you can run commands from command line sessions such as the command prompt, terminal, and bash shell to implement functionality that's equivalent to that provided by the browser-based AWS Management Console.
All AWS infrastructure as a service (IaaS) administration, management, and access functions in the AWS Management Console are available in the AWS API and AWS CLI. You can install the AWS CLI on an IBM z/OS mainframe to directly access, manage, and interact with AWS services from z/OS. The AWS CLI enables users and applications to perform various tasks, such as:
+ Transferring files or datasets between z/OS and Amazon Simple Storage Service (Amazon S3) object storage and viewing content of buckets
+ Starting and stopping different AWS resources; for example, starting a batch job in an AWS Mainframe Modernization environment
+ Calling an AWS Lambda function to implement common business logic
+ Integrating with artificial intelligence and machine learning (AI/ML) and analytics services
This pattern describes how to install, configure, and use the AWS CLI on z/OS. You can install it globally, so it's available to all z/OS users, or at a user level. The pattern also details how to use the AWS CLI in an interactive command line session from z/OS Unix System Services (USS) or as a batch job.
## Prerequisites and limitations
**Prerequisites**
+ **Network communication from z/OS to AWS**
By default, the AWS CLI sends requests to AWS services by using HTTPS on TCP port 443. To use the AWS CLI successfully, you must be able to make outbound connections on TCP port 443. You can use any of the following z/OS USS commands (some of these might not be installed in your environment) to test network connectivity from z/OS to AWS:
```
ping amazonaws.com
dig amazonaws.com
traceroute amazonaws.com
curl -k https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-welcome.html
```
+ **AWS credentials**
In order to communicate with AWS Cloud services from z/OS, the AWS CLI requires you to configure some credentials with privileges to access the target AWS account. For programmatic commands to AWS, you can use access keys, which consist of an access key ID and secret access key. If you don't have access keys, you can create them from the AWS Management Console. As a best practice, do not use the access keys for the AWS account root user for any task unless the root user is required. Instead, [create a new administrator IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-set-up.html#create-an-admin) and [prepare for least-privilege permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-set-up.html#LeastPrivilege)** **to set up the user with access keys. After you create the user, you can [create an access key ID and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for this user.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html)
+ **IBM Python for z/OS**
The AWS CLI requires Python 3.8 or later. IBM has enabled Python to run on z/OS with [IBM Open Enterprise Python for z/OS](https://www.ibm.com/products/open-enterprise-python-zos). IBM Open Enterprise Python is available at no charge through Shopz SMP/E, or you can download the PAX file from the [IBM website](https://www.ibm.com/account/reg/signup?formid=urx-49465). For instructions, see the [installation and configuration documentation](https://www.ibm.com/docs/en/python-zos) for IBM Open Enterprise Python for z/OS.
**Limitations**
+ The installation instructions provided in this pattern are applicable to **AWS CLI version 1 only**. The latest version of the AWS CLI is version 2. However, this pattern uses the older version because the installation methods are different for version 2, and the binary executables available for version 2 aren't compatible with the z/OS system.
**Product versions**
+ AWS CLI version 1
+ Python 3.8 or later
## Architecture
**Technology stack**
+ Mainframe running z/OS
+ Mainframe z/OS UNIX System Services (USS)
+ Mainframe Open MVS (OMVS) – z/OS UNIX shell environment command interface
+ Mainframe disk, such as a direct-access storage device (DASD)
+ AWS CLI
**Target architecture**
The following diagram shows an AWS CLI deployment on IBM z/OS. You can invoke the AWS CLI from an interactive user session, such as SSH, and telnet sessions. You can also invoke it from a batch job by using job control language (JCL), or from any program that can call a z/OS Unix shell command.
![\[AWS CLI on an IBM z/OS mainframe accessing AWS services.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4e3188d8-287f-4ced-8c29-80a01cbbdf50/images/c3883500-bd00-4c56-982a-26d5e0b8b093.png)
The AWS CLI communicates with AWS service endpoints over a TCP/IP network. This network connection can happen over the internet or through a private AWS Direct Connect connection from the customer data center to AWS Cloud data centers. The communication is authenticated with AWS credentials and encrypted.
**Automation and scale**
You can explore the capabilities of an AWS service with the AWS CLI and develop USS shell scripts to manage your AWS resources from z/OS. You can also run AWS CLI commands and shell scripts from the z/OS batch environment, and you can automate batch jobs to run on a specific schedule by integrating with mainframe schedulers. AWS CLI commands or scripts can be coded inside parameters (PARMs) and procedures (PROCs), and can be scaled by following the standard approach of calling the PARM or PROC from different batch jobs with different parameters.
## Tools
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell.
## Best practices
+ For security reasons, restrict the access permissions to the USS directory where the AWS access key details are stored. Allow access to only the users or programs that use the AWS CLI.
+ Do not use the AWS account root user access keys for any task. Instead, [create a new administrator IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-set-up.html#create-an-admin) for yourself and set it up with access keys.
|
|
| IAM users have long-term credentials that present a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. |
| --- |
## Epics
### Install AWS CLI version 1 on z/OS USS
| Task | Description | Skills required |
| --- | --- | --- |
| Install Python 3.8 or later. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | Mainframe z/OS administrator |
| Set USS environment variables. | Add environment variables to the profile. You can add these either to the `/u/cliuser/.profile` file for an individual user (`cliuser`) or to the `/etc/profile` file for all users.This pattern assumes that Python has been installed in the `/u/awscli/python` directory. If your installation directory is different, update the code accordingly.
| Mainframe z/OS administrator |
| Test the Python installation. | Run the **python** command:
python --version
The output should confirm that you have Python 3.8 or later installed correctly. | Mainframe z/OS administrator |
| Verify or install **pip**. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | Mainframe z/OS administrator |
| Install AWS CLI version 1. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | Mainframe z/OS administrator |
### Configure AWS CLI access from z/OS
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the AWS access keys, default Region, and output. | The [AWS CLI documentation](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) describes different options for setting up AWS access. You can choose a configuration according to your organization's standards. This example uses the short-term credential configuration.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | AWS administrator, Mainframe z/OS administrator, Mainframe z/OS developer |
| Test the AWS CLI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | Mainframe z/OS administrator, Mainframe z/OS developer |
### Option 1 ‒ Transfer data from USS to Amazon S3 interactively from a USS session
| Task | Description | Skills required |
| --- | --- | --- |
| Download and transfer the sample CSV file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | App developer, Mainframe z/OS developer |
| Create an S3 bucket and upload the CSV file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | App developer, Mainframe z/OS developer |
| View the S3 bucket and uploaded file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html)For more information about uploading objects, see [Getting started with Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html) in the Amazon S3 documentation. | General AWS |
| Run a SQL query on an Amazon Athena table. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html)The output of the SQL query will display the contents of your CSV file. | General AWS, App developer |
### Option 2 ‒ Transfer data from USS to Amazon S3 by using batch JCL
| Task | Description | Skills required |
| --- | --- | --- |
| Upload the sample file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | Mainframe z/OS developer |
| Create batch JCL. | Code the batch JCL as follows to create the destination S3 bucket, upload the dataset, and list the bucket content. Make sure to replace the directory name, file names, and bucket name to your own values.
| Mainframe z/OS developer |
| Submit the batch JCL job. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | Mainframe z/OS developer |
| View the dataset uploaded to the S3 bucket. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-aws-services-from-ibm-z-os-by-installing-aws-cli.html) | General AWS |
## Related resources
+ [AWS CLI version 1 documentation](https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-welcome.html)
+ [AWS Mainframe Modernization CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/m2/)
+ [AWS Mainframe Modernization](https://aws.amazon.com/mainframe-modernization/)
## Additional information
**USER.DATA.FIXED in ISPF option 3.4 (dataset list utility)**
![\[Viewing the contents of the dataset in z/OS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4e3188d8-287f-4ced-8c29-80a01cbbdf50/images/96c25145-3d4d-4007-99f6-5eeb9e88642d.png)
**SYSOUT of the submitted batch job**
![\[Standard output from job log.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4e3188d8-287f-4ced-8c29-80a01cbbdf50/images/03fffbd2-7d2b-43b2-bf14-736b3d150e38.png)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/4e3188d8-287f-4ced-8c29-80a01cbbdf50/attachments/attachment.zip)
# Back up and archive mainframe data to Amazon S3 using BMC AMI Cloud Data
*Santosh Kumar Singh, Gilberto Biondo, and Maggie Li, Amazon Web Services*
*Mikhael Liberman, Model9 Mainframe Software*
## Summary
This pattern demonstrates how to back up and archive mainframe data directly to Amazon Simple Storage Service (Amazon S3), and then recall and restore that data to the mainframe by using BMC AMI Cloud Data (previously known as Model9 Manager). If you are looking for a way to modernize your backup and archive solution as part of a mainframe modernization project or to meet compliance requirements, this pattern can help meet those goals.
Typically, organizations that run core business applications on mainframes use a virtual tape library (VTL) to back up data stores such as files and logs. This method can be expensive because it consumes billable MIPS, and the data stored on tapes outside the mainframe is inaccessible. To avoid these issues, you can use BMC AMI Cloud Data to quickly and cost-effectively transfer operational and historical mainframe data directly to Amazon S3. You can use BMC AMI Cloud Data to back up and archive data over TCP/IP to AWS while taking advantage of IBM z Integrated Information Processor (zIIP) engines to reduce cost, parallelism, and transfer times.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ BMC AMI Cloud Data with a valid license key
+ TCP/IP connectivity between the mainframe and AWS
+ An AWS Identity and Access Management (IAM) role for read/write access to an S3 bucket
+ Mainframe security product (RACF) access in place to run BMC AMI Cloud processes
+ A BMC AMI Cloud z/OS agent (Java version 8 64-bit SR5 FP16 or later) that has available network ports, firewall rules permitting access to S3 buckets, and a dedicated z/FS file system
+ [Requirements](https://docs.bmc.com/docs/cdacv27/management-server-requirements-1245343255.html) met for the BMC AMI Cloud management server
**Limitations**
+ BMC AMI Cloud Data stores its operational data in a PostgreSQL database that runs as a Docker container on the same Amazon Elastic Compute Cloud (Amazon EC2) instance as the management server. Amazon Relational Database Service (Amazon RDS) is not currently supported as a backend for BMC AMI Cloud Data. For more information about the latest product updates, see [What's New?](https://docs.bmc.com/docs/cdacv27/what-s-new-1245343246.html) in the BMC documentation.
+ This pattern backs up and archives z/OS mainframe data only. BMC AMI Cloud Data backs up and archives only mainframe files.
+ This pattern doesn’t convert data into standard open formats such as JSON or CSV. Use an additional transformation service such as [BMC AMI Cloud Analytics](https://www.bmc.com/it-solutions/bmc-ami-cloud-analytics.html) (previously known as Model9 Gravity) to convert the data into standard open formats. Cloud-native applications and data analytics tools can access the data after it's is written to the cloud.
**Product versions**
+ BMC AMI Cloud Data version 2.x
## Architecture
**Source technology stack**
+ Mainframe running z/OS
+ Mainframe files such as datasets and z/OS UNIX System Services (USS) files
+ Mainframe disk, such as a direct-access storage device (DASD)
+ Mainframe tape (virtual or physical tape library)
**Target technology stack**
+ Amazon S3
+ Amazon EC2 instance in a virtual private cloud (VPC)
+ AWS Direct Connect
+ Amazon Elastic File System (Amazon EFS)
**Target architecture**
The following diagram shows a reference architecture where BMC AMI Cloud Data software agents on a mainframe drive the legacy data backup and archive processes that store the data in Amazon S3.
![\[BMC AMI Cloud Data software agents on a mainframe driving legacy data backup and archive processes\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bde3b029-184e-4eb0-933b-f8caf6cc40ab/images/a24cd6c1-b131-49ea-8238-f3aea5ab8134.png)
The diagram shows the following workflow:
1. BMC AMI Cloud Data software agents run on mainframe logical partitions (LPARs). The software agents read and write mainframe data from DASD or tape directly to Amazon S3 over TCP/IP.
1. AWS Direct Connect sets up a physical, isolated connection between the on-premises network and AWS. For enhanced security, run a site-to-site VPN on top of Direct Connect to encrypt data in transit.
1. The S3 bucket stores mainframe files as object storage data, and BMC AMI Cloud Data agents directly communicate with the S3 buckets. Certificates are used for HTTPS encryption of all communications between the agent and Amazon S3. Amazon S3 data encryption is used to encrypt and protect the data at rest.
1. BMC AMI Cloud Data management servers run as Docker containers on EC2 instances. The instances communicate with agents running on mainframe LPARs and S3 buckets.
1. Amazon EFS is mounted on both active and passive EC2 instances to share the Network File System (NFS) storage. This is to make sure that metadata related to a policy created on the management server isn't lost in the event of a failover. In the event of a failover by the active server, the passive server can be accessed without any data loss. If the passive server fails, the active server can be accessed without any data loss.
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve nearly any amount of data.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a AWS Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
**BMC tools**
+ [BMC AMI Cloud management server](https://docs.bmc.com/docs/cdacv27/bmc-ami-cloud-overview-1245343249.html) is a GUI application that runs as a Docker container on an Amazon Linux Amazon Machine Image (AMI) for Amazon EC2. The management server provides the functionality to manage BMC AMI Cloud activities such as reporting, creating and managing policies, running archives, and performing backups, recalls, and restores.
+ [BMC AMI Cloud agent](https://docs.bmc.com/docs/cdacv27/bmc-ami-cloud-overview-1245343249.html) runs on an on-premises mainframe LPAR that reads and writes files directly to object storage by using TCP/IP. A started task runs on a mainframe LPAR and is responsible for reading and writing backup and archive data to and from Amazon S3.
+ [BMC AMI Cloud Mainframe Command Line Interface (M9CLI)](https://docs.bmc.com/docs/cdacv27/command-line-interface-cli-reference-1245343519.html) provides you with a set of commands to perform BMC AMI Cloud actions directly from TSO/E or in batch operations, without the dependency on the management server.
## Epics
### Create an S3 bucket and IAM policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to store the files and volumes that you want to back up and archive from your mainframe environment. | General AWS |
| Create an IAM policy. | All BMC AMI Cloud management servers and agents require access to the S3 bucket that you created in the previous step.To grant the required access, create the following IAM policy:
| General AWS |
### Get the BMC AMI Cloud software license and download the software
| Task | Description | Skills required |
| --- | --- | --- |
| Get a BMC AMI Cloud software license. | To get a software license key, contact the [BMC AMI Cloud team](https://www.bmc.com/it-solutions/bmc-ami-cloud.html?vd=model9-io). The output of the z/OS `D M=CPU` command is required for generating a license. | Build lead |
| Download the BMC AMI Cloud software and license key. | Obtain the installation files and license key by following the instructions in the [BMC documentation](https://docs.bmc.com/docs/cdacv27/preparing-to-install-the-bmc-ami-cloud-agent-1245343285.html). | Mainframe infrastructure administrator |
### Install the BMC AMI Cloud software agent on the mainframe
| Task | Description | Skills required |
| --- | --- | --- |
| Install the BMC AMI Cloud software agent. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html) | Mainframe infrastructure administrator |
### Set up a BMC AMI Cloud management server on an EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create Amazon EC2 Linux 2 instances. | Launch two Amazon EC2 Linux 2 instances in different Availability Zones by following the instructions from [Step 1: Launch an instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2_GetStarted.html#ec2-launch-instance) in the Amazon EC2 documentation.The instance must meet the following recommended hardware and software requirements:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html)For more information, see the [BMC documentation](https://docs.bmc.com/docs/cdacv27/preparing-to-install-the-management-server-on-linux-1245343268.html). | Cloud architect, Cloud administrator |
| Create an Amazon EFS file system. | Create an Amazon EFS file system by following the instructions from [Step 1: Create your Amazon EFS file system](https://docs.aws.amazon.com/efs/latest/ug/gs-step-two-create-efs-resources.html) in the Amazon EFS documentation.When creating the file system, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html) | Cloud administrator, Cloud architect |
| Install Docker and configure the management server. | **Connect to your EC2 instances:**Connect to your EC2 instances by following the instructions from [Connect to your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html) in the Amazon EC2 documentation.**Configure your EC2 instances:**For each EC2 instance, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html) | Cloud architect, Cloud administrator |
| Install the management server software. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html)To troubleshoot issues, go to the logs stored in the `/data/model9/logs/`** **folder. For more information, see the [BMC documentation](https://docs.bmc.com/docs/cdacv27/performing-the-management-server-installation-on-linux-1245343272.html). | Cloud architect, Cloud administrator |
### Add an agent and define a backup or archive policy on the BMC AMI Cloud management server
| Task | Description | Skills required |
| --- | --- | --- |
| Add a new agent. | Before you add a new agent, confirm the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html)You must create an agent on the management server before you define any backup and archive policies. To create the agent, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html)After the agent is created, you'll see the **connected** status against the object storage and mainframe agent in a new window that appears in the table. | Mainframe storage administrator or developer |
| Create a backup or archive policy. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html) | Mainframe storage administrator or developer |
### Run the backup or archive policy from the management server
| Task | Description | Skills required |
| --- | --- | --- |
| Run the backup or archive policy. | Run the data backup or archive policy that you created earlier from the management server either manually or automatically (based on a schedule). To run the policy manually:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html) | Mainframe storage administrator or developer |
| Restore the backup or archive policy. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html) | Mainframe storage administrator or developer |
### Run the backup or archive policy from the mainframe
| Task | Description | Skills required |
| --- | --- | --- |
| Run the backup or archive policy by using M9CLI. | Use the M9CLI to perform backup and restore processes from TSO/E, REXX, or through JCLs without setting up rules on the BMC AMI Cloud management server.**Using TSO/E:**If you use TSO/E, make sure that `M9CLI REXX` is concatenated to `TSO`. To back up a dataset through TSO/E, use the `TSO M9CLI BACKDSN ` command.For more information about M9CLI commands, see [CLI reference ](https://docs.bmc.com/docs/cdacv27/command-line-interface-cli-reference-1245343519.html)in the BMC documentation.**Using JCLs:**To run the backup and archive policy by using JCLs, run the `M9CLI` command.**Using batch operations:**The following example shows you how to archive a dataset by running the `M9CLI` command in batch:
| Mainframe storage administrator or developer |
| Run the backup or archive policy in JCL batch. | BMC AMI Cloud provides a sample JCL routine called **M9SAPIJ**. You can customize **M9SAPIJ** to run a specific policy created on the management server with a JCL. This job can also be part of a batch scheduler for running backup and restore processes automatically.The batch job expects the following mandatory values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.html)You can also change other values by following the instructions on the sample job. | Mainframe storage administrator or developer |
## Related resources
+ [Mainframe Modernization with AWS](https://aws.amazon.com/mainframe/) (AWS documentation)
+ [How Cloud Backup for Mainframes Cuts Costs with Model9 and AWS](https://aws.amazon.com/blogs/apn/how-cloud-backup-for-mainframes-cuts-costs-with-model9-and-aws/) (AWS Partner Network Blog)
+ [How to Enable Mainframe Data Analytics on AWS Using Model9](https://aws.amazon.com/blogs/apn/how-to-enable-mainframe-data-analytics-on-aws-using-model9/) (AWS Partner Network Blog)
+ [AWS Direct Connect Resiliency Recommendations](https://aws.amazon.com/directconnect/resiliency-recommendation/?nc=sn&loc=4&dn=2) (AWS documentation)
+ [BMC AMI Cloud documentation](https://docs.bmc.com/docs/cdacv27/getting-started-1245343248.html) (BMC website)
# Build COBOL Db2 programs by using AWS Mainframe Modernization and AWS CodeBuild
*Luis Gustavo Dantas and Eduardo Zimelewicz, Amazon Web Services*
## Summary
**Note**
AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
This pattern explains how to create a simple AWS CodeBuild project to precompile and bind COBOL Db2 programs by using the AWS Mainframe Modernization Replatform tools. This enables the deployment and execution of these programs in the AWS Mainframe Modernization Replatform runtime environment.
COBOL, a business-oriented programming language, powers many critical applications due to its reliability and readability. IBM Db2, a relational database management system, manages large volumes of data efficiently and integrates with COBOL programs through SQL. Together, COBOL and Db2 form the backbone of mission-critical operations in industries such as finance and government, despite the emergence of newer technologies.
Migrating COBOL and Db2 components from the mainframe environment to other platforms leads to challenges such as platform compatibility, integration complexity, data migration, and performance optimization. Moving these critical components requires careful planning, technical expertise, and resources to ensure a smooth migration while maintaining reliability and functionality.
The AWS Mainframe Modernization service provides tools and resources to replatform mainframe applications and databases to run on AWS infrastructure, such as Amazon Elastic Compute Cloud (Amazon EC2) instances. This involves moving mainframe workloads to the cloud without major code changes.
The Db2 precompile and bind process is essential for optimizing the performance and reliability of database applications. Precompilation transforms embedded SQL statements into executable code, which reduces runtime overhead and enhances efficiency. The bind process links the precompiled code with database structures, facilitating access paths and query optimization. This process ensures data integrity, improves application responsiveness, and guards against security vulnerabilities. Properly precompiled and bound applications minimize resource consumption, enhance scalability, and mitigate the risks of SQL injection attacks.
## Prerequisites and limitations
**Prerequisites **
+ An AWS account and administrative-level console access.
+ An IBM Db2 database system, such as IBM Db2 for z/OS or Db2 for Linux, Unix, and Windows (LUW).
+ The IBM Data Server Client software, which is available for download from the [IBM website](https://www.ibm.com/support/pages/download-initial-version-115-clients-and-drivers). For more information, see [IBM Data Server Client and Data Server Driver types](https://www.ibm.com/docs/en/db2/11.5?topic=overviews-data-server-clients).
+ A COBOL Db2 program to be compiled and bound. Alternatively, this pattern provides a basic sample program that you can use.
+ A virtual private cloud (VPC) on AWS with a private network. For information about creating a VPC, see the [Amazon Virtual Private Cloud (Amazon VPC) documentation](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html).
+ A source control repository such as GitHub or GitLab.
**Limitations **
+ For AWS CodeBuild quotas, see [Quotas for AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/limits.html).
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
## Architecture
**Source technology stack**
The source stack includes:
+ COBOL programs that use a Db2 database to store data
+ IBM COBOL compiler and Db2 for z/OS precompiler
+ Other parts of the mainframe setup, such as the file system, transaction manager, and spool
**Target technology stack**
This pattern's approach works for two options: moving data from Db2 for z/OS to Db2 for LUW, or staying on Db2 for z/OS. The target architecture includes:
+ COBOL programs that use a Db2 database to store data
+ AWS Mainframe Modernization Replatform compilation tools
+ AWS CodeBuild as the infrastructure to build the application
+ Other AWS Cloud resources such as Amazon Linux
**Target architecture**
![\[Architecture for building COBOL Db2 programs on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5895fa34-f05b-4cc3-a59f-a596f9116c66/images/0dda414a-21a7-41d1-b86b-7ff3b1c6fbda.png)
The diagram illustrates the following:
1. The user uploads their code to a source control repository such as GitHub or GitLab.
1. AWS CodePipeline notices the change and gets the code from the repository.
1. CodePipeline starts AWS CodeBuild and sends the code.
1. CodeBuild follows the instructions in the `buildspec.yml` template (provided in the [Additional information](#build-cobol-db2-programs-mainframe-modernization-codebuild-additional) section) to:
1. Get the IBM Data Server Client from an Amazon Simple Storage Service (Amazon S3) bucket.
1. Install and set up the IBM Data Server Client.
1. Retrieve Db2 credentials from AWS Secrets Manager.
1. Connect to the Db2 server.
1. Precompile, compile, and bind the COBOL program.
1. Save the finished products in an S3 bucket for AWS CodeDeploy to use.
1. CodePipeline starts CodeDeploy.
1. CodeDeploy coordinates its agents, which are already installed in the runtime environments. The agents fetch the application from Amazon S3 and install it based on the instructions in `appspec.yml`.
To keep things simple and focused on the build, the instructions in this pattern cover steps 1 through 4 but don't include the deployment of the COBOL Db2 program.
**Automation and scale**
For simplicity, this pattern describes how to provision resources manually. However, there are numerous automation options available, such as CloudFormation, AWS Cloud Development Kit (AWS CDK), and HashiCorp Terraform, which automate these tasks. For more information, see the [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) and [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html) documentation.
## Tools
**AWS services**
+ [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html) is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.
+ [AWS CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html) automates deployments to Amazon EC2 or on-premises instances, AWS Lambda functions, or Amazon Elastic Container Service (Amazon ECS) services.
+ [AWS CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html) helps you quickly model and configure the different stages of a software release and automate the steps required to release software changes continuously.
+ [AWS Mainframe Modernization](https://docs.aws.amazon.com/m2/latest/userguide/what-is-m2.html) provides tools and resources to help you plan and implement migration and modernization from mainframes to AWS managed runtime environments.
**Other tools**
+ **Amazon ECR image for AWS Mainframe Modernization Replatform tools**. To compile a COBOL application, you'll need to initiate CodeBuild by using an Amazon Elastic Container Registry (Amazon ECR) image that contains the AWS Mainframe Modernization Replatform tools:
`673918848628.dkr.ecr..amazonaws.com/m2-enterprise-build-tools:9.0.7.R1`
For more information about the ECR image available, see the [tutorial](https://docs.aws.amazon.com/m2/latest/userguide/tutorial-build-mf.html) in the *AWS Mainframe Modernization User Guide*.
+ [IBM Data Server Client](https://www.ibm.com/docs/en/db2/11.5?topic=overviews-data-server-clients) software is essential for precompiling and binding COBOL Db2 programs in CodeBuild. It acts as a bridge between the COBOL compiler and Db2.
## Best practices
+ Not every COBOL program relies on Db2 as its data persistence layer. Make sure that compilation directives for accessing Db2 are applied only to COBOL programs that are specifically designed to interact with Db2. Implement a logic to distinguish between COBOL Db2 programs and COBOL programs that do not use Db2.
+ We recommend that you avoid compiling programs that haven't been modified. Implement a process to identify which programs require compilation.
## Epics
### Create the cloud infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket to host the IBM Data Server Client and pipeline artifacts. | You need to set up an S3 bucket to (a) upload the IBM Data Server Client, (b) store your code from the repository, and (c) store the results of the build process.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html)For ways to create an S3 bucket, see the [Amazon S3 documentation.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) | General AWS |
| Upload the IBM Data Server Client to the S3 bucket. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html) | General AWS |
| Create an AWS Secrets Manager secret for your Db2 credentials. | To create a secret to securely store your DB2 credentials:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html)For more information about creating secrets, see the [Secrets Manager documentation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html). | General AWS |
| Verify that Db2 is accessible from the VPC subnet. | AWS CodeBuild needs a connection to the Db2 server so that the Data Server Client can perform precompilation and bind operations. Make sure that CodeBuild can reach the Db2 server through a secure connection.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html) | Network administrator, General AWS |
### Create the application artifacts
| Task | Description | Skills required |
| --- | --- | --- |
| Create the COBOL Db2 asset. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html) | App developer |
| Create the `buildspec.yml` file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html) | AWS DevOps |
| Connect your repository to CodePipeline. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html)You will need the Amazon Resource Name (ARN) for the connection when you create the AWS Identity and Access Management (IAM) policy for CodePipeline in a later step. | AWS DevOps |
### Configure permissions
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM policy for CodeBuild. | The CodeBuild project requires access to some resources, including Secrets Manager and Amazon S3.To set up the necessary permissions:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html)For more information about creating IAM policies, see the [IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html). | General AWS |
| Create an IAM role for CodeBuild. | To make the security policies available for CodeBuild, you need to configure an IAM role.To create this role:1. On the[ IAM console](https://console.aws.amazon.com/iam), in the navigation pane, choose **Roles**, **Create Role**.3. For **Trusted entity type**, keep the default **AWS service** setting.4. For **Use case**, select the CodeBuild service, and then choose **Next**.4. In the list of available IAM policies, locate the policy you created for CodeBuild, and then choose **Next** to attach it to the role.5. Specify a name for the role, and choose **Create role **to save it for future reference in CodeBuild.For more information about creating an IAM role for an AWS service, see the[ IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html). | General AWS |
| Create an IAM policy for CodePipeline. | The AWS CodePipeline pipeline requires access to some resources, including your code repository and Amazon S3.Repeat the steps provided previously for CodeBuild to create an IAM policy for CodePipeline (in step 2, choose **CodePipeline** instead of **CodeBuild**). | AWS DevOps |
| Create an IAM role for CodePipeline. | To make the security policies available for CodePipeline, you need to configure an IAM role.To create this role:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html) | AWS DevOps |
### Compile and bind the COBOL Db2 program
| Task | Description | Skills required |
| --- | --- | --- |
| Create a CodePipeline pipeline and CodeBuild project. | To create a CodePipeline pipeline and the CodeBuild project that compiles and binds the COBOL Db2 program:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-cobol-db2-programs-mainframe-modernization-codebuild.html) | AWS DevOps |
| Review the output. | Verify the success of the build by reviewing the CodePipeline build logs. | AWS DevOps |
| Check results in Db2. | Verify the package version on the SYSPLAN table.
select CAST(NAME AS VARCHAR(10)) as name, VALIDATE, LAST_BIND_TIME, LASTUSED, CAST(PKGVERSION AS VARCHAR(10)) as PKGVERSION from SYSIBM.SYSPLAN where NAME = 'CDB2SMP' order by LAST_BIND_TIME desc
The version must match the CodeBuild build ID, which is `CDB2SMP` in our example:
NAME VALIDATE LAST_BIND_TIME LASTUSED PKGVERSION ---------- -------- -------------------------- ---------- ---------- CDB2SMP B 2024-05-18-11.53.11.503738 01/01/0001 19
| |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Occasionally, the AWS console switches Regions when you move between services. | Make sure to verify the selected AWS Region whenever you switch between services.The AWS Region selector is in the upper-right corner of the console window. |
| It can be difficult to identify Db2 connectivity issues from CodeBuild. | To troubleshoot connectivity problems, add the following DB2 connect command to the `buildspec.yml` file. This addition helps you debug and resolve connectivity issues.
db2 connect to $DB_NAME user $DB2USER using $DB2PASS
|
| Occasionally, the role pane in the IAM console doesn't immediately show the IAM policy you've created. | If you encounter a delay, refresh the screen to display the latest information. |
## Related resources
**IBM documentation**
+ [IBM Data Server Client and driver types](https://www.ibm.com/docs/en/db2/11.5?topic=overviews-data-server-clients)
+ [Download IBM Data Server Client and driver types](https://www.ibm.com/support/pages/download-initial-version-115-clients-and-drivers)
**AWS documentation**
+ [Amazon S3 User Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)
+ [AWS CodeBuild User Guide](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html)
+ [AWS Mainframe Modernization User Guide](https://docs.aws.amazon.com/m2/latest/userguide/what-is-m2.html)
+ [AWS Secrets Manager User Guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)
+ [AWS CodePipeline User Guide](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html)
+ [AWS CodeDeploy User Guide** **](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-codedeploy.html)
## Additional information
**CodeBuild policy**
Replace the placeholders ``, ``, ``, ``, and `` with your values.
```
{"Version": "2012-10-17",
"Statement": [
{"Action": "ecr:GetAuthorizationToken", "Effect": "Allow", "Resource": "*" },
{"Action": ["ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"],
"Effect": "Allow",
"Resource": "arn:aws:ecr:*:673918848628:repository/m2-enterprise-build-tools"},
{"Action": "s3:PutObject", "Effect": "Allow", "Resource": "arn:aws:s3:::aws-m2-repo-*/*"},
{"Action": ["logs:PutLogEvents", "logs:CreateLogStream", "logs:CreateLogGroup"],
"Effect": "Allow", "Resource": "arn:aws:logs:::*"},
{"Action": ["ec2:DescribeVpcs", "ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups", "ec2:DescribeNetworkInterfaces",
"ec2:DescribeDhcpOptions", "ec2:DeleteNetworkInterface",
"ec2:CreateNetworkInterface"],
"Effect": "Allow", "Resource": "*"},
{"Action": "ec2:CreateNetworkInterfacePermission",
"Effect": "Allow", "Resource": [""]},
{"Action": "s3:*", "Effect": "Allow", "Resource": ["/*",""]},
{"Action": "secretsmanager:GetSecretValue",
"Effect": "Allow", "Resource": ""}
]
}
```
**CodePipeline policy**
Replace the placeholders `` and `` with your values.
```
{
"Version": "2012-10-17",
"Statement": [
{"Action": ["s3:List*", "s3:GetObjectVersion", "s3:GetObject", "s3:GetBucketVersioning" ],
"Effect": "Allow",
"Resource": ["/*", ""]},
{"Action": ["codebuild:StartBuild", "codebuild:BatchGetBuilds"],
"Effect": "Allow", "Resource": "*"},
{"Action": ["codestar-connections:UseConnection"],
"Effect": "Allow", "Resource": ""}
]
}
```
**`buildspec.yml`**
Replace the `` placeholder with your actual S3 bucket name.
```
version: 0.2
phases:
pre_build:
commands:
- /var/microfocuslicensing/bin/mfcesd -no > /var/microfocuslicensing/logs/mfcesd_startup.log 2>&1 &
- |
mkdir $CODEBUILD_SRC_DIR/db2client
aws s3 cp s3:///v11.5.8_linuxx64_client.tar.gz $CODEBUILD_SRC_DIR/db2client/ >> /dev/null 2>&1
tar -xf $CODEBUILD_SRC_DIR/db2client/v11.5.8_linuxx64_client.tar.gz -C $CODEBUILD_SRC_DIR/db2client/
cd $CODEBUILD_SRC_DIR/db2client/
./client/db2_install -f sysreq -y -b /opt/ibm/db2/V11.5 >> /dev/null 2>&1
useradd db2cli
/opt/ibm/db2/V11.5/instance/db2icrt -s client -u db2cli db2cli
DB2CRED=$(aws secretsmanager get-secret-value --secret-id dev-db2-cred | jq -r '.SecretString | fromjson')
read -r DB2USER DB2PASS DB_NODE DB_HOST DB_PORT DB_NAME DB_QUAL <<<$(echo $DB2CRED | jq -r '.username, .password, .db2node, .db2host, .db2port, .db2name, .qualifier')
. /home/db2cli/sqllib/db2profile
db2 catalog tcpip node $DB_NODE remote $DB_HOST server $DB_PORT
db2 catalog db $DB_NAME as $DB_NAME at node $DB_NODE authentication server
build:
commands:
- |
revision=$CODEBUILD_SRC_DIR/loadlib
mkdir -p $revision; cd $revision
. /opt/microfocus/EnterpriseDeveloper/bin/cobsetenv
cob -zU $CODEBUILD_SRC_DIR/CDB2SMP.cbl -C "DB2(DB==${DB_NAME} PASS==${DB2USER}.${DB2PASS} VERSION==${CODEBUILD_BUILD_NUMBER} COLLECTION==DB2AWSDB"
artifacts:
files:
- "**/*"
base-directory: $revision
```
# Build a Micro Focus Enterprise Server PAC with Amazon EC2 Auto Scaling and Systems Manager
*Kevin Yung and Krithika Palani Selvam, Amazon Web Services*
*Peter Woods, None*
*Abraham Rondon, Micro Focus*
## Summary
This pattern introduces a scalable architecture for mainframe applications using [Micro Focus Enterprise Server in Scale-Out Performance and Availability Cluster (PAC)](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-F6E1BBB7-AEC2-45B1-9E36-1D86B84D2B85.html) and an Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling group on Amazon Web Services (AWS). The solution is fully automated with AWS Systems Manager and Amazon EC2 Auto Scaling lifecycle hooks. By using this pattern, you can set up your mainframe online and batch applications to achieve high resiliency by automatically scaling in and out based on your capacity demands.
**Note**
This pattern was tested with Micro Focus Enterprise Server version 6.0. For version 8, see [Set up Micro Focus Runtime (on Amazon EC2)](https://docs.aws.amazon.com/m2/latest/userguide/mf-runtime-setup.html).
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ Micro Focus Enterprise Server software and license. For details, contact [Micro Focus sales](https://www.microfocus.com/en-us/contact/contactme).
+ An understanding of the concept of rebuilding and delivering a mainframe application to run in Micro Focus Enterprise Server. For a high-level overview, see [Micro Focus Enterprise Server Data Sheet](https://www.microfocus.com/media/data-sheet/enterprise_server_ds.pdf).
+ An understanding of the concepts in Micro Focus Enterprise Server scale-out Performance and Availability Cluster. For more information, see the [Micro Focus Enterprise Server documentation](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-F6E1BBB7-AEC2-45B1-9E36-1D86B84D2B85.html).
+ An understanding of the overall concept of mainframe application DevOps with continuous integration (CI). For an AWS Prescriptive Guidance pattern that was developed by AWSand Micro Focus, see [Mainframe modernization: DevOps on AWS with Micro Focus](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/mainframe-modernization-devops-on-aws-with-micro-focus.html).
**Note**
This pattern was tested with Micro Focus Enterprise Server version 6. For version 8, see [Set up Micro Focus Runtime (on Amazon EC2)](https://docs.aws.amazon.com/m2/latest/userguide/mf-runtime-setup.html).
**Limitations **
+ For a list of platforms that are supported by Micro Focus Enterprise Server, see the [Micro Focus Enterprise Server Data Sheet](https://www.microfocus.com/media/data-sheet/enterprise_server_ds.pdf).
+ The scripts and tests used in this pattern are based on Amazon EC2 Windows Server 2019; other Windows Server versions and operating systems were not tested for this pattern.
+ The pattern is based on Micro Focus Enterprise Server 6.0 for Windows; earlier or later releases were not tested in the development of this pattern.
**Product versions**
+ Micro Focus Enterprise Server 6.0
+ Windows Server 2019
## Architecture
In the conventional mainframe environment, you must provision hardware to host your applications and corporate data. To cater for and meet the spikes in seasonal, monthly, quarterly, or even unprecedented or unexpected demands, mainframe users must *scale out* by purchasing additional storage and compute capacity. Increasing the number of storage and compute capacity resources improves overall performance, but the scaling is not linear.
This is not the case when you start adopting an on-demand consumption model on AWS by using Amazon EC2 Auto Scaling and Micro Focus Enterprise Servers. The following sections explain detail how to build a fully automated, scalable mainframe application architecture using Micro Focus Enterprise Server Scale-Out Performance and Availability Cluster (PAC) with an Amazon EC2 Auto Scaling group.
**Micro Focus Enterprise Server automatic scaling architecture**
First, it is important to understand the basic concepts of Micro Focus Enterprise Server. This environment provides a mainframe-compatible, x86 deployment environment for applications that have traditionally run on the IBM mainframe. It delivers both online and batch runs and a transaction environment that supports the following:
+ IBM COBOL
+ IBM PL/I
+ IBM JCL batch jobs
+ IBM CICS and IMS TM transactions
+ Web services
+ Common batch utilities, including SORT
Micro Focus Enterprise Server enables mainframe applications to run with minimal changes. Existing mainframe workloads can be moved to x86 platforms and modernized to take advantage of AWS Cloud native extensions for rapid expansion to new markets or geographies.
The AWS Prescriptive Guidance pattern [Mainframe modernization: DevOps on AWS with Micro Focus](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/mainframe-modernization-devops-on-aws-with-micro-focus.html) introduced the architecture to accelerate the development and testing of mainframe applications on AWS using Micro Focus Enterprise Developer and Enterprise Test Server with AWS CodePipeline and AWS CodeBuild. This pattern focuses on the deployment of mainframe applications to the AWS production environment to achieve high availability and resiliency.
In a mainframe production environment, you might have set up IBM Parallel Sysplex in the mainframe to achieve high performance and high availability. To create a scale-out architecture similar to Sysplex, Micro Focus introduced the Performance and Availability Cluster (PAC) to Enterprise Server. PACs support mainframe application deployment onto multiple Enterprise Server regions managed as a single image and scaled out in Amazon EC2 instances. PACs also support predictable application performance and system throughput on demand.
In a PAC, multiple Enterprise Server instances work together as a single logical entity. Failure of one Enterprise Server instance, therefore, will not interrupt business continuity because capacity is shared with other regions while new instances are automatically started using industry standard functionality such as an Amazon EC2 Auto Scaling group. This removes single points of failure, improving resilience to hardware, network, and application issues. Scaled-out Enterprise Server instances can be operated and managed by using the Enterprise Server Common Web Administration (ESCWA) APIs, simplifying the operational maintenance and serviceability of Enterprise Servers.
**Note**
Micro Focus recommends that the [Performance and Availability Cluster (PAC)](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-C06DC883-8A67-44DB-8553-8F0DD2062DAB.html) should consist of at least three Enterprise Server regions so that availability is not compromised in the event an Enterprise Server region fails or requires maintenance.
PAC configuration requires a supported relational database management service (RDBMS) to manage the region database, a cross-region database, and optional data store databases. A data store database should be used to managed Virtual Storage Access Method (VSAM) files using the Micro Focus Database File Handler support to improve availability and scalability. Supported RDBMSs include the following:
+ Microsoft SQL Server 2009 R2 and later
+ PostgreSQL 10.x, including Amazon Aurora PostgreSQL-Compatible Edition
+ DB2 10.4 and later
For details of supported RDBMS and PAC requirements, see [Micro Focus Enterprise Server - Prerequisites](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-486C5A4B-E3CD-4B17-81F3-32F9DE970EA5.html) and [Micro Focus Enterprise Server - Recommended PAC Configuration](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-7038DB6F-E89F-4B5F-BCAA-BD1456F6CCA3.html).
The following diagram shows a typical AWS architecture setup for a Micro Focus PAC.
![\[A three-Availability Zone architecture with five steps described in a table after the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/64e3c22b-1058-4ab8-855f-18bbbed5dc13/images/df291568-a442-454f-80bf-49e4ffff4f6d.png)
|
|
| | ** Component** | **Description** |
| --- |--- |--- |
| 1 | Enterprise Server instances automatic scaling group | Set up an automatic scaling group deployed with Enterprise Server instances in a PAC. The number of instances can be scaled out or in initiated by Amazon CloudWatch alarms using CloudWatch metrics. |
| 2 | Enterprise Server ESCWA instances automatic scaling group | Set up an automatic scaling group deployed with Enterprise Server Common Web Administration (ESCWA). ESCWA provides cluster management APIs. The ESCWA servers act as a control plane to add or remove Enterprise Servers and start or stop Enterprise Server regions in the PAC during the Enterprise Server instance automatic scaling events. Because the ESCWA instance is used only for the PAC management, its traffic pattern is predictable, and its automatic scaling desired capacity requirement can be set to 1. |
| 3 | Amazon Aurora instance in a Multi-AZ setup | Set up a relational database management system (RDBMS) to host both user and system data files to be shared across the Enterprise Server instances. |
| 4 | Amazon ElastiCache (Redis OSS) instance and replica | Set up an ElastiCache (Redis OSS) primary instance and at least one replica to host user data and act as a scale-out repository (SOR) for the Enterprise Server instances. You can configure one or more [scale-out repository](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-3840E10F-80AA-4109-AF2C-894237D3AD00.html) to store specific types of user data. Enterprise Server uses a Redis NoSQL database as an SOR, [a requirement to maintain PAC integrity](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-176B97CA-4F9F-4CE1-952F-C3F4FB0ADD25.html). |
| 5 | Network Load Balancer | Set up a load balancer, providing a hostname for applications to connect to the services provided by Enterprise Server instances (for example, accessing the application through a 3270 emulator). |
These components form the minimum requirement for a Micro Focus Enterprise Server PAC cluster. The next section covers cluster management automation.
**Using AWS Systems Manager Automation for scaling**
After the PAC cluster is deployed on AWS, the PAC is managed through the Enterprise Server Common Web Administration (ESCWA) APIs.
To automate the cluster management tasks during automatic scaling events, you can use Systems Manager Automation runbooks and Amazon EC2 Auto Scaling with Amazon EventBridge. The architecture of these automations is shown in the following diagram.
![\[AWS architecture diagram showing EventBridge, Systems Manager, and EC2 Auto Scaling integration.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/64e3c22b-1058-4ab8-855f-18bbbed5dc13/images/6f9e4035-fafd-4aee-a6cc-d5e95d6514c2.png)
|
|
| | **Component** | **Description** |
| --- |--- |--- |
| 1 | Automatic scaling lifecycle hook | Set up automatic scaling lifecycle hooks and send notifications to Amazon EventBridge when new instances are launched and existing instances are terminated in the automatic scaling group. |
| 2 | Amazon EventBridge | Set up an Amazon EventBridge rule to route automatic scaling events to Systems Manager Automation runbook targets. |
| 3 | Automation runbooks | Set up Systems Manager Automation runbooks to run Windows PowerShell scripts and invoke ESCWA APIs to manage the PAC. For examples, see the *Additional information* section. |
| 4 | Enterprise Server ESCWA instance in an automatic scaling group | Set up an Enterprise Server ESCWA instance in an automatic scaling group. The ESCWA instance provides APIs to manage the PAC. |
## Tools
+ [Micro Focus Enterprise Server](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-A2F23243-962B-440A-A071-480082DF47E7.html) – Micro Focus Enterprise Server provides the run environment for applications created with any integrated development environment (IDE) variant of Enterprise Developer.
+ [Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html) – Amazon EC2 Auto Scaling helps you ensure that you have the correct number of Amazon EC2 instances available to handle the load for your application. You create collections of EC2 instances, called Auto Scaling groups, and specify minimum and maximum numbers of instances.
+ [Amazon ElastiCache (Redis OSS)](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) – Amazon ElastiCache is a web service for setting up, managing, and scaling a distributed in-memory data store or cache environment in the cloud. It provides a high-performance, scalable, and cost-effective caching solution.
+ [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) – Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for a relational database and manages common database administration tasks.
+ [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) – AWS Systems Manager is an AWS service that you can use to view and control your infrastructure on AWS. Using the Systems Manager console, you can view operational data from multiple AWS services and automate operational tasks across your AWS resources. Systems Manager helps you maintain security and compliance by scanning your managed instances and reporting on (or taking corrective action on) any policy violations it detects.
## Epics
### Create an Amazon Aurora instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS CloudFormation template for an Amazon Aurora instance. | Use the [AWS example code snippet](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_RDS.html) to make a CloudFormation template that will create an Amazon Aurora PostgreSQL-Compatible Edition instance. | Cloud architect |
| Deploy a CloudFormation stack to create the Amazon Aurora instance. | Use the CloudFormation template to create an Aurora PostgreSQL-Compatible instance that has Multi-AZ replication enabled for production workloads. | Cloud architect |
| Configure database connection settings for Enterprise Server. | Follow the instructions in the [Micro Focus documentation](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-40748F62-84B3-4B7B-8E96-5484ADEDFB5F.html) to prepare the connection strings and database configuration for Micro Focus Enterprise Server. | Data engineer, DevOps engineer |
### Create an Amazon ElastiCache cluster for the Redis instance
| Task | Description | Skills required |
| --- | --- | --- |
| Create a CloudFormation template for the Amazon ElastiCache cluster for the Redis instance. | Use the [AWS example code snippet](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ElastiCache.html) to make a CloudFormation template that will create an Amazon ElastiCache cluster for the Redis instance. | Cloud architect |
| Deploy the CloudFormation stack to create an Amazon ElastiCache cluster for the Redis instance. | Create the Amazon ElastiCache cluster for the Redis instance that has Multi-AZ replication enabled for production workloads. | Cloud architect |
| Configure Enterprise Server PSOR connection settings. | Follow the instructions in the [Micro Focus documentation](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-2A420ADD-4CA6-472D-819F-371C037C0653.html) to prepare the PAC Scale-Out Repository (PSOR) connection configuration for Micro Focus Enterprise Server PAC. | DevOps engineer |
### Create a Micro Focus Enterprise Server ESCWA automatic scaling group
| Task | Description | Skills required |
| --- | --- | --- |
| Create a Micro Focus Enterprise Server AMI. | Create an Amazon EC2 Windows Server instance and install the Micro Focus Enterprise Server binary in the EC2 instance. Create an Amazon Machine Image (AMI) of the EC2 instance. For more information, see the [Enterprise Server installation documentation](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-FACEF60F-BAE3-446C-B2B4-4379A5DF6D9F.html). | Cloud architect |
| Create a CloudFormation template for Enterprise Server ESCWA. | Use the [AWS example code snippet](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_AutoScaling.html) to make a template for creating a custom stack of Enterprise Server ESCWA in an automatic scaling group. | Cloud architect |
| Deploy the CloudFormation stack to create an Amazon EC2 scaling group for Enterprise Server ESCWA. | Use the CloudFormation template to deploy the automatic scaling group with the Micro Focus Enterprise Server ESCWA AMI created in the previous story. | Cloud architect |
### Create an AWS Systems Manager Automation runbook
| Task | Description | Skills required |
| --- | --- | --- |
| Create a CloudFormation template for a Systems Manager Automation runbook. | Use the example code snippets in the *Additional information* section to make a CloudFormation template that will create a Systems Manager Automation runbook for automating PAC creation, Enterprise Server scale in, and Enterprise Server scale out. | Cloud architect |
| Deploy the CloudFormation stack that contains the Systems Manager Automation runbook. | Use the CloudFormation template to deploy a stack that contains the Automation runbook for PAC creation, Enterprise Server scale in, and Enterprise Server scale out. | Cloud architect |
### Create an automatic scaling group for Micro Focus Enterprise Server
| Task | Description | Skills required |
| --- | --- | --- |
| Create a CloudFormation template for setting up an automatic scaling group for Micro Focus Enterprise Server. | Use the [AWS example code snippet](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_AutoScaling.html) to make a CloudFormation template that will create an automatic scaling group. This template will reuse the same AMI that was created for the Micro Focus Enterprise Server ESCWA instance. Then use an [AWS example code snippet](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html) to create the automatic scaling lifecycle event and set up Amazon EventBridge to filter for scale-out and scale-in events in the same CloudFormation template. | Cloud architect |
| Deploy the CloudFormation stack for the automatic scaling group for Micro Focus Enterprise Servers. | Deploy the CloudFormation stack that contains the automatic scaling group for Micro Focus Enterprise Servers. | Cloud architect |
## Related resources
+ [Micro Focus Enterprise Server Performance and Availability Cluster (PAC)](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-613F5E2D-2FBC-47AE-9327-48CA4FF84C5B.html)
+ [Amazon EC2 Auto Scaling lifecycle hooks](https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html)
+ [Running automations with triggers using EventBridge](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-cwe-target.html)
## Additional information
The following scenarios must be automated for scaling in or scaling out the PAC clusters.
**Automation for starting or recreating a PAC**
At the start of a PAC cluster, Enterprise Server requires ESCWA to invoke APIs to create a PAC configuration. This starts and adds Enterprise Server regions into the PAC. To create or recreate a PAC, use the following steps:
1. Configure a [PAC Scale-Out Repository (PSOR)](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-2A420ADD-4CA6-472D-819F-371C037C0653.html) in ESCWA with a given name.
```
POST /server/v1/config/groups/sors
```
1. Create a PAC with a given name and attach the PSOR to it.
```
POST /server/v1/config/groups/pacs
```
1. Configure the region database and cross-region database if this is the first time you are setting up a PAC.
**Note**
This step uses SQL queries and the Micro Focus Enterprise Suite command line **dbhfhadmin** tool to create the database and import initial data.
1. Install the PAC definition into the Enterprise Server regions.
```
POST /server/v1/config/mfds
POST /native/v1/config/groups/pacs/${pac_uid}/install
```
1. Start Enterprise Server regions in the PAC.
```
POST /native/v1/regions/${host_ip}/${port}/${region_name}/start
```
The previous steps can be implemented by using a Windows PowerShell script.
The following steps explain how to build an automation for creating a PAC by reusing the Windows PowerShell script.
1. Create an Amazon EC2 launch template that downloads or creates the Windows PowerShell script as part of the bootstrap process. For example, you can use EC2 user data to download the script from an Amazon Simple Storage Service (Amazon S3) bucket.
1. Create an AWS Systems Manager Automation runbook to invoke the Windows PowerShell script.
1. Associate the runbook to the ESCWA instance by using the instance tag.
1. Create an ESCWA automatic scaling group by using the launch template.
You can use the following example AWS CloudFormation snippet to create the Automation runbook.
*Example CloudFormation snippet for a Systems Manager Automation runbook used for PAC creation*
```
PACInitDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: Command
Content:
schemaVersion: '2.2'
description: Operation Runbook to create Enterprise Server PAC
mainSteps:
- action: aws:runPowerShellScript
name: CreatePAC
inputs:
onFailure: Abort
timeoutSeconds: "1200"
runCommand:
- |
C:\Scripts\PAC-Init.ps1
PacInitAutomation:
Type: AWS::SSM::Document
Properties:
DocumentType: Automation
Content:
description: Prepare Micro Focus PAC Cluster via ESCWA Server
schemaVersion: '0.3'
assumeRole: !GetAtt SsmAssumeRole.Arn
mainSteps:
- name: RunPACInitDocument
action: aws:runCommand
timeoutSeconds: 300
onFailure: Abort
inputs:
DocumentName: !Ref PACInitDocument
Targets:
- Key: tag:Enterprise Server - ESCWA
Values:
- "true"
PacInitDocumentAssociation:
Type: AWS::SSM::Association
Properties:
DocumentVersion: "$LATEST"
Name: !Ref PACInitDocument
Targets:
- Key: tag:Enterprise Server - ESCWA
Values:
- "true"
```
For more information, see [Micro Focus Enterprise Server - Configuring a PAC](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-2B15EBA5-84AF-47C3-9F8E-EE57EB17245F.html).
**Automation for scaling out with a new Enterprise Server instance**
When an Enterprise Server instance is scaled out, its Enterprise Server region must be added to the PAC. The following steps explain how to invoke ESCWA APIs and add the Enterprise Server region into the PAC.
1. Install the PAC definition into the Enterprise Server regions.
```
POST '/server/v1/config/mfds'
POST /native/v1/config/groups/pacs/${pac_uid}/install
```
1. Warm Start the region in the PAC.
```
POST /native/v1/regions/${host_ip}/${port}/${region_name}/start
```
1. Add the Enterprise Server instance to the load balancer by associating the automatic scaling group to the load balancer.
The previous steps can be implemented by using a Windows PowerShell script. For more information, see [Micro Focus Enterprise Server - Configuring a PAC](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-2B15EBA5-84AF-47C3-9F8E-EE57EB17245F.html).
The following steps can be used to build an event driven automation to add a newly launched Enterprise Server instance into a PAC by reusing the Windows PowerShell script.
1. Create an Amazon EC2 launch template for Enterprise Server instance that provisions an Enterprise Server Region during its bootstrap. For example, you can use the Micro Focus Enterprise Server command mfds to import a region configuration. For further details and options available for this command, see the [Enterprise Server Reference](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/HRADRHCOMM06.html).
1. Create an Enterprise Server automatic scaling group that uses the launch template created in the previous step.
1. Create a Systems Manager Automation runbook to invoke the Windows PowerShell script.
1. Associate the runbook to the ESCWA instance by using the instance tag.
1. Create an Amazon EventBridge rule to filter for the EC2 Instance Launch Successful event for the Enterprise Server automatic scaling group, and create the target to use the Automation runbook.
You can use the following example CloudFormation snippet to create the Automation runbook and the EventBridge rule.
*Example CloudFormation snippet for Systems Manager used for scaling out Enterprise Server instances*
```
ScaleOutDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: Command
Content:
schemaVersion: '2.2'
description: Operation Runbook to Adding MFDS Server into an existing PAC
parameters:
MfdsPort:
type: String
InstanceIpAddress:
type: String
default: "Not-Available"
InstanceId:
type: String
default: "Not-Available"
mainSteps:
- action: aws:runPowerShellScript
name: Add_MFDS
inputs:
onFailure: Abort
timeoutSeconds: "300"
runCommand:
- |
$ip = "{{InstanceIpAddress}}"
if ( ${ip} -eq "Not-Available" ) {
$ip = aws ec2 describe-instances --instance-id {{InstanceId}} --output text --query "Reservations[0].Instances[0].PrivateIpAddress"
}
C:\Scripts\Scale-Out.ps1 -host_ip ${ip} -port {{MfdsPort}}
PacScaleOutAutomation:
Type: AWS::SSM::Document
Properties:
DocumentType: Automation
Content:
parameters:
MfdsPort:
type: String
InstanceIpAddress:
type: String
default: "Not-Available"
InstanceId:
type: String
default: "Not-Available"
description: Scale Out 1 New Server in Micro Focus PAC Cluster via ESCWA Server
schemaVersion: '0.3'
assumeRole: !GetAtt SsmAssumeRole.Arn
mainSteps:
- name: RunScaleOutCommand
action: aws:runCommand
timeoutSeconds: 300
onFailure: Abort
inputs:
DocumentName: !Ref ScaleOutDocument
Parameters:
InstanceIpAddress: "{{InstanceIpAddress}}"
InstanceId: "{{InstanceId}}"
MfdsPort: "{{MfdsPort}}"
Targets:
- Key: tag:Enterprise Server - ESCWA
Values:
- "true"
```
**Automation for scaling in an Enterprise Server instance**
Similar to scaling out, when an Enterprise Server instance is *scaled in*, the event EC2 Instance-terminate Lifecycle Action is initiated, and the following process and API calls are needed to remove a Micro Focus Enterprise Server instance from the PAC.
1. Stop the region in the terminating Enterprise Server instance.
```
POST "/native/v1/regions/${host_ip}/${port}/${region_name}/stop"
```
1. Remove the Enterprise Server Instance from the PAC.
```
DELETE "/server/v1/config/mfds/${uid}"
```
1. Send signal to continue terminating the Enterprise Server instance.
The previous steps can be implemented in a Windows PowerShell script. For additional details of this process, see [Micro Focus Enterprise Server document - Administering a PAC](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-E864E2E9-EB49-43BF-9AAD-7FE334749441.html).
The following steps explain how to build an event-driven automation to terminate an Enterprise Server instance from a PAC by reusing the Windows PowerShell script.
1. Create a Systems Manager Automation runbook to invoke the Windows PowerShell script.
1. Associate the runbook to the ESCWA instance by using the instance tag.
1. Create an automatic scaling group lifecycle hook for EC2 instance termination.
1. Create an Amazon EventBridge rule to filter EC2 Instance-terminate Lifecycle Action event for the Enterprise Server automatic scaling group, and create the target to use the Automation runbook.
You can use the following example CloudFormation template for creating a Systems Manager Automation runbook, lifecycle hook, and EventBridge rule.
*Example CloudFormation snippet for a Systems Manager Automation runbook used for scaling in an Enterprise Server instance*
```
ScaleInDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: Command
Content:
schemaVersion: '2.2'
description: Operation Runbook to Remove MFDS Server from PAC
parameters:
MfdsPort:
type: String
InstanceIpAddress:
type: String
default: "Not-Available"
InstanceId:
type: String
default: "Not-Available"
mainSteps:
- action: aws:runPowerShellScript
name: Remove_MFDS
inputs:
onFailure: Abort
runCommand:
- |
$ip = "{{InstanceIpAddress}}"
if ( ${ip} -eq "Not-Available" ) {
$ip = aws ec2 describe-instances --instance-id {{InstanceId}} --output text --query "Reservations[0].Instances[0].PrivateIpAddress"
}
C:\Scripts\Scale-In.ps1 -host_ip ${ip} -port {{MfdsPort}}
PacScaleInAutomation:
Type: AWS::SSM::Document
Properties:
DocumentType: Automation
Content:
parameters:
MfdsPort:
type: String
InstanceIpAddress:
type: String
default: "Not-Available"
InstanceId:
type: String
default: "Not-Available"
description: Scale In 1 New Server in Micro Focus PAC Cluster via ESCWA Server
schemaVersion: '0.3'
assumeRole: !GetAtt SsmAssumeRole.Arn
mainSteps:
- name: RunScaleInCommand
action: aws:runCommand
timeoutSeconds: "600"
onFailure: Abort
inputs:
DocumentName: !Ref ScaleInDocument
Parameters:
InstanceIpAddress: "{{InstanceIpAddress}}"
MfdsPort: "{{MfdsPort}}"
InstanceId: "{{InstanceId}}"
Targets:
- Key: tag:Enterprise Server - ESCWA
Values:
- "true"
- name: TerminateTheInstance
action: aws:executeAwsApi
inputs:
Service: autoscaling
Api: CompleteLifecycleAction
AutoScalingGroupName: !Ref AutoScalingGroup
InstanceId: "{{ InstanceId }}"
LifecycleActionResult: CONTINUE
LifecycleHookName: !Ref ScaleInLifeCycleHook
```
**Automation for an Amazon EC2 automatic scaling trigger**
The process of setting up a scaling policy for Enterprise Server instances requires an understanding of the application behavior. In most cases, you can set up target tracking scaling policies. For example, you can use the average CPU utilization as the Amazon CloudWatch metric to set for the automatic scaling policy. For more information, see [Target tracking scaling policies for Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-target-tracking.html). For applications that have regular traffic patterns, consider using a predictive scaling policy. For more information, see [Predictive scaling for Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-predictive-scaling.html).
# Build an advanced mainframe file viewer in the AWS Cloud
*Boopathy GOPALSAMY and Jeremiah O'Connor, Amazon Web Services*
## Summary
This pattern provides code samples and steps to help you build an advanced tool for browsing and reviewing your mainframe fixed-format files by using AWS serverless services. The pattern provides an example of how to convert a mainframe input file to an Amazon OpenSearch Service document for browsing and searching. The file viewer tool can help you achieve the following:
+ Retain the same mainframe file structure and layout for consistency in your AWS target migration environment (for example, you can maintain the same layout for files in a batch application that transmits files to external parties)
+ Speed up development and testing during your mainframe migration
+ Support maintenance activities after the migration
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A virtual private cloud (VPC) with a subnet that’s reachable by your legacy platform
+
**Note**
An input file and its corresponding common business-oriented language (COBOL) copybook (: For input file and COBOL copybook examples, see [gfs-mainframe-solutions](https://github.com/aws-samples/gfs-mainframe-patterns.git) on the GitHub repository. For more information about COBOL copybooks, see the [Enterprise COBOL for z/OS 6.3](https://publibfp.boulder.ibm.com/epubs/pdf/igy6pg30.pdf) Programming Guide on the IBM website.)
**Limitations**
+ Copybook parsing is limited to no more than two nested levels (OCCURS)
## Architecture
**Source technology stack **
+ Input files in [FB (Fixed Blocked)](https://www.ibm.com/docs/en/zos-basic-skills?topic=set-data-record-formats) format
+ COBOL copybook layout
**Target technology stack **
+ Amazon Athena
+ Amazon OpenSearch Service
+ Amazon Simple Storage Service (Amazon S3)
+ AWS Lambda
+ AWS Step Functions
**Target architecture**
The following diagram shows the process of parsing and converting a mainframe input file to an OpenSearch Service document for browsing and searching.
![\[Process to parse and convert mainframe input file to OpenSearch Service.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36d72b00-d163-455f-9e59-e2c872e7c28a/images/cce68438-bcf2-48c1-b86b-01242235ec76.png)
The diagram shows the following workflow:
1. An admin user or application pushes input files to one S3 bucket and COBOL copybooks to another S3 bucket.
1.
**Note**
The S3 bucket with the input files invokes a Lambda function that kicks off a serverless Step Functions workflow. : The use of an S3 event trigger and Lambda function to drive the Step Functions workflow in this pattern is optional. The GitHub code samples in this pattern don’t include the use of these services, but you can use these services based on your requirements.
1. The Step Functions workflow coordinates all the batch processes from the following Lambda functions:
+ The `s3copybookparser.py` function parses the copybook layout and extracts field attributes, data types, and offsets (required for input data processing).
+ The `s3toathena.py` function creates an Athena table layout. Athena parses the input data that’s processed by the `s3toathena.py` function and converts the data to a CSV file.
+ The `s3toelasticsearch.py` function ingests the results file from the S3 bucket and pushes the file to OpenSearch Service.
1. Users access OpenSearch Dashboards with OpenSearch Service to retrieve the data in various table and column formats and then run queries against the indexed data.
## Tools
**AWS services**
+ [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is an interactive query service that helps you analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. In this pattern, you use Lambda to implement core logic, such as parsing files, converting data, and loading data into OpenSearch Service for interactive file access.
+ [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html) is a managed service that helps you deploy, operate, and scale OpenSearch Service clusters in the AWS Cloud. In this pattern, you use OpenSearch Service to index the converted files and provide interactive search capabilities for users.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine Lambda functions and other AWS services to build business-critical applications. In this pattern, you use Step Functions to orchestrate Lambda functions.
**Other tools**
+ [GitHub](https://github.com/) is a code-hosting service that provides collaboration tools and version control.
+ [Python](https://www.python.org/) is a high-level programming language.
**Code**
The code for this pattern is available in the GitHub [gfs-mainframe-patterns](https://github.com/aws-samples/gfs-mainframe-patterns.git) repository.
## Epics
### Prepare the target environment
| Task | Description | Skills required |
| --- | --- | --- |
| Create the S3 bucket. | [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) for storing the copybooks, input files, and output files. We recommend the following folder structure for your S3 bucket:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
| Create the s3copybookparser function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
| Create the s3toathena function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
| Create the s3toelasticsearch function. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
| Create the OpenSearch Service cluster. | **Create the cluster**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html)**Grant access to the IAM role**To provide fine-grained access to the Lambda function’s IAM role (`arn:aws:iam::**:role/service-role/s3toelasticsearch-role-**`), do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
| Create Step Functions for orchestration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
### Deploy and run
| Task | Description | Skills required |
| --- | --- | --- |
| Upload the input files and copybooks to the S3 bucket. | Download sample files from the [GitHub ](https://github.com/aws-samples/gfs-mainframe-patterns.git)repository sample folder and upload the files to the S3 bucket that you created earlier.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
| Invoke the Step Functions. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html)
| General AWS |
| Validate the workflow execution in Step Functions. | In the [Step Functions console](https://console.aws.amazon.com/states/home), review the workflow execution in the **Graph inspector**. The execution run states are color coded to represent execution status. For example, blue indicates **In Progress**, green indicates **Succeeded**, and red indicates **Failed**. You can also review the table in the **Execution event history** section for more detailed information about the execution events.For an example of a graphical workflow execution, see *Step Functions graph* in the *Additional information* section of this pattern. | General AWS |
| Validate the delivery logs in Amazon CloudWatch. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html)For an example of successful delivery logs, see *CloudWatch delivery logs* in the *Additional information* section of this pattern. | General AWS |
| Validate the formatted file in OpenSearch Dashboards and perform file operations. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-an-advanced-mainframe-file-viewer-in-the-aws-cloud.html) | General AWS |
## Related resources
**References **
+ [Example COBOL copybook](https://www.ibm.com/docs/en/record-generator/3.0?topic=SSMQ4D_3.0.0/documentation/cobol_rcg_examplecopybook.html) (IBM documentation)
+ [BMC Compuware File-AID](https://www.bmc.com/it-solutions/bmc-compuware-file-aid.html) (BMC documentation)
**Tutorials**
+ [Tutorial: Using an Amazon S3 trigger to invoke a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html) (AWS Lambda documentation)
+ [How do I create a serverless workflow with AWS Step Functions and AWS Lambda](https://aws.amazon.com/getting-started/hands-on/create-a-serverless-workflow-step-functions-lambda/) (AWS documentation)
+ [Using OpenSearch Dashboards with Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/dashboards.html) (AWS documentation)
## Additional information
**Step Functions graph**
The following example shows a Step Functions graph. The graph shows the execution run status for the Lambda functions used in this pattern.
![\[Step Functions graph shows execution run status for the Lambda functions used in this pattern.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36d72b00-d163-455f-9e59-e2c872e7c28a/images/11093e5d-2f9e-4bbf-8abc-f3b2980dd550.png)
**CloudWatch delivery logs**
The following example shows successful delivery logs for the execution of the `s3toelasticsearch` execution.
|
|
| 2022-08-10T15:53:33.033-05:00 | Number of processing documents: 100 | |
| --- |--- |--- |
| | 2022-08-10T15:53:33.171-05:00 | [INFO] 2022-08-10T20:53:33.171Z a1b2c3d4-5678-90ab-cdef-EXAMPLE11111POST https://search-essearch-3h4uqclifeqaj2vg4mphe7ffle.us-east-2.es.amazonaws.com:443/\$1bulk [status:200 request:0.100s] |
| | 2022-08-10T15:53:33.172-05:00 | Bulk write succeed: 100 documents |
# Containerize mainframe workloads that have been modernized by Blu Age
*Richard Milner-Watts, Amazon Web Services*
## Summary
This pattern provides a sample container environment for running mainframe workloads that have been modernized by using the [Blu Age](https://www.bluage.com/) tool. Blu Age converts legacy mainframe workloads into modern Java code. This pattern provides a wrapper around the Java application so you can run it by using container orchestration services such as [Amazon Elastic Container Service (Amazon ECS)](https://aws.amazon.com/ecs/) or [Amazon Elastic Kubernetes Service (Amazon EKS)](https://aws.amazon.com/eks/).
For more information about modernizing your workloads by using Blu Age and AWS services, see these AWS Prescriptive Guidance publications:
+ [Running modernized Blu Age mainframe workloads on serverless AWS infrastructure](https://docs.aws.amazon.com/prescriptive-guidance/latest/run-bluage-modernized-mainframes/)
+ [Deploy an environment for containerized Blu Age applications by using Terraform](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.html)
For assistance with using Blu Age to modernize your mainframe workloads, contact the Blu Age team by choosing **Contact our experts** on the [Blu Age website](https://www.bluage.com/). For assistance with migrating your modernized workloads to AWS, integrating them with AWS services, and moving them into production, contact your AWS account manager or fill out the [AWS Professional Services form](https://pages.awscloud.com/AWS-Professional-Services.html).
## Prerequisites and limitations
**Prerequisites**
+ A modernized Java application that was created by Blu Age. For testing purposes, this pattern provides a sample Java application that you can use as a proof of concept.
+ A [Docker](https://aws.amazon.com/docker/) environment that you can use to build the container.
**Limitations**
Depending on the container orchestration platform that you use, the resources that can be made available to the container (such as CPU, RAM, and storage) might be limited. For example, if you’re using Amazon ECS with AWS Fargate, see the [Amazon ECS documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html) for limits and considerations.
## Architecture
**Source technology stack**
+ Blu Age
+ Java
**Target technology stack**
+ Docker
**Target architecture**
The following diagram shows the architecture of the Blu Age application within a Docker container.
![\[Blu Age application in Docker container\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/c1747094-357b-4222-b4eb-b1336d810f83/images/0554332d-eff5-49ca-9789-da39b5a10045.png)
1. The entry point for the container is the wrapper script. This bash script is responsible for preparing the runtime environment for the Blu Age application and processing outputs.
1. Environment variables within the container are used to configure variables in the wrapper script, such as the Amazon Simple Storage Service (Amazon S3) bucket names and database credentials. Environment variables are supplied by either AWS Secrets Manager or Parameter Store, a capability of AWS Systems Manager. If you’re using Amazon ECS as your container orchestration service, you can also hardcode the environment variables in the Amazon ECS task definition.
1. The wrapper script is responsible for pulling any input files from the S3 bucket into the container before you run the Blu Age application. The AWS Command Line Interface (AWS CLI) is installed within the container. This provides a mechanism for accessing objects that are stored in Amazon S3 through the gateway virtual private cloud (VPC) endpoint.
1. The Java Archive (JAR) file for the Blu Age application might need to communicate with other data sources, such as Amazon Aurora.
1. After completion, the wrapper script delivers the resulting output files into an S3 bucket for further processing (for example, by Amazon CloudWatch logging services). The pattern also supports delivering zipped log files to Amazon S3, if you’re using an alternative to standard CloudWatch logging.
## Tools
**AWS services**
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable.
+ [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a fast and scalable container management service that helps you run, stop, and manage containers on a cluster.
**Tools**
+ [Docker](https://aws.amazon.com/docker/) is a software platform for building, testing, and deploying applications. Docker packages software into standardized units called [containers](https://aws.amazon.com/containers/), which have everything the software needs to run, including libraries, system tools, code, and runtime. You can use Docker to deploy and scale applications into any environment.
+ [Bash](https://www.gnu.org/software/bash/manual/) is a command language interface (shell) for the GNU operating system.
+ [Java](https://www.java.com/) is the programming language and development environment used in this pattern.
+ [Blu Age](https://www.bluage.com/) is an AWS mainframe modernization tool that converts legacy mainframe workloads, including application code, dependencies, and infrastructure, into modern workloads for the cloud.
**Code repository**
The code for this pattern is available in the GitHub [Blu Age sample container repository](https://github.com/aws-samples/aws-blu-age-sample-container).
## Best practices
+ Externalize the variables for altering your application’s behavior by using environment variables. These variables enable the container orchestration solution to alter the runtime environment without rebuilding the container. This pattern includes examples of environment variables that can be useful for Blu Age applications.
+ Validate any application dependencies before you run your Blu Age application. For example, verify that the database is available and credentials are valid. Write tests in the wrapper script to verify dependencies, and fail early if they are not met.
+ Use verbose logging within the wrapper script. Interacting directly with a running container can be challenging, depending on the orchestration platform and how long the job takes. Make sure that useful output is written to `STDOUT` to help diagnose any issues. For example, output might include the contents of the application’s working directory both before and after you run the application.
## Epics
### Obtain a Blu Age application JAR file
| Task | Description | Skills required |
| --- | --- | --- |
| Option 1 - Work with Blu Age to obtain your application's JAR file. | The container in this pattern requires a Blu Age application. Alternatively, you can use the sample Java application that’s provided with this pattern for a prototype.Work with the Blu Age team to obtain a JAR file for your application that can be baked into the container. If the JAR file isn’t available, see the next task to use the sample application instead. | Cloud architect |
| Option 2 - Build or use the supplied sample application JAR file. | This pattern provides a prebuilt sample JAR file. This file outputs the application’s environment variables to `STDOUT` before sleeping for 30 seconds and exiting.This file is named `bluAgeSample.jar` and is located in the [docker folder](https://github.com/aws-samples/aws-blu-age-sample-container/tree/main/docker) of the GitHub repository.If you want to alter the code and build your own version of the JAR file, use the source code located at [./java\$1sample/src/sample\$1java\$1app.java](https://github.com/aws-samples/aws-blu-age-sample-container/tree/main/java_sample/src) in the GitHub repository. You can use the build script at [./java\$1sample/build.sh](https://github.com/aws-samples/aws-blu-age-sample-container/tree/main/java_sample) to compile the Java source and build a new JAR fie. | App developer |
### Build the Blu Age container
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the GitHub repository. | Clone the sample code repository by using the command:
| AWS DevOps |
| Use Docker to build the container. | Use Docker to build the container before you push it to a Docker registry such as Amazon ECR:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.html) | AWS DevOps |
| Test the Blu Age container. | (Optional) If necessary, test the container locally by using the command:
docker run -it /bin/bash
| AWS DevOps |
| Authenticate to your Docker repository. | If you plan to use Amazon ECR, follow the instructions in the [Amazon ECR documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html) to install and configure the AWS CLI and authenticate the Docker CLI to your default registry.We recommend that you use the [get-login-password command](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecr/get-login-password.html) for authentication. The [Amazon ECR console](https://console.aws.amazon.com/ecr/) provides a pre-populated version of this command if you use the **View push commands** button. For more information, see the [Amazon ECR documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-console.html).
If you don’t plan to use Amazon ECR, follow the instructions provided for your container registry system. | AWS DevOps |
| Create a container repository. | Create a repository in Amazon ECR. For instructions, see the pattern [Deploy an environment for containerized Blu Age applications by using Terraform](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.html).If you’re using another container registry system, follow the instructions provided for that system. | AWS DevOps |
| Tag and push your container to the target repository. | If you're using Amazon ECR:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.html)For more information, see [Pushing a Docker image](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html) in the *Amazon ECR User Guide*. | AWS DevOps |
## Related resources
**AWS resources**
+ [AWS Blu Age sample container repository](https://github.com/aws-samples/aws-blu-age-sample-container)
+ [Running modernized Blu Age mainframe workloads on serverless AWS infrastructure](https://docs.aws.amazon.com/prescriptive-guidance/latest/run-bluage-modernized-mainframes/)
+ [Deploy an environment for containerized Blu Age applications by using Terraform](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.html)
+ [Using Amazon ECR with the AWS CLI](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html) (*Amazon ECR User Guide*)
+ [Private registry authentication](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html) (*Amazon ECR User Guide*)
+ [Amazon ECS documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html)
+ [Amazon EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html)
**Additional resources**
+ [Blu Age website](https://www.bluage.com/)
+ [Docker website](https://docker.com/)
# Convert and unpack EBCDIC data to ASCII on AWS by using Python
*Luis Gustavo Dantas, Amazon Web Services*
## Summary
Because mainframes typically host critical business data, modernizing data is one of the most important tasks when migrating data to the Amazon Web Services (AWS) Cloud or other American Standard Code for Information Interchange (ASCII) environment. On mainframes, data is typically encoded in extended binary-coded decimal interchange code (EBCDIC) format. Exporting database, Virtual Storage Access Method (VSAM), or flat files generally produces packed, binary EBCDIC files, which are more complex to migrate. The most commonly used database migration solution is change data capture (CDC), which, in most cases, automatically converts data encoding. However, CDC mechanisms might not be available for these database, VSAM, or flat files. For these files, an alternative approach is required to modernize the data.
This pattern describes how to modernize EBCDIC data by converting it to ASCII format. After conversion, you can load the data into distributed databases or have applications in the cloud process the data directly. The pattern uses the conversion script and sample files in the [mainframe-data-utilities](https://github.com/aws-samples/mainframe-data-utilities) GitHub repository.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ An EBCDIC input file and its corresponding common business-oriented language (COBOL) copybook. A sample EBCDIC file and COBOL copybook are included in the [mainframe-data-utilities](https://github.com/aws-samples/mainframe-data-utilities) GitHub repository. For more information about COBOL copybooks, see [Enterprise COBOL for z/OS 6.4 Programming Guide](https://publibfp.dhe.ibm.com/epubs/pdf/igy6pg40.pdf) on the IBM website.
**Limitations**
+ File layouts defined inside COBOL programs are not supported. They must be made available separately.
**Product versions**
+ Python version 3.8 or later
## Architecture
**Source technology stack**
+ EBCDIC data on a mainframe
+ COBOL copybook
**Target technology stack**
+ Amazon Elastic Compute Cloud (Amazon EC2) instance in a virtual private cloud (VPC)
+ Amazon Elastic Block Store (Amazon EBS)
+ Python and its required packages, JavaScript Object Notation (JSON), sys, and datetime
+ ASCII flat file ready to be read by a modern application or loaded in a relational database table
**Target architecture**
![\[EBCDIC data converted to ASCII on an EC2 instance by using Python scripts and a COBOL copybook\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f5907bfe-7dff-4cd0-8523-57015ad48c4b/images/4f97b1dd-3f20-4966-a291-22180680ea99.png)
The architecture diagram shows the process of converting an EBCDIC file to an ASCII file on an EC2 instance:
1. Using the **parse\$1copybook\$1to\$1json.py** script, you convert the COBOL copybook to a JSON file.
1. Using the JSON file and the **extract\$1ebcdic\$1to\$1ascii.py** script, you convert the EBCDIC data to an ASCII file.
**Automation and scale**
After the resources needed for the first manual file conversions are in place, you can automate file conversion. This pattern doesn’t include instructions for automation. There are multiple ways to automate the conversion. The following is an overview of one possible approach:
1. Encapsulate the AWS Command Line Interface (AWS CLI) and Python script commands into a shell script.
1. Create an AWS Lambda function that asynchronously submits the shell script job into an EC2 instance. For more information, see [Scheduling SSH jobs using AWS Lambda](https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/).
1. Create an Amazon Simple Storage Service (Amazon S3) trigger that invokes the Lambda function every time a legacy file is uploaded. For more information, see [Using an Amazon S3 trigger to invoke a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html).
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/?id=docs_gateway) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need, and quickly scale them up or down.
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) provides block-level storage volumes for use with Amazon Elastic Compute Cloud (Amazon EC2) instances.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
**Other tools**
+ [GitHub](https://github.com/) is a code-hosting service that provides collaboration tools and version control.
+ [Python](https://www.python.org/) is a high-level programming language.
**Code repository**
The code for this pattern is available in the [mainframe-data-utilities](https://github.com/aws-samples/mainframe-data-utilities) GitHub repository.
## Epics
### Prepare the EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Launch an EC2 instance. | The EC2 instance must have outbound internet access. This allows the instance to access the Python source code available on GitHub. To create the instance:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html) | General AWS |
| Install Git. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html) | General AWS, Linux |
| Install Python. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html) | General AWS, Linux |
| Clone the GitHub repository. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html) | General AWS, GitHub |
### Create the ASCII file from the EBCDIC data
| Task | Description | Skills required |
| --- | --- | --- |
| Parse the COBOL copybook into the JSON layout file. | Inside the `mainframe-data-utilities` folder, run the **parse\$1copybook\$1to\$1json.py** script. This automation module reads the file layout from a COBOL copybook and creates a JSON file. The JSON file contains the information required to interpret and extract the data from the source file. This creates the JSON metadata from the COBOL copybook. The following command converts the COBOL copybook to a JSON file.
For more information about the arguments, see the [README file](https://github.com/aws-samples/mainframe-data-utilities/blob/main/README.md) in the GitHub repository. | General AWS, Linux |
| Inspect the JSON layout file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html)
The most important attributes of the JSON layout file are:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html)For more information about the JSON layout file, see the [README file](https://github.com/aws-samples/mainframe-data-utilities/blob/main/README.md) in the GitHub repository. | General AWS, JSON |
| Create the ASCII file. | Run the **extract\$1ebcdic\$1to\$1ascii.py** script, which is included in the cloned GitHub repository. This script reads the EBCDIC file and writes a converted and readable ASCII file.
As the script processes the EBCDIC data, it prints a message for every batch of 10,000 records. See the following example.
------------------------------------------------------------------ 2023-05-15 21:21:46.322253 | Local Json file | -local-json | sample-data/cobpack2-list.json 2023-05-15 21:21:47.034556 | Records processed | 10000 2023-05-15 21:21:47.736434 | Records processed | 20000 2023-05-15 21:21:48.441696 | Records processed | 30000 2023-05-15 21:21:49.173781 | Records processed | 40000 2023-05-15 21:21:49.874779 | Records processed | 50000 2023-05-15 21:21:50.705873 | Records processed | 60000 2023-05-15 21:21:51.609335 | Records processed | 70000 2023-05-15 21:21:52.292989 | Records processed | 80000 2023-05-15 21:21:52.938366 | Records processed | 89280 2023-05-15 21:21:52.938448 Seconds 6.616232
For information about how to change the print frequency, see the [README file](https://github.com/aws-samples/mainframe-data-utilities/blob/main/README.md) in the GitHub repository. | General AWS |
| Examine the ASCII file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html)If you used the sample EBCDIC file provided, the following is the first record in the ASCII file.
| General AWS, Linux |
| Evaluate the EBCDIC file. | In the Amazon EC2 console, enter the following command. This opens the first record of the EBCDIC file.
head sample-data/COBPACK.OUTFILE.txt -c 150 | xxd
If you used the sample EBCDIC file, the following is the result.
To evaluate the equivalence between the source and target files, comprehensive knowledge of EBCDIC is required. For example, the first character of the sample EBCDIC file is a hyphen (`-`). In hexadecimal notation of the EBCDIC file, this character is represented by `60`, and in hexadecimal notation of the ASCII file, this character is represented by `2D`. For an EBCDIC-to-ASCII conversion table, see [EBCDIC to ASCII](https://www.ibm.com/docs/en/iis/11.3?topic=tables-ebcdic-ascii) on the IBM website. | General AWS, Linux, EBCDIC |
## Related resources
**References**
+ [The EBCDIC character set](https://www.ibm.com/docs/en/zos-basic-skills?topic=mainframe-ebcdic-character-set) (IBM documentation)
+ [EBCDIC to ASCII](https://www.ibm.com/docs/en/iis/11.3?topic=tables-ebcdic-ascii) (IBM documentation)
+ [COBOL](https://www.ibm.com/docs/en/i/7.1?topic=languages-cobol) (IBM documentation)
+ [Basic JCL concepts](https://www.ibm.com/docs/en/zos-basic-skills?topic=collection-basic-jcl-concepts) (IBM documentation)
+ [Connect to your Linux instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html) (Amazon EC2 documentation)
**Tutorials**
+ [Scheduling SSH jobs using AWS Lambda](https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/) (AWS blog post)
+ [Using an Amazon S3 trigger to invoke a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html) (AWS Lambda documentation)
# Convert mainframe files from EBCDIC format to character-delimited ASCII format in Amazon S3 using AWS Lambda
*Luis Gustavo Dantas, Amazon Web Services*
## Summary
This pattern shows you how to launch an AWS Lambda function that automatically converts mainframe Extended Binary Coded Decimal Interchange Code (EBCDIC) files to character-delimited American Standard Code for Information Interchange (ASCII) files. The Lambda function runs after the ASCII files are uploaded to an Amazon Simple Storage Service (Amazon S3) bucket. After the file conversion, you can read the ASCII files on x86-based workloads or load the files into modern databases.
The file conversion approach demonstrated in this pattern can help you overcome the challenges of working with EBCDIC files on modern environments. Files encoded in EBCDIC often contain data represented in a binary or packed decimal format, and fields are fixed-length. These characteristics create obstacles because modern x86-based workloads or distributed environments generally work with ASCII-encoded data and can’t process EBCDIC files.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An Amazon S3 bucket
+ An AWS Identity and Access Management (IAM) user with administrative permissions
+ AWS CloudShell
+ [Python 3.8.0](https://www.python.org/downloads/release/python-380/) or later
+ A flat file encoded in EBCDIC and its corresponding data structure in a common business-oriented language (COBOL) copybook
**Note**
This pattern uses a sample EBCDIC file ([CLIENT.EBCDIC.txt](https://github.com/aws-samples/mainframe-data-utilities/blob/main/sample-data/CLIENT.EBCDIC.txt)) and its corresponding COBOL copybook ([COBKS05.cpy](https://github.com/aws-samples/mainframe-data-utilities/blob/main/LegacyReference/COBKS05.cpy)). Both files are available in the GitHub [mainframe-data-utilities](https://github.com/aws-samples/mainframe-data-utilities) repository.
**Limitations**
+ COBOL copybooks usually hold multiple layout definitions. The [mainframe-data-utilities](https://github.com/aws-samples/mainframe-data-utilities) project can parse this kind of copybook but can't infer which layout to consider on data conversion. This is because copybooks don't hold this logic (which remains on COBOL programs instead). Consequently, you must manually configure the rules for selecting layouts after you parse the copybook.
+ This pattern is subject to [Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html).
## Architecture
**Source technology stack**
+ IBM z/OS, IBM i, and other EBCDIC systems
+ Sequential files with data encoded in EBCDIC (such as IBM Db2 unloads)
+ COBOL copybook
**Target technology stack**
+ Amazon S3
+ Amazon S3 event notification
+ IAM
+ Lambda function
+ Python 3.8 or later
+ Mainframe Data Utilities
+ JSON metadata
+ Character-delimited ASCII files
**Target architecture**
The following diagram shows an architecture for converting mainframe EBCDIC files to ASCII files.
![\[Architecture for converting mainframe EBCDIC files to ASCII files\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/97ab4129-2639-4733-86cb-962d91526df4/images/3ca7ca44-373a-434f-8c40-09e7c2abf5ec.png)
The diagram shows the following workflow:
1. The user runs the copybook parser script, which converts the COBOL copybook into a JSON file.
1. The user uploads the JSON metadata to an Amazon S3 bucket. This makes the metadata readable by the data conversion Lambda function.
1. The user or an automated process uploads the EBCDIC file to the Amazon S3 bucket.
1. The Amazon S3 notification event triggers the data conversion Lambda function.
1. AWS verifies the Amazon S3 bucket read-write permissions for the Lambda function.
1. Lambda reads the file from the Amazon S3 bucket and locally converts the file from EBCDIC to ASCII.
1. Lambda logs the process status in Amazon CloudWatch.
1. Lambda writes the ASCII file back to Amazon S3.
**Note**
The copybook parser script runs a single time to perform the metadata conversion to JSON format, which is subsequently stored in an Amazon S3 bucket. After the initial conversion, all subsequent EBCDIC files that reference the same JSON file in the Amazon S3 bucket will use the existing metadata configuration.
## Tools
**AWS services**
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) helps you monitor the metrics of your AWS resources and the applications that you run on AWS in real time.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html) is a browser-based shell that you can use to manage AWS services by using the AWS Command Line Interface (AWS CLI) and a range of preinstalled development tools.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. Lambda runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
**Other tools**
+ [GitHub](https://github.com/) is a code-hosting service that provides collaboration tools and version control.
+ [Python](https://www.python.org/) is a high-level programming language.
**Code**
The code for this pattern is available in the GitHub [mainframe-data-utilities](https://github.com/aws-samples/mainframe-data-utilities) repository.
## Best practices
Consider the following best practices:
+ Set the required permissions at the Amazon Resource Name (ARN) level.
+ Always grant least-privilege permissions for IAM policies. For more information, see [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation.
## Epics
### Create environment variables and a working folder
| Task | Description | Skills required |
| --- | --- | --- |
| Create the environment variables. | Copy the following environment variables to a text editor, and then replace the `` values in the following example with your resource values:
bucket= account= region=
You will create references to your Amazon S3 bucket, AWS account, and AWS Region later.To define environment variables, open the [CloudShell console](https://console.aws.amazon.com/cloudshell/), and then copy and paste your updated environment variables onto the command line.You must repeat this step every time the CloudShell session restarts. | General AWS |
| Create a working folder. | To simplify the resource clean-up process later on, create a working folder in CloudShell by running the following command:
mkdir workdir; cd workdir
You must change the directory to the working directory (`workdir`) every time you lose a connection to your CloudShell session. | General AWS |
### Define an IAM role and policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create a trust policy for the Lambda function. | The EBCDIC converter runs in a Lambda function. The function must have an IAM role. Before you create the IAM role, you must define a trust policy document that enables resources to assume that policy.From the CloudShell working folder, create a policy document by running the following command:
| General AWS |
| Create the IAM role for Lambda conversion. | To create an IAM role, run the following AWS CLI command from the CloudShell working folder:
aws iam create-role --role-name E2AConvLambdaRole --assume-role-policy-document file://E2ATrustPol.json
| General AWS |
| Create the IAM policy document for the Lambda function. | The Lambda function must have read-write access to the Amazon S3 bucket and write permissions for Amazon CloudWatch Logs.To create an IAM policy, run the following command from the CloudShell working folder:
| General AWS |
| Attach the IAM policy document to the IAM role. | To attach the IAM policy to the IAM role, enter the following command from your CloudShell working folder:
aws iam put-role-policy --role-name E2AConvLambdaRole --policy-name E2AConvLambdaPolicy --policy-document file://E2AConvLambdaPolicy.json
| General AWS |
### Create the Lambda function for EBCDIC conversion
| Task | Description | Skills required |
| --- | --- | --- |
| Download the EBCDIC conversion source code. | From the CloudShell working folder, run the following command to download the mainframe-data-utilities source code from GitHub:
| General AWS |
| Create the ZIP package. | From the CloudShell working folder, enter the following command to create the ZIP package that creates the Lambda function for EBCDIC conversion:
cd mdu; zip ../mdu.zip *.py; cd ..
| General AWS |
| Create the Lambda function. | From the CloudShell working folder, enter the following command to create the Lambda function for EBCDIC conversion:
The environment variable layout tells the Lambda function where the JSON metadata resides. | General AWS |
| Create the resource-based policy for the Lambda function. | From the CloudShell working folder, enter the following command to allow your Amazon S3 event notification to trigger the Lambda function for EBCDIC conversion:
| General AWS |
### Create the Amazon S3 event notification
| Task | Description | Skills required |
| --- | --- | --- |
| Create the configuration document for the Amazon S3 event notification. | The Amazon S3 event notification initiates the EBCDIC conversion Lambda function when files are placed in the input folder.From the CloudShell working folder, run the following command to create the JSON document for the Amazon S3 event notification:
| General AWS |
| Create the Amazon S3 event notification. | From the CloudShell working folder, enter the following command to create the Amazon S3 event notification:
| General AWS |
### Create and upload the JSON metadata
| Task | Description | Skills required |
| --- | --- | --- |
| Parse the COBOL copybook. | From the CloudShell working folder, enter the following command to parse a sample COBOL copybook into a JSON file (which defines how to read and slice the data file properly):
| General AWS |
| Add the transformation rule. | The sample data file and its corresponding COBOL copybook is a multi-layout file. This means that the conversion must slice data based on certain rules. In this case, bytes on position 3 and 4 in each row define the layout.From the CloudShell working folder, edit the `CLIENT.json` file and change the contents from `"transf-rule": [],` to the following:
| General AWS, IBM Mainframe, Cobol |
| Upload the JSON metadata to the Amazon S3 bucket. | From the CloudShell working folder, enter the following AWS CLI command to upload the JSON metadata to your Amazon S3 bucket:
| General AWS |
### Convert the EBCDIC file
| Task | Description | Skills required |
| --- | --- | --- |
| Send the EBCDIC file to the Amazon S3 bucket. | From the CloudShell working folder, enter the following command to send the EBCDIC file to the Amazon S3 bucket:
We recommend that you set different folders for input (EBCDIC) and output (ASCII) files to avoid calling the Lambda conversion function again when the ASCII file is uploaded to the Amazon S3 bucket. | General AWS |
| Check the output. | From the CloudShell working folder, enter the following command to check if the ASCII file is generated in your Amazon S3 bucket:
aws s3 ls s3://$bucket/
The data conversion can take several seconds to happen. We recommend that you check for the ASCII file a few times.After the ASCII file is available, enter the following command to view the contents of the converted file in the Amazon S3 bucket. As needed, you can download it or use it directly from the Amazon S3 bucket:
| General AWS |
### Clean the environment
| Task | Description | Skills required |
| --- | --- | --- |
| (Optional) Prepare the variables and folder. | If you lose connection with CloudShell, reconnect and then enter the following command to change the directory to the working folder:
cd workdir
Ensure that the environment variables are defined:
bucket= account= region=
| General AWS |
| Remove the notification configuration for the bucket. | From the CloudShell working folder, run the following command to remove the Amazon S3 event notification configuration:
| General AWS |
| Delete the Lambda function. | From the CloudShell working folder, enter the following command to delete the Lambda function for the EBCDIC converter:
aws lambda delete-function \ --function-name E2A
| General AWS |
| Delete the IAM role and policy. | From the CloudShell working folder, enter the following command to remove the EBCDIC converter role and policy:
aws iam delete-role-policy \ --role-name E2AConvLambdaRole \ --policy-name E2AConvLambdaPolicy
aws iam delete-role \ --role-name E2AConvLambdaRole
| General AWS |
| Delete the files generated in the Amazon S3 bucket. | From the CloudShell working folder, enter the following command to delete the files generated in the Amazon S3 bucket:
| General AWS |
| Delete the working folder. | From the CloudShell working folder, enter the following command to remove `workdir` and its contents:
cd ..; rm -Rf workdir
| General AWS |
## Related resources
+ [Mainframe Data Utilities README](https://github.com/aws-samples/mainframe-data-utilities/blob/main/README.md) (GitHub)
+ [The EBCDIC character set](https://www.ibm.com/docs/en/zos-basic-skills?topic=mainframe-ebcdic-character-set) (IBM documentation)
+ [EBCDIC to ASCII](https://www.ibm.com/docs/en/iis/11.7.0?topic=tables-ebcdic-ascii) (IBM documentation)
+ [COBOL](https://www.ibm.com/docs/en/i/7.6.0?topic=languages-cobol) (IBM documentation)
+ [Using an Amazon S3 trigger to invoke a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html) (AWS Lambda documentation)
# Convert mainframe data files with complex record layouts using Micro Focus
*Peter West, Amazon Web Services*
## Summary
Note: AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
This pattern shows you how to convert mainframe data files with non-text data and complex record layouts from EBCDIC (Extended Binary Coded Decimal Interchange Code) character encoding to ASCII (American Standard Code for Information Interchange) character encoding by using a Micro Focus structure file. To complete the file conversion, you must do the following:
1. Prepare a single source file that describes all the data items and record layouts in your mainframe environment.
1. Create a structure file that contains the record layout of the data by using the Micro Focus Data File Editor as part of the Micro Focus Classic Data File Tools or Data File Tools. The structure file identifies the non-text data so that you can correctly convert your mainframe files from EBCDIC to ASCII.
1. Test the structure file by using the Classic Data File Tools or Data File Tools.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Micro Focus Enterprise Developer for Windows, available through [AWS Mainframe Modernization](https://aws.amazon.com/mainframe-modernization/)
**Product versions**
+ Micro Focus Enterprise Server 7.0 and later
## Tools
+ [Micro Focus Enterprise Developer](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/GUID-8D6B7358-AC35-4DAF-A445-607D8D97EBB2.html) provides the run environment for applications created with any integrated development environment (IDE) variant of Enterprise Developer.
+ Micro Focus [Classic Data File Tools](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/GUID-06115324-0FBC-4CB7-BE9D-04BCFEA5821A.html) help you convert, navigate, edit, and create data files. The Classic Data File Tools include [Data File Converter](https://www.microfocus.com/documentation/visual-cobol/vc60/VS2017/BKFHFHDFCV.html), [Record Layout Editor](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/BKFHFHRLMF.html), and [Data File Editor](https://www.microfocus.com/documentation/visual-cobol/vc60/VS2017/BKFHFHDFED.html).
+ Micro Focus [Data File Tools](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/GUID-B1BCB613-6947-451C-8F71-72FB8254076A.html) help you create, edit, and move data files. The Data File Tools include [Data File Editor](https://www.microfocus.com/documentation/visual-cobol/vc60/VS2017/BKFHFHDFED.html), [File Conversion Utilities](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/BKFHFHCONV.html), and the [Data File Structure Command Line Utility](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/GUID-E84348EB-A93A-481A-A47C-61B0E1C076E6.html).
## Epics
### Prepare the source file
| Task | Description | Skills required |
| --- | --- | --- |
| Identify source components. | Identify all possible record layouts for the file, including any redefinitions that contain non-text data.If you have layouts that contain redefinitions, you must factor these layouts down to unique layouts that describe each possible permutation of the data structure. Typically, a data file’s record layouts can be described by the following archetypes:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html)For more information on creating flattened record layouts for files that contain complex record layouts, see [Rehosting EBCDIC applications on ASCII environments for mainframe migrations](https://docs.aws.amazon.com/prescriptive-guidance/latest/mainframe-rehost-ebcdic-ascii/introduction.html). | App developer |
| Identify record layout conditions. | For files with multiple record layouts or files that contain complex layouts with a REDEFINES clause, identify the data and conditions within a record that you can use to define which layout to use during conversion. We recommend that you discuss this task with a subject matter expert (SME) who understands the programs that process these files.For example, a file might contain two record types that contain non-text data. You can inspect the source and possibly find code similar to the following:
MOVE "M" TO PART-TYPE MOVE "MAIN ASSEMBLY" TO PART-NAME MOVE "S" TO PART-TYPE MOVE "SUB ASSEMBLY 1" TO PART-NAME
The code helps you identify the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html)You can document the values that are used by this field to associate the record layouts with the correct data records in the file. | App developer |
| Build the source file. | If the file is described over multiple source files or if the record layout contains non-text data that’s subordinate to a REDEFINES clause, then create a new source file that contains the record layouts. The new program doesn’t need to describe the file using SELECT and FD statements. The program can simply contain the record descriptions as 01 levels within Working-Storage.You can create a source file for each data file or create a master source file that describes all the data files. | App developer |
| Compile the source file. | Compile the source file to build the data dictionary. We recommend that you compile the source file by using the EBCDIC character set. If the IBMCOMP directive or ODOSLIDE directives are being used, then you must use these directives in the source file too.IBMCOMP affects the byte storage of COMP fields and ODOSLIDE affects padding on OCCURS VARYING structures. If these directives are set incorrectly, then the conversion tool won’t read the data record correctly. This results in bad data in the converted file. | App developer |
### (Option A) Create the structure file using Classic Data File Tools
| Task | Description | Skills required |
| --- | --- | --- |
| Start the tool and load the dictionary. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
| Create the default record layout. | Use the default record layout for all records that don’t match any conditional layouts.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html)The default layout appears in the **Layouts** pane and can be identified by the red folder icon. | App developer |
| Create a conditional record layout. | Use the conditional record layout when there is more than one record layout in a file.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
### (Option B) Create the structure file using Data File Tools
| Task | Description | Skills required |
| --- | --- | --- |
| Start the tool and load the dictionary. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
| Create the default record layout. | Use the default record layout for all records that do not match any conditional layouts.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html)The default layout appears in the **Layouts** pane and can be identified by the blue "D" icon. | App developer |
| Create a conditional record layout. | Use the conditional record layout when there is more than one record layout in a file.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
### (Option A) Test the structure file using Classic Data File Tools
| Task | Description | Skills required |
| --- | --- | --- |
| Test an EBCDIC data file. | Confirm that you can use your structure file to view an EBCDIC test data file correctly.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
### (Option B) Test the structure file using Data File Tools
| Task | Description | Skills required |
| --- | --- | --- |
| Test an EBCDIC data file. | Confirm that you can use your structure file to view an EBCDIC test data file correctly.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
### Test data file conversion
| Task | Description | Skills required |
| --- | --- | --- |
| Test the conversion of an EBCDIC file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-data-files-with-complex-record-layouts-using-micro-focus.html) | App developer |
## Related resources
+ [Micro Focus](https://www.microfocus.com/en-us/products/enterprise-suite/overview) (Micro Focus documentation)
+ [Mainframe and legacy code](https://aws.amazon.com/blogs/?awsf.blog-master-category=category%23mainframe-and-legacy) (AWS Blog posts)
+ [AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/) (AWS documentation)
+ [AWS Documentation](https://docs.aws.amazon.com/index.html) (AWS documentation)
+ [AWS General Reference](https://docs.aws.amazon.com/general/latest/gr/Welcome.html) (AWS documentation)
+ [AWS glossary](https://docs.aws.amazon.com/general/latest/gr/glos-chap.html) (AWS documentation)
# Deploy an environment for containerized Blu Age applications by using Terraform
*Richard Milner-Watts, Amazon Web Services*
## Summary
Migrating legacy mainframe workloads into modern cloud architectures can eliminate the costs of maintaining a mainframe—costs that only increase as the environment ages. However, migrating jobs from a mainframe can pose unique challenges. Internal resources might not be familiar with the job logic, and the high performance of mainframes at these specialized tasks can be difficult to replicate when compared to commodity, generalized CPUs. Rewriting these jobs can be a large undertaking and require significant effort.
Blu Age converts legacy mainframe workloads into modern Java code, which you can then run as a container.
This pattern provides a sample serverless architecture for running a containerized application that has been modernized with the Blu Age tool. The included HashiCorp Terraform files will build a secure architecture for the orchestration of Blu Age containers, supporting both batch tasks and real-time services.
For more information about modernizing your workloads by using Blu Age and AWS services, see these AWS Prescriptive Guidance publications:
+ [Running mainframe workloads that have been modernized with Blu Age on AWS serverless infrastructure](https://docs.aws.amazon.com/prescriptive-guidance/latest/run-bluage-modernized-mainframes/)
+ [Containerize mainframe workloads that have been modernized by Blu Age](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.html)
For assistance with using Blu Age to modernize your mainframe workloads, contact the Blu Age team by choosing **Contact our experts** on the [Blu Age website](https://www.bluage.com/). For assistance with migrating your modernized workloads to AWS, integrating them with AWS services, and moving them into production, contact your AWS account manager or fill out the [AWS Professional Services form](https://pages.awscloud.com/AWS-Professional-Services.html).
## Prerequisites and limitations
**Prerequisites**
+ The sample containerized Blu Age application provided by the [Containerize mainframe workloads that have been modernized by Blu Age](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.html) pattern. The sample application provides the logic to handle the processing of input and output for the modernized application, and it can integrate with this architecture.
+ Terraform is required to deploy these resources.
**Limitations**
+ Amazon Elastic Container Service (Amazon ECS) places limits on the task resources that can be made available to the container. These resources include CPU, RAM, and storage. For example, when using Amazon ECS with AWS Fargate, the [task resource limits apply](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html).
**Product versions**
This solution was tested with the following versions:
+ Terraform 1.3.6
+ Terraform AWS Provider 4.46.0
## Architecture
**Source technology stack**
+ Blu Age
+ Terraform
**Target technology stack**
+ Amazon Aurora PostgreSQL-Compatible Edition
+ AWS Backup
+ Amazon Elastic Container Registry (Amazon ECR)
+ Amazon ECS
+ AWS Identity and Access Management Service (IAM)
+ AWS Key Management Server (AWS KMS)
+ AWS Secrets Manager
+ Amazon Simple Notification Service (Amazon SNS)
+ Amazon Simple Storage Service (Amazon S3)
+ AWS Step Functions
+ AWS Systems Manager
**Target architecture**
The following diagram shows the solution architecture.
![\[The description follows the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/12825490-2622-4f0b-80c9-2c5076d50fa3/images/c0708b0a-aa36-458a-8d6c-d42e3dec7727.png)
1. The solution deploys the following IAM roles:
+ Batch task role
+ Batch task execution role
+ Service task role
+ Service task execution role
+ Step Functions role
+ AWS Backup role
+ RDS Enhanced Monitoring role.
The roles conform to least-privileged access principles.
1. Amazon ECR is used to store the container image that is orchestrated by this pattern.
1. AWS Systems Manager Parameter Store provides configuration data about each environment to the Amazon ECS task definition at runtime.
1. AWS Secrets Manager provides sensitive configuration data about the environment to the Amazon ECS task definition at runtime. The data has been encrypted by AWS KMS.
1. The Terraform modules create Amazon ECS task definitions for all real-time and batch tasks.
1. Amazon ECS runs a batch task by using AWS Fargate as the compute engine. This is a short-lived task, initiated as required by AWS Step Functions.
1. Amazon Aurora PostgreSQL-Compatible provides a database to support the modernized application. This replaces mainframe databases such as IBM Db2 or IBM IMS DB.
1. Amazon ECS runs a long-lived service to deliver a modernized real-time workload. These stateless applications run permanently with containers spread across Availability Zones.
1. A Network Load Balancer is used to grant access to the real-time workload. The Network Load Balancer supports earlier protocols, such as IBM CICS. Alternatively, you can use an Application Load Balancer with HTTP-based workloads.
1. Amazon S3 provides object storage for job inputs and outputs. The container should handle pull and push operations into Amazon S3 to prepare the working directory for the Blu Age application.
1. The AWS Step Functions service is used to orchestrate running the Amazon ECS tasks to process batch workloads.
1. SNS topics for each batch workload are used to integrate the modernized application with other systems, such as email, or to initiate additional actions, such as delivering output objects from Amazon S3 into FTP.
**Note**
By default, the solution has no access to the internet. This pattern assumes that the virtual private cloud (VPC) will be connected to other networks using a service such as [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/). As such, multiple interface VPC endpoints are deployed to grant access to the AWS services used by the solution. To turn on direct internet access, you can use the toggle in the Terraform module to replace the VPC endpoints with an internet gateway and the associated resources.
**Automation and scale**
The use of serverless resources throughout this pattern helps to ensure that, by scaling out, there are few limits on the scale of this design. This reduces *noisy neighbor concerns*, such as the competition for compute resources that might be experienced on the original mainframe. Batch tasks can be scheduled to run simultaneously as needed.
Individual containers are limited by the maximum sizes supported by Fargate. For more information, see the [https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#fargate-tasks-size](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#fargate-tasks-size) section in the Amazon ECS documentation.
To [scale real-time workloads horizontally](https://nathanpeck.com/amazon-ecs-scaling-best-practices/), you can add containers.
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html) is a fully managed service that helps you centralize and automate data protection across AWS services, in the cloud, and on premises.
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable.
+ [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a fast and scalable container management service that helps you run, stop, and manage containers on a cluster.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
+ [Amazon Simple Notification Service (Amazon SNS)](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications.
+ [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) provides secure, hierarchical storage for configuration data management and secrets management.
**Other services**
+ [HashiCorp Terraform](https://www.terraform.io/docs) is an infrastructure as code (IaC) tool that helps you use code to provision and manage cloud infrastructure and resources. This pattern uses Terraform to create the sample architecture.
**Code repository**
The source code for this pattern is available in the GitHub [Blu Age Sample ECS Infrastructure (Terraform)](https://github.com/aws-samples/aws-blu-age-sample-ecs-infrastructure-using-terraform#aws-blu-age-sample-ecs-infrastructure-terraform) repository.
## Best practices
+ For test environments, use features such as the `forceDate` option to configure the modernized application to generate consistent test results by always running for a known time period.
+ Tune each task individually to consume the optimal amount of resources. You can use [Amazon CloudWatch Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html) to obtain guidance on potential bottlenecks.
## Epics
### Prepare the environment for deployment
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the solution source code. | Clone the solution code from the [GitHub project](https://github.com/aws-samples/aws-blu-age-sample-ecs-infrastructure-using-terraform). | DevOps engineer |
| Bootstrap the environment by deploying resources to store the Terraform state. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.html) | DevOps engineer |
### Deploy the solution infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Review and update the Terraform configuration. | In the root directory, open the file `main.tf,` review the contents, and consider making the following updates:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.html) | DevOps engineer |
| Deploy the Terraform file. | From your terminal, run the `terraform apply` command to deploy all resources. Review the changes generated by Terraform, and enter **yes** to initiate the build.Note that it can take over 15 minutes to deploy this infrastructure. | DevOps engineer |
### (Optional) Deploy a valid Blu Age containerized application
| Task | Description | Skills required |
| --- | --- | --- |
| Push the Blu Age container image to Amazon ECR. | Push the container into the Amazon ECR repository that you created in the previous epic. For instructions, see the [Amazon ECR documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html).Make a note of the container image URI. | DevOps engineer |
| Update the Terraform to reference the Blu Age container image. | Update the file `main.tf`** **to reference the container image that you uploaded. | DevOps engineer |
| Redeploy the Terraform file. | From your terminal, run `terraform apply` to deploy all resources. Review the suggested updates from Terraform, and then enter **yes** to proceed with the deployment. | DevOps engineer |
## Related resources
+ [Blu Age](https://www.bluage.com/)
+ [Running mainframe workloads that have been modernized with Blu Age on AWS serverless infrastructure](https://docs.aws.amazon.com/prescriptive-guidance/latest/run-bluage-modernized-mainframes/)
+ [Containerize mainframe workloads that have been modernized by Blu Age](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.html)
# Generate Db2 z/OS data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight
*Shubham Roy, Roshna Razack, and Santosh Kumar Singh, Amazon Web Services*
## Summary
Note: AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
If your organization is hosting business-critical data in an IBM Db2 mainframe environment, gaining insights from that data is crucial for driving growth and innovation. By unlocking mainframe data, you can build faster, secure, and scalable business intelligence to accelerate data-driven decision-making, growth, and innovation in the Amazon Web Services (AWS) Cloud.
This pattern presents a solution for generating business insights and creating sharable narratives from mainframe data in IBM Db2 for z/OS tables. Mainframe data changes are streamed to [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) topic using [AWS Mainframe Modernization Data Replication with Precisely](https://docs.aws.amazon.com/m2/latest/userguide/precisely.html). Using [Amazon Redshift streaming ingestion](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html), Amazon MSK topic data is stored in [Amazon Redshift Serverless](https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-whatis.html) data warehouse tables for analytics in Amazon Quick Sight.
After the data is available in Quick Sight, you can use natural language prompts with [Amazon Q in Quick Sight](https://docs.aws.amazon.com/quicksight/latest/user/quicksight-gen-bi.html) to create summaries of the data, ask questions, and generate data stories. You don't have to write SQL queries or learn a business intelligence (BI) tool.
**Business context**
This pattern presents a solution for mainframe data analytics and data insights use cases. Using the pattern, you build a visual dashboard for your company's data. To demonstrate the solution, this pattern uses a health care company that provides medical, dental, and vision plans to its members in the US. In this example, member demographics and plan information are stored in the IBM Db2 for z/OS data tables. The visual dashboard shows the following:
+ Member distribution by region
+ Member distribution by gender
+ Member distribution by age
+ Member distribution by plan type
+ Members who have not completed preventive immunization
For examples of member distribution by region and members who have not completed preventive immunization, see the Additional information section.
After you create the dashboard, you generate a data story that explains the insights from the previous analysis. The data story provides recommendations for increasing the number of members who have completed preventive immunizations.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account. This solution was built and tested on Amazon Linux 2 on Amazon Elastic Compute Cloud (Amazon EC2).
+ An virtual private cloud (VPC) with a subnet that can be accessed by your mainframe system.
+ A mainframe database with business data. For the example data used to build and test this solution, see the *Attachments* section.
+ Change data capture (CDC) enabled on the Db2 z/OS tables. To enable CDC on Db2 z/OS, see the [IBM documentation](https://www.ibm.com/docs/en/daafz/7.5?topic=cdc-enabling-data-capture-changes).
+ Precisely Connect CDC for z/OS installed on the z/OS system that's hosting the source databases. The Precisely Connect CDC for z/OS image is provided as a zip file within the [AWS Mainframe Modernization - Data Replication for IBM z/OS](https://aws.amazon.com/marketplace/pp/prodview-doe2lroefogia?applicationId=AWSMPContessa&ref_=beagle&sr=0-1) Amazon Machine Image (AMI). To install Precisely Connect CDC for z/OS on the mainframe, see the [Precisely installation documentation](https://help.precisely.com/r/AWS-Mainframe-Modernization/Latest/en-US/AWS-Mainframe-Modernization-Data-Replication-for-IBM-z/OS/Install-Precisely-Connect-CDC-z/OS).
**Limitations**
+ Your mainframe Db2 data should be in a data type that's supported by Precisely Connect CDC. For a list of supported data types, see the [Precisely Connect CDC documentation](https://help.precisely.com/r/AWS-Mainframe-Modernization/Latest/en-US/AWS-Mainframe-Modernization-Data-Replication-for-IBM-z/OS/Data-replication-overview/Supported-source-data-types).
+ Your data at Amazon MSK should be in a data type that's supported by Amazon Redshift. For a list of supported data types, see the [Amazon Redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html).
+ Amazon Redshift has different behaviors and size limits for different data types. For more information, see the [Amazon Redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html#materialized-view-streaming-ingestion-limitations).
+ The near real-time data in Quick Sight depends on the refresh interval set for the Amazon Redshift database.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). Amazon Q in Quick Sight is currently not available in every Region that supports Quick Sight. For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
**Product versions**
+ AWS Mainframe Modernization Data Replication with Precisely version 4.1.44
+ Python version 3.6 or later
+ Apache Kafka version** **3.5.1
## Architecture
**Target architecture**
The following diagram shows an architecture for generating business insights from mainframe data by using [AWS Mainframe Modernization Data Replication with Precisely](https://aws.amazon.com/mainframe-modernization/capabilities/data-replication/) and Amazon Q in Quick Sight.
![\[Seven-step process from z/OS mainframe to Amazon QuickSight.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/cddb6d20-14ae-4276-90d8-14df435db824.png)
The diagram shows the following workflow:
1. The Precisely Log Reader Agent reads data from Db2 logs and writes the data into transient storage on an OMVS file system on the mainframe.
1. The Publisher Agent reads the raw Db2 logs from transient storage.
1. The on-premises controller daemon authenticates, authorizes, monitors, and manages operations.
1. The Apply Agent is deployed on Amazon EC2 by using the preconfigured AMI. It connects with the Publisher Agent through the controller daemon by using TCP/IP. The Apply Agent pushes data to Amazon MSK using multiple workers for high-throughput.
1. The workers write the data to the Amazon MSK topic in JSON format. As the intermediate target for the replicated messages, Amazon MSK provides the highly available and automated failover capabilities.
1. Amazon Redshift streaming ingestion provides low-latency, high-speed data ingestion from Amazon MSK to an Amazon Redshift Serverless database. A stored procedure in Amazon Redshift performs the mainframe change data (insert/update/deletes) reconciliation into Amazon Redshift tables. These Amazon Redshift tables serves as the data analytics source for Quick Sight.
1. Users access the data in Quick Sight for analytics and insights. You can use Amazon Q in Quick Sight to interact with the data by using natural language prompts.
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them out or in.
+ [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data.
+ [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) is a fully managed service that helps you build and run applications that use Apache Kafka to process streaming data.
+ [Amazon Quick Sight](https://docs.aws.amazon.com/quicksight/latest/user/welcome.html) is a cloud-scale business intelligence (BI) service that helps you visualize, analyze, and report your data in a single dashboard. This pattern uses the generative BI capabilities of Amazon Q in Quick Sight.
+ [Amazon Redshift Serverless](https://aws.amazon.com/redshift/redshift-serverless/) is a serverless option of Amazon Redshift that makes it more efficient to run and scale analytics in seconds without the need to set up and manage data warehouse infrastructure.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
**Other tools**
+ [Precisely Connect CDC](https://support.precisely.com/products/connect-cdc-formerly-sqdata/) collects and integrates data from legacy systems into cloud and data platforms.
**Code repository**
The code for this pattern is available in the GitHub [Mainframe\$1DataInsights\$1change\$1data\$1reconciliation](https://github.com/aws-samples/Mainframe_DataInsights_change_data_reconcilition) repository. The code is a stored procedure in Amazon Redshift. This stored procedure reconciles mainframe data changes (inserts, updates, and deletes) from Amazon MSK into the Amazon Redshift tables. These Amazon Redshift tables serve as the data analytics source for Quick Sight.
## Best practices
+ Follow [best practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html) while setting up your Amazon MSK cluster.
+ Follow Amazon Redshift [data parsing best practices](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html#materialized-view-streaming-recommendations) for improving performance.
+ When you create the AWS Identity and Access Management (IAM) roles for the Precisely setup, follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation.
## Epics
### Set up AWS Mainframe Modernization Data Replication with Precisely on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Set up a security group. | To connect to the controller daemon and the Amazon MSK cluster, [create a security group](https://docs.aws.amazon.com/vpc/latest/userguide/creating-security-groups.html) for the EC2 instance. Add the following inbound and outbound rules:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html)Note the name of the security group. You will need to reference the name when you launch the EC2 instance and configure the Amazon MSK cluster. | DevOps engineer, AWS DevOps |
| Create an IAM policy and an IAM role. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | DevOps engineer, AWS systems administrator |
| Provision an EC2 instance. | To provision an EC2 instance to run Precisely CDC and connect to Amazon MSK, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | AWS administrator, DevOps engineer |
### Set up Amazon MSK
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Amazon MSK cluster. | To create an Amazon MSK cluster, do the following :[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html)A typical provisioned cluster takes up to 15 minutes to create. After the cluster is created, its status changes from **Creating** to **Active**. | AWS DevOps, Cloud administrator |
| Set up SASL/SCRAM authentication. | To set up SASL/SCRAM authentication for an Amazon MSK cluster, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Cloud architect |
| Create the Amazon MSK topic. | To create the Amazon MSK topic, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Cloud administrator |
### Configure the Precisely Apply Engine on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the Precisely scripts to replicate data changes. | To set up the Precisely Connect CDC scripts to replicate changed data from the mainframe to the Amazon MSK topic, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html)For example .ddl files, see the [Additional information](#generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight-additional) section. | App developer, Cloud architect |
| Generate the network ACL key. | To generate the network access control list (network ACL) key, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Cloud architect, AWS DevOps |
### Prepare the mainframe source environment
| Task | Description | Skills required |
| --- | --- | --- |
| Configure defaults in the ISPF screen. | To configure default settings in the Interactive System Productivity Facility (ISPF), follow the instructions in the [Precisely documentation](https://help.precisely.com/r/AWS-Mainframe-Modernization/Latest/en-US/AWS-Mainframe-Modernization-Data-Replication-for-IBM-z/OS/Install-Precisely-Connect-CDC-z/OS/Start-ISPF-Panel-Interface). | Mainframe system administrator |
| Configure the controller daemon. | To configure the controller daemon, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Mainframe system administrator |
| Configure the publisher. | To configure the publisher, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Mainframe system administrator |
| Update the daemon configuration file. | To update the publisher details in the controller daemon configuration file, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Mainframe system administrator |
| Create the job to start the controller daemon. | To create the job, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Mainframe system administrator |
| Generate the capture publisher JCL file. | To generation the capture publisher JCL file, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Mainframe system administrator |
| Check and update CDC. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Mainframe system administrator |
| Submit the JCL files. | Submit the following JCL files that you configured in the previous steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html)After you submit the JCL files, you can start the Apply Engine in Precisely on the EC2 instance. | Mainframe system administrator |
### Run and validate CDC
| Task | Description | Skills required |
| --- | --- | --- |
| Start the Apply Engine and validate the CDC. | To start the Apply Engine on the EC2 instance and validate the CDC, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Cloud architect, App developer |
| Validate the records on the Amazon MSK topic. | To read the message from the Kafka topic, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | App developer, Cloud architect |
### Store mainframe change data in an Amazon Redshift Serverless data warehouse
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Amazon Redshift Serverless. | To create an Amazon Redshift Serverless data warehouse, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/redshift/latest/gsg/new-user-serverless.html).On the Amazon Redshift Serverless dashboard, validate that the namespace and workgroup were created and are available. For this example pattern, the process might take 2‒5 minutes. | Data engineer |
| Set up the IAM role and trust policy required for streaming ingestion. | To set up Amazon Redshift Serverless streaming ingestion from Amazon MSK, do following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Data engineer |
| Connect Amazon Redshift Serverless to Amazon MSK. | To connect to the Amazon MSK topic, create an external schema in Amazon Redshift Serverless. In Amazon Redshift query editor v2, run the following SQL command, replacing `'iam_role_arn'` with the role that you created previously and replacing `'MSK_cluster_arn`' with the ARN for your cluster.
CREATE EXTERNAL SCHEMA member_schema FROM MSK IAM_ROLE 'iam_role_arn' AUTHENTICATION iam URI 'MSK_cluster_arn';
| Migration engineer |
| Create a materialized view. | To consume the data from the Amazon MSK topic in Amazon Redshift Serverless, create a materialized view. In Amazon Redshift query editor v2, run the following SQL commands, replacing `` with the name of your Amazon MSK topic.
CREATE MATERIALIZED VIEW member_view AUTO REFRESH YES AS SELECT kafka_partition, kafka_offset, refresh_time, json_parse(kafka_value) AS Data FROM member_schema. WHERE CAN_JSON_PARSE(kafka_value);
| Migration engineer |
| Create target tables in Amazon Redshift. | Amazon Redshift tables provide the input for Quick Sight. This pattern uses the tables `member_dtls` and `member_plans`, which match the source Db2 tables on the mainframe.To create the two tables in Amazon Redshift, run the following SQL commands in Amazon Redshift query editor v2:
-- Table 1: members_dtls CREATE TABLE members_dtls ( memberid INT ENCODE AZ64, member_name VARCHAR(100) ENCODE ZSTD, member_type VARCHAR(50) ENCODE ZSTD, age INT ENCODE AZ64, gender CHAR(1) ENCODE BYTEDICT, email VARCHAR(100) ENCODE ZSTD, region VARCHAR(50) ENCODE ZSTD ) DISTSTYLE AUTO;
| Migration engineer |
| Create a stored procedure in Amazon Redshift. | This pattern uses a stored procedure to sync-up change data (`INSERT`, `UPDATE`, `DELETE`) from the source mainframe to the target Amazon Redshift data warehouse table for analytics in Quick Sight.To create the stored procedure in Amazon Redshift, use query editor v2 to run the stored procedure code that's in the GitHub repository. | Migration engineer |
| Read from the streaming materialized view and load to the target tables. | The stored procedure reads data change from the streaming materialized view and loads the data changes to the target tables. To run the stored procedure, use the following command:
call SP_Members_Load();
You can use [Amazon EventBridge](https://aws.amazon.com/eventbridge/) to schedule the jobs in your Amazon Redshift data warehouse to call this stored procedure based on your data latency requirements. EventBridge runs jobs at fixed intervals. To monitor whether the previous call to the procedure completed, you might need to use a mechanism such as an [AWS Step Functions](https://aws.amazon.com/step-functions/) state machine. For more information, see the following resources:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html)Another option is to use Amazon Redshift query editor v2 to schedule the refresh. For more information, see [Scheduling a query with query editor v2](https://docs.aws.amazon.com/redshift/latest/mgmt/query-editor-v2-schedule-query.html). | Migration engineer |
### Connect Quick Sight to data in Amazon Redshift
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Quick Sight. | To set up Quick Sight, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/setting-up.html). | Migration engineer |
| Set up a secure connection between Quick Sight and Amazon Redshift. | To set up secure a connection between Quick Sight and Amazon Redshift, do the following[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Migration engineer |
| Create a dataset for Quick Sight. | To create a dataset for Quick Sight from Amazon Redshift, do following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Migration engineer |
| Join the dataset. | To create analytics in Quick Sight, join the two tables by following the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/joining-data.html#create-a-join).In the **Join Configuration** pane, choose **Left** for **Join type**. Under **Join clauses**, use `memberid from member_plans = memberid from members_details`. | Migration engineer |
### Get business insights from the mainframe data by using Amazon Q in Quick Sight
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Amazon Q in Quick Sight. | To set up the Amazon Q in Quick Sight generative BI capability, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/generative-bi-get-started.html). | Migration engineer |
| Analyze mainframe data and build a visual dashboard. | To analyze and visualize your data in Quick Sight, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html)When you're finished, you can publish your dashboard to share with others in your organization. For examples, see *Mainframe visual dashboard* in the [Additional information](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html#generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight-additional) section. | Migration engineer |
### Create a data story with Amazon Q in Quick Sight from mainframe data
| Task | Description | Skills required |
| --- | --- | --- |
| Create a data story. | Create a data story to explain insights from the previous analysis, and generate a recommendation to increase preventive immunization for members:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Migration engineer |
| View the generated data story. | To view the generated data story, choose that story on the **Data stories** page. | Migration engineer |
| Edit a generated data story. | To change the formatting, layout, or visuals in a data story, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/working-with-stories-edit.html). | Migration engineer |
| Share a data story. | To share a data story, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/working-with-stories-share.html). | Migration engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| For Quick Sight to Amazon Redshift dataset creation, `Validate Connection` has faled. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) |
| Trying to start the Apply engine on the EC2 instance returns the following error:`-bash: sqdeng: command not found` | Export the `sqdata` installation path by running following command:
|
| Trying to start the Apply Engine returns one of the following connection errors:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-db2-zos-data-insights-aws-mainframe-modernization-amazon-q-in-quicksight.html) | Check the mainframe spool to make sure that the controller daemon jobs are running. |
## Related resources
+ [Generate insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html?did=pg_card&trk=pg_card) (pattern)
+ [Generate data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight](https://youtu.be/F8b7l79p6TM?si=gASuQtFbMVuEm7IJ) (demo)
+ [AWS Mainframe Modernization - Data Replication for IBM z/OS](https://aws.amazon.com/marketplace/pp/prodview-doe2lroefogia?sr=0-4&ref_=beagle&applicationId=AWSMPContessa)
+ [Amazon Redshift streaming ingestion to a materialized view](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html)
## Additional information
**Example .ddl files**
*members\$1details.ddl*
```
CREATE TABLE MEMBER_DTLS (
memberid INTEGER NOT NULL,
member_name VARCHAR(50),
member_type VARCHAR(20),
age INTEGER,
gender CHAR(1),
email VARCHAR(100),
region VARCHAR(20)
);
```
*member\$1plans.ddl*
```
CREATE TABLE MEMBER_PLANS (
memberid INTEGER NOT NULL,
medical_plan CHAR(1),
dental_plan CHAR(1),
vision_plan CHAR(1),
preventive_immunization VARCHAR(20)
);
```
**Example .sqd file**
Replace** **`` with your Amazon MSK topic name.
*script.sqd*
```
-- Name: DB2ZTOMSK: DB2z To MSK JOBNAME DB2ZTOMSK;REPORT EVERY 1;OPTIONS CDCOP('I','U','D');-- Source Descriptions
JOBNAME DB2ZTOMSK;
REPORT EVERY 1;
OPTIONS CDCOP('I','U','D');
-- Source Descriptions
BEGIN GROUP DB2_SOURCE;
DESCRIPTION DB2SQL /var/precisely/di/sqdata/apply/DB2ZTOMSK/ddl/mem_details.ddl AS MEMBER_DTLS;
DESCRIPTION DB2SQL /var/precisely/di/sqdata/apply/DB2ZTOMSK/ddl/mem_plans.ddl AS MEMBER_PLANS;
END GROUP;
-- Source Datastore
DATASTORE cdc:///DB2ZTOMSK/DB2ZTOMSK
OF UTSCDC
AS CDCIN
DESCRIBED BY GROUP DB2_SOURCE ;
-- Target Datastore(s)
DATASTORE 'kafka:////key'
OF JSON
AS TARGET
DESCRIBED BY GROUP DB2_SOURCE;
PROCESS INTO TARGET
SELECT
{
REPLICATE(TARGET)
}
FROM CDCIN;
```
**Mainframe visual dashboard**
The following data visual was created by Amazon Q in Quick Sight for the analysis question `show member distribution by region`*.*
![\[Northeast and Southwest have 8 members, Southwest has 5 members, Midwest has 4 members.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/b40a784c-c1fc-444b-b6df-8bd1f7a6abaa.png)
The following data visual was created by Amazon Q in Quick Sight for the question `show member distribution by Region who have not completed preventive immunization, in pie chart`.
![\[Southeast shows 6, Southwest shows 5, and Midwest shows 4.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/8a95da3c-df4a-458b-9cfe-44e34f80a235.png)
**Data story output**
The following screenshots show sections of the data story created by Amazon Q in Quick Sight for the prompt `Build a data story about Region with most numbers of members. Also show the member distribution by age, member distribution by gender. Recommend how to motivate members to complete immunization. Include 4 points of supporting data for this pattern`.
In the introduction, the data story recommends choosing the region with the most members to gain the greatest impact from immunization efforts.
![\[Introduction screen for analysis based on geographic, demographic, and age of the member base.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/40f13957-2db4-42b7-b7a4-a0dd3dad6899.png)
The data story provides an analysis of member numbers for the four regions. The Northeast, Southwest, and Southeast regions have the most members.
![\[Northeast and Southwest regions have 8 members, Southeast has 6 members, and Midwest has 4 members.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/fc6ed0a0-b79c-4397-95ac-a2fc4c87482a.png)
The data story presents an analysis of members by age.
![\[Chart showing that the member base skews toward younger and middle-aged adults.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/8c56f1ec-3a2e-47a6-bbc4-3631782aa333.png)
The data story focuses on immunization efforts in the Midwest.
![\[Recommendation for personal outreach campaign and regional challenges.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/84a647e8-c7d5-4637-94f0-03a611f899b3.png)
![\[Continuation of data story analysis, with anticipated outcomes and conclusion.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/18e72bcb-1b9a-406a-8220-83aca7743ad2/images/fc9094fc-2a20-485d-b238-e5e4ec70f1d3.png)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/18e72bcb-1b9a-406a-8220-83aca7743ad2/attachments/attachment.zip)
# Generate data insights by using AWS Mainframe Modernization and Amazon Q in Quick Sight
*Shubham Roy, Roshna Razack, and Santosh Kumar Singh, Amazon Web Services*
## Summary
Note: AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
If your organization is hosting business-critical data in a mainframe environment, gaining insights from that data is crucial for driving growth and innovation. By unlocking mainframe data, you can build faster, secure, and scalable business intelligence to accelerate data-driven decision-making, growth, and innovation in the Amazon Web Services (AWS) Cloud.
This pattern presents a solution for generating business insights and creating sharable narratives from mainframe data by using [AWS Mainframe Modernization file transfer](https://docs.aws.amazon.com/m2/latest/userguide/filetransfer.html) with BMC and [Amazon Q in Quick Sight](https://docs.aws.amazon.com/quicksight/latest/user/quicksight-gen-bi.html). Mainframe datasets are transferred to [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) by using AWS Mainframe Modernization file transfer with BMC. An AWS Lambda function formats and prepares mainframe data file for loading into Quick Sight.
After the data is available in Quick Sight, you can use natural language prompts with [Amazon Q](https://docs.aws.amazon.com/quicksight/latest/user/quicksight-gen-bi.html) in Quick Sight to create summaries of the data, ask questions, and generate data stories. You don't have to write SQL queries or learn a business intelligence (BI) tool.
**Business context**
This pattern presents a solution for mainframe data analytics and data insights use cases. Using the pattern, you build a visual dashboard for your company's data. To demonstrate the solution, this pattern uses a health care company that provides medical, dental, and vision plans to its members in the US. In this example, member demographics and plan information are stored in the mainframe datasets. The visual dashboard shows the following:
+ Member distribution by region
+ Member distribution by gender
+ Member distribution by age
+ Member distribution by plan type
+ Members who have not completed preventive immunization
After you create the dashboard, you generate a data story that explains the insights from the previous analysis. The data story provides recommendations for increasing the number of members who have completed preventive immunizations.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Mainframe datasets with business data
+ Access to install a file transfer agent on the mainframe
**Limitations**
+ Your mainframe data file should be in one of the file formats supported by Quick Sight. For a list supported file formats, see [Supported data sources](https://docs.aws.amazon.com/quicksuite/latest/userguide/supported-data-sources.html).
+ This pattern uses a Lambda function to convert the mainframe file into a format supported by Quick Sight.
## Architecture
The following diagram shows an architecture for generating business insights from mainframe data by using AWS Mainframe Modernization file transfer with BMC and Amazon Q in Quick Sight.
![\[Architecture diagram description follows the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/53572abb-06c6-4dd7-add4-8fad7e9bfa68/images/6fe0f1d9-961c-4089-a746-e5b8d5fd6c1e.png)
The diagram shows the following workflow:
1. A mainframe dataset containing business data is transferred to Amazon S3 by using AWS Mainframe Modernization file transfer with BMC.
1. The Lambda function converts the file that's in the file-transfer destination S3 bucket to comma-separated values (CSV) format.
1. The Lambda function sends the converted file to the source dataset S3 bucket.
1. The data in the file is ingested by Quick Sight.
1. Users access the data in Quick Sight. You can use Amazon Q in Quick Sight to interact with the data by using natural language prompts.
## Tools
**AWS services**
+ [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
+ [AWS Mainframe Modernization file transfer with BMC](https://docs.aws.amazon.com/m2/latest/userguide/filetransfer.html) converts and transfers mainframe datasets to Amazon S3 for mainframe modernization, migration, and augmentation use cases.
+ [Amazon Quick Sight](https://docs.aws.amazon.com/quicksight/latest/user/welcome.html) is a cloud-scale BI service that helps you visualize, analyze, and report your data in a single dashboard. This pattern uses the generative BI capabilities of [Amazon Q in Quick Sight](https://docs.aws.amazon.com/quicksight/latest/user/working-with-quicksight-q.html).
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
## Best practices
+ When you create the AWS Identity and Access Management (IAM) roles for AWS Mainframe Modernization file transfer with BMC and the Lambda function, follow the principle of [least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
+ Ensure that your source dataset has [supported data types](https://docs.aws.amazon.com/quicksight/latest/user/supported-data-types-and-values.html) for Quick Sight. If your source dataset contains unsupported data types, convert them to supported data types. For information about unsupported mainframe data types and how to convert them to data types supported by Amazon Q in Quick Sight, see the [Related resources](#generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight-resources) section.
## Epics
### Set up AWS Mainframe Modernization file transfer with BMC
| Task | Description | Skills required |
| --- | --- | --- |
| Install the file transfer agent. | To install AWS Mainframe Modernization file transfer agent, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/m2/latest/userguide/m2-agent-installation.html). | Mainframe system administrator |
| Create an S3 bucket for mainframe file transfer. | [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to store the output file from AWS Mainframe Modernization file transfer with BMC. In the architecture diagram, this is the file-transfer destination bucket. | Migration engineer |
| Create the data transfer endpoint. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html) | AWS Mainframe Modernization specialist |
### Convert the mainframe file name extension for Quick Sight integration
| Task | Description | Skills required |
| --- | --- | --- |
| Create an S3 bucket. | [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) for the Lambda function to copy the converted mainframe file from the source to the final destination bucket. | Migration engineer |
| Create a Lambda function. | To create a Lambda function that changes the file extension and copies the mainframe file to the destination bucket, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html) | Migration engineer |
| Create an Amazon S3 trigger to invoke the Lambda function. | To configure a trigger that invokes the Lambda function, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html)For more information, see [Tutorial: Using an Amazon S3 trigger to invoke a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html). | Migration lead |
| Provide IAM permissions for the Lambda function. | IAM permissions are required for the Lambda function to access the file-transfer destination and source dataset S3 buckets. Update the policy associated with the Lambda function execution role by allowing `s3:GetObject` and `s3:DeleteObject`** **permissions** **for the file-transfer destination S3 bucket and `s3:PutObject` access for the source dataset S3 bucket.For more information, see the [Create a permissions policy](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html#with-s3-example-create-policy)** **section in *Tutorial: Using an Amazon S3 trigger to invoke a Lambda function*. | Migration lead |
### Define a mainframe data transfer task
| Task | Description | Skills required |
| --- | --- | --- |
| Create a transfer task to copy the mainframe file to the S3 bucket. | To create a mainframe file transfer task, follow the instructions in the [AWS Mainframe Modernization documentation.](https://docs.aws.amazon.com/m2/latest/userguide/filetransfer-transfer-tasks.html)Specify **Source code page** encoding as **IBM1047 **and **Target code page** encoding as** UTF-8**. | Migration engineer |
| Verify the transfer task. | To verify that the data transfer is successful, follow the instructions in the [AWS Mainframe Modernization documentation](https://docs.aws.amazon.com/m2/latest/userguide/filetransfer-transfer-tasks.html#filetransfer-ts-view-console). Confirm that the mainframe file is in the file-transfer destination S3 bucket. | Migration lead |
| Verify the Lambda copy function. | Verify that the Lambda function is initiated and that the file is copied with a .csv extension to the source dataset S3 bucket.The .csv file created by the Lambda function is the input data file for Quick Sight. For example data, see the `Sample-data-member-healthcare-APG` file in the [Attachments](#attachments-53572abb-06c6-4dd7-add4-8fad7e9bfa68) section. | Migration lead |
### Connect Quick Sight to the mainframe data
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Quick Sight. | To set up Quick Sight, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/setting-up.html). | Migration lead |
| Create a dataset for Quick Sight. | To create a dataset for Quick Sight, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/create-a-data-set-s3.html). The input data file is the converted mainframe file that was created when you defined the mainframe data transfer task. | Migration lead |
### Get business insights from the mainframe data by using Amazon Q in Quick Sight
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Amazon Q in Quick Sight. | This capability requires Enterprise Edition. To set up Amazon Q in Quick Sight, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html) | Migration lead |
| Analyze mainframe data and build a visual dashboard. | To analyze and visualize your data in Quick Sight, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html)When you're finished, you can publish your dashboard to share with others in your organization. For examples, see *Mainframe visual dashboard* in the [Additional information](#generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight-additional) section. | Migration engineer |
### Create a data story with Amazon Q in Quick Sight from the mainframe data
| Task | Description | Skills required |
| --- | --- | --- |
| Create a data story. | Create a data story to explain insights from the previous analysis, and generate a recommendation to increase preventive immunization for members:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html) | Migration engineer |
| View the generated data story. | To view the generated data story, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/working-with-stories-view.html). | Migration lead |
| Edit a generated data story. | To change the formatting, layout, or visuals in a data story, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/working-with-stories-edit.html). | Migration lead |
| Share a data story. | To share a data story, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/quicksight/latest/user/working-with-stories-share.html). | Migration engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Unable to discover the mainframe files or datasets entered in **Data sets search criteria **for **Create transfer task** in AWS Mainframe Modernization file transfer with BMC. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-data-insights-by-using-aws-mainframe-modernization-and-amazon-q-in-quicksight.html) |
## Related resources
To convert mainframe data types such as [PACKED-DECIMAL (COMP-3)](https://www.ibm.com/docs/en/cobol-zos/6.3?topic=v6-packed-decimal-comp-3) or [BINARY (COMP or COMP-4)](https://www.ibm.com/docs/en/cobol-zos/6.3?topic=v6-binary-comp-comp-4) to a [data type](https://docs.aws.amazon.com/quicksight/latest/user/supported-data-types-and-values.html) supported by Quick Sight, see the following patterns:
+ [Convert and unpack EBCDIC data to ASCII on AWS by using Python](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.html)
+ [Convert mainframe files from EBCDIC format to character-delimited ASCII format in Amazon S3 using AWS Lambda](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-mainframe-files-from-ebcdic-format-to-character-delimited-ascii-format-in-amazon-s3-using-aws-lambda.html)
## Additional information
**S3CopyLambda.py**
The following Python code was generated by using a prompt with Amazon Q in an IDE:
```
#Create a lambda function triggered by S3. display the S3 bucket name and key
import boto3
s3 = boto3.client('s3')
def lambda_handler(event, context):
print(event)
bucket = event['Records'][0]['s3']['bucket']['name']
key = event['Records'][0]['s3']['object']['key']
print(bucket, key)
#If key starts with object_created, skip copy, print "copy skipped". Return lambda with key value.
if key.startswith('object_created'):
print("copy skipped")
return {
'statusCode': 200,
'body': key
}
# Copy the file from the source bucket to the destination bucket. Destination_bucket_name = 'm2-filetransfer-final-opt-bkt'. Destination_file_key = 'healthdata.csv'
copy_source = {'Bucket': bucket, 'Key': key}
s3.copy_object(Bucket='m2-filetransfer-final-opt-bkt', Key='healthdata.csv', CopySource=copy_source)
print("file copied")
#Delete the file from the source bucket.
s3.delete_object(Bucket=bucket, Key=key)
return {
'statusCode': 200,
'body': 'Copy Successful'
}
```
**Mainframe visual dashboard**
The following data visual was created by Amazon Q in Quick Sight for the analysis question `show member distribution by region`*.*
![\[Chart showing numbers of members for southwest, midwest, northeast, and southeast.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/53572abb-06c6-4dd7-add4-8fad7e9bfa68/images/e5c1d049-407d-42ff-bc51-28f9d2b24d4f.png)
The following data visual was created by Amazon Q in Quick Sight for the question `show member distribution by Region who have not completed preventive immunization, in pie chart`.
![\[Pie chart showing preventive immunization incompletion by region: Southeast 40%, Southwest 33%, Midwest 27%.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/53572abb-06c6-4dd7-add4-8fad7e9bfa68/images/47efa1c1-54c9-47cc-b668-416090021d34.png)
**Data story output**
The following screenshots show sections of the data story created by Amazon Q in Quick Sight for the prompt `Build a data story about Region with most numbers of members. Also show the member distribution by medical plan, vision plan, dental plan. Recommend how to motivate members to complete immunization. Include 4 points of supporting data.`
In the introduction, the data story recommends choosing the region with the most members to gain the greatest impact from immunization efforts.
![\[Introduction page for data story focusing on immunization completion rates.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/53572abb-06c6-4dd7-add4-8fad7e9bfa68/images/4612fcc7-51fd-48a5-bc58-b6b0aa9b0ef3.png)
The data story provides an analysis of member numbers for the top three regions, and names the Southwest as the leading region for focusing on immunization efforts.
![\[Pie chart showing member distribution by region, with Southwest and Northeast leading at 31% each.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/53572abb-06c6-4dd7-add4-8fad7e9bfa68/images/30d3b56b-3b92-4748-9cef-a73ff9339fee.png)
**Note**
The Southwest and Northeast regions each have eight members. However, the Southwest has more members that aren't fully vaccinated, so it has more potential to benefit from initiatives to increase immunization rates.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/53572abb-06c6-4dd7-add4-8fad7e9bfa68/attachments/attachment.zip)
# Implement Microsoft Entra ID-based authentication in an AWS Blu Age modernized mainframe application
*Vishal Jaswani and Rimpy Tewani, Amazon Web Services*
## Summary
**Note**
AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
Mainframe applications that are modernized by using refactoring patterns, such as those by [AWS Mainframe Modernization Refactor with AWS Blu Age](https://docs.aws.amazon.com/m2/latest/userguide/refactoring-m2.html), require careful integration of authentication mechanisms into the new application architecture. This integration is typically addressed as a post-modernization activity. The task can be complex and often involves the migration or externalization of existing authentication systems to align with modern security standards and cloud-native practices. Developers need to consider how to implement authentication effectively while they work within the constraints of the modernized application's runtime environment and libraries. After modernization, AWS provides ways to make it easier for you to integrate your AWS Blu Age modern code with identity and access management systems such as [Amazon Cognito](https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html) and [Microsoft Entra ID](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id) (formerly known as Azure AD).
This pattern explains how to implement an authentication mechanism in your modernized application when the authentication provider is Microsoft Entra ID, without spending time on research and trials. The pattern provides:
+ Field-tested and relevant Angular libraries from the Microsoft Authentication Library (MSAL) and other Microsoft Entra ID documentation that are essential to the authentication implementation.
+ Configurations required on the AWS Blu Age Runtime to enable Spring Security by using OAuth 2.0.
+ A library that captures authenticated users’ identities and passes them to the AWS Blu Age Runtime.
+ Security measures that we recommend implementing.
+ Troubleshooting tips for commonly encountered problems with the Microsoft Entra ID setup.
**Note**
This pattern uses the AWS Blu Age OAuth extension library, which is provided to customers as part of their [AWS Professional Services](https://aws.amazon.com/professional-services/) engagement. This library isn’t part of the AWS Blu Age Runtime.
## Prerequisites and limitations
**Prerequisites**
+ A modernized application that was produced by AWS Blu Age mainframe modernization refactoring tools. This pattern uses [CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo) as a sample open source mainframe application.
+ The AWS Blu Age OAuth extension library, which is provided by the AWS Blu Age team during your engagement with [AWS Professional Services](https://aws.amazon.com/professional-services/).
+ An active AWS account to deploy and test the modernized application.
+ Familiarity with AWS Blu Age configuration files and Microsoft Entra ID fundamentals.
**Limitations**
+ This pattern covers OAuth 2.0 authentication and basic token-based authorization flows. Advanced authorization scenarios and fine-grained access control mechanisms are not in scope.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
**Product versions**
This pattern was developed by using:
+ AWS Blu Age Runtime version 4.1.0 (the pattern also works with later versions that are backward compatible)
+ MSAL library version 3.0.23
+ Java Development Kit (JDK) version 17
+ Angular version 16.1
## Architecture
**Source technology stack**
In typical mainframe environments, authentication is implemented through user profiles. These profiles identify users to the system, define who can sign in, and specify which functions users can perform on system resources. User profiles are managed by security officers or security administrators.
**Target technology stack**
+ Microsoft Entra ID
+ Modernized Java Spring Boot-based backend
+ AWS Blu Age Runtime
+ Spring Security with OAuth 2.0
+ Angular single-page application (SPA)
**Target architecture**
AWS Blu Age runtime supports OAuth 2.0-based authentication by default, so the pattern uses that standard to protect backend APIs.
The following diagram illustrates the process flow.
**Note**
The diagram includes Amazon Aurora as an example of database modernization although Aurora isn’t included in the steps for this pattern.
![\[Process flow for Entra ID-based authentication for an AWS Blu Age application.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e51f24b8-178f-4974-aae9-23a0cc8540f5/images/0fdcdb22-9e46-4b02-86b2-395cba3e2f81.png)
where:
1. A user tries to authenticate with Microsoft Entra ID.
1. Microsoft Entra ID returns refresh, access, and ID tokens that the application uses in subsequent calls.
1. The MSAL interceptor includes the access token in the `Authorization` header of an HTTPS request to call the AWS Blu Age Runtime.
1. The AWS Blu Age `extension-oauth` library extracts the user information from the header by using an AWS Blu Age Runtime configuration file (`application-main.yml`) and places this information in a `SharedContext` object so that the business logic can consume it.
**Note**
`SharedContext` is a runtime component provided by AWS Blu Age that manages application context and state information across the modernized application. For more information about AWS Blu Age Runtime components and updates, see [AWS Blu Age release notes](https://docs.aws.amazon.com/m2/latest/userguide/ba-release-notes.html) in the AWS Mainframe Modernization documentation. For more information about the `application-main.yml` file, see [Set up configuration for AWS Blu Age Runtime](https://docs.aws.amazon.com/m2/latest/userguide/ba-runtime-config.html) in the AWS Mainframe Modernization documentation.
1. The AWS Blu Age Runtime checks if the token is present.
1. If the token is present, it checks the validity of the token by communicating with Microsoft Entra ID.
1. If the token isn’t present, the AWS Blu Age Runtime returns an error with HTTP status code 403.
1. If the token is valid, the AWS Blue Age Runtime allows the business logic to continue. If the token is invalid, the AWS Blu Age Runtime returns an error with HTTP status code 403.
**OAuth 2.0 workflow**
For a high-level diagram of the OAuth 2.0 workflow, see the [Microsoft Entra documentation](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-auth-code-flow#protocol-details).
## Tools
**AWS services**
[AWS Mainframe Modernization](https://docs.aws.amazon.com/m2/latest/userguide/what-is-m2.html) provides tools and resources to help you plan and implement migration and modernization from mainframes to AWS managed runtime environments. You can use this service’s refactoring features, which are provided by AWS Blu Age, to convert and modernize your legacy mainframe applications.
**Note**
AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
**Code repository**
The CardDemo application has been updated to demonstrate integration with Microsoft Entra ID. You can access the code from the [GitHub repository for this pattern](https://github.com/aws-samples/sample-microsoft-entra-id-based-auth-in-aws-bluage-modernized-mainframe-app).
**Backend configuration**
This pattern requires changes to the `application-main.yml`** **configuration file to enable Spring Security by using OAuth 2.0 on the backend application. The `.yml` file looks like this:
```
gapwalk-application.security: enabled
gapwalk-application:
security:
identity: oauth
issuerUri: ${issuerUrl}
claim:
claims:
-
claimName: upn
claimMapValue: username
spring:
autoconfigure:
exclude:
- org.springframework.boot.autoconfigure.security.oauth2.client.servlet.OAuth2ClientAutoConfiguration
- org.springframework.boot.autoconfigure.security.oauth2.resource.servlet.OAuth2ResourceServerAutoConfiguration
security:
oauth2:
client:
registration:
azure:
client-id: {clientId}
client-secret: ${clientSecret}
provider: azure
authorization-grant-type: authorization_code
redirect-uri: ${redirectUri}
scope: openid
provider:
azure:
authorization-uri: ${gapwalk-application.security.issuerUri}/oauth2/v2.0/authorize
token-uri: ${gapwalk-application.security.issuerUri}/oauth2/v2.0/token
jwk-set-uri: ${gapwalk-application.security.issuerUri}/discovery/v2.0/keys
resourceserver:
jwt:
jwk-set-uri: ${gapwalk-application.security.issuerUri}/discovery/v2.0/keys
```
**AWS Blu Age OAuth extension filter library**
The AWS Blu Age OAuth extension library is provided by the AWS Blu Age team during your engagement with [AWS Professional Services](https://aws.amazon.com/professional-services/).
This library reads the `claim.claims` configuration in the `application-main.yml` fie that’s shown in the previous code block. This configuration is a list. Each item in the list provides two values: `claimName` and `claimMapValue`. `claimName` represents a key name in a JSON Web Token (JWT) sent by the frontend, and `claimMapValue` is the name of the key in `SharedContext`. For example, if you want to capture the user ID on the backend, set `claimName` to the key name in the JWT that holds the `userId` that’s provided by Microsoft Entra ID, and set `claimMapValue` to the key name to fetch the user ID in the backend code.
For example, if you set `UserId` in `claimMapValue`, you can use the following code to extract the user ID:
```
SharedContext.get().getValue("userId", [UserId]);
```
## Best practices
In your implementation of this pattern, take the following important security considerations into account.
**Important**
This pattern provides a foundation for authentication integration. We recommend that you implement security measures in addition to those discussed in this section based on your business requirements before you deploy it to production.
+ **AWS configuration security. **Move sensitive configuration values from `application-main.yml` to AWS Secrets Manager. For example, configure the following properties by using Secrets Manager:
```
security:
oauth2:
client:
registration:
azure:
client-id: {clientId}
client-secret: ${clientSecret}
```
For more information about how you can use Secrets Manager to configure AWS Blu Age parameters, see [AWS Blu Age Runtime secrets](https://docs.aws.amazon.com/m2/latest/userguide/ba-runtime-config-app-secrets.html) in the AWS Mainframe Modernization documentation.
+ **Runtime environment protection.** Configure the modernized application environment with proper AWS security controls:
```
server:
tomcat:
remoteip:
protocol-header: X-Forwarded-Proto
remote-ip-header: X-Forwarded-For
forward-headers-strategy: NATIVE
```
+ **Amazon CloudWatch logging.** Consider adding the file `logback-spring.xml to src/main/resources`:
```
/aws/bluage/application${AWS_REGION}-${ENVIRONMENT}%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n
```
For information about enabling tracing with CloudWatch, see [Enable trace to log correlation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-TraceLogCorrelation.html) in the CloudWatch documentation.
+ **Token configuration and handling.** Configure token lifetimes in Microsoft Entra ID to align with your security requirements. Set access tokens to expire within 1 hour and refresh tokens to expire within 24 hours. In the AWS Blu Age Runtime configuration (`application-main.yml`), make sure that JWT validation is configured properly with the exact issuer URI and audience values from your Entra ID application registration.
When a token expires and is refreshed:
1. The Angular application's error interceptor handles the 401 response by obtaining a new token through MSAL.
1. The new token is sent with the subsequent request.
1. The AWS Blu Age Runtime's OAuth filter validates the new token and automatically updates `SharedContext` with the current user information. This ensures that business logic continues to have access to valid user context through `SharedContext.get().getValue()` calls.
For more information about the AWS Blu Age Runtime components and their updates, see [AWS Blu Age release notes](https://docs.aws.amazon.com/m2/latest/userguide/ba-release-notes.html).
+ **AWS Blu Age Runtime security.** The `oauth2-ext` library provided by AWS Blu Age must be placed in the correct shared directory location (`{app-server-home}/shared/`) with proper file permissions. Verify that the library successfully extracts user information from JWTs by checking the `SharedContext` object population in your logs.
+ **Specific claims configuration.** In `application-main.yml`, define the claims that you need from Microsoft Entra ID explicitly. For example, to capture the user's email and roles, specify:
```
gapwalk-application:
security:
claim:
claims:
- claimName: upn
claimMapValue: username
- claimName: roles
claimMapValue: userRoles
- claimName: email
claimMapValue: userEmail
```
+ **Error handling.** Add error handling to address authentication failures in your Angular application; for example:
```
@Injectable()
export class AuthErrorInterceptor implements HttpInterceptor {
intercept(request: HttpRequest, next: HttpHandler): Observable> {
return next.handle(request).pipe(
catchError((error: HttpErrorResponse) => {
if (error.status === 401) {
// Handle token expiration
this.authService.login();
}
if (error.status === 403) {
// Handle unauthorized access
this.router.navigate(['/unauthorized']);
}
return throwError(() => error);
})
);
}
}
```
+ **Session timeout configuration.** Configure session timeout settings in both the AWS Blu Age Runtime and Microsoft Entra ID. For example, add the following code to your `application-main.yml` file:
```
server:
servlet:
session:
timeout: 3600 # 1 hour in seconds
```
+ **MsalGuard.** You must implement the MsalGuard feature for all protected routes to prevent unauthorized access. For example:
```
const routes: Routes = [
{ path: '', redirectTo: '/transaction-runner', pathMatch: 'full' },
{ path: 'transaction-runner', component: TransactionRunnerComponent, canActivate:guards },
{ path: 'user-info', component: UserInfoComponent, canActivate:guards },
{ path: 'term/:transid/:commarea', component: TermComponent, canActivate:guards },
{ path: 'code', component: TransactionRunnerComponent }
];
```
Routes that don’t have MsalGuard protection will be accessible without authentication, potentially exposing sensitive functionality. Make sure that all routes that require authentication include the guards in their configuration.
## Epics
### Set up a Microsoft Entra ID
| Task | Description | Skills required |
| --- | --- | --- |
| Set up a Microsoft Azure account to create an Entra ID. | For options and instructions, see the [Microsoft Azure website](https://azure.microsoft.com/en-us/free/). | App developer |
| Set up a Microsoft Entra ID in your application. | To learn how to add Microsoft Entra ID B2C (Azure AD B2C) authentication to your Angular SPA, see the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/active-directory-b2c/enable-authentication-angular-spa-app#add-the-authentication-components). Specifically:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application.html) | App developer |
### Clone the repository and deploy your AWS Blu Age code
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the GitHub repository to get the Angular code required for authentication. | Run the following command to clone the [GitHub repository](https://github.com/aws-samples/sample-microsoft-entra-id-based-auth-in-aws-bluage-modernized-mainframe-app) that’s provided with this pattern into your local current working directory:
| App developer |
| Deploy the AWS Blu Age modernized code on a Tomcat server to implement authentication. | To set up the local environment that includes Tomcat and the Angular development server, follow the installation steps provided by the AWS Blu Age team as part of your customer engagement with AWS Professional Services. | App developer |
### Build the authentication solution
| Task | Description | Skills required |
| --- | --- | --- |
| Enable AWS Blu Age Runtime security to protect the AWS Blu Age REST API endpoints. | Configure the `application-main.yml` file that the AWS Blu Age Runtime uses as follows. For an example of this file, see the [Code repository](#implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application-tools) section earlier in this pattern.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application.html) | App developer |
| Incorporate the example code from your local environment into your Blu Age modernized Angular code base. | For information about how to incorporate the example into your AWS Blu Age modernized Angular code base, see the [Code repository](#implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application-tools) section earlier in this pattern. | App developer |
| Place the `oauth2-ext` library in the shared directory. | Place the `oauth2-ext` library in the** **shared directory of the application server so that your** **AWS Blu Age modernized application can use it**. **Run the following commands:
cd oauth2-ext/target cp extension-oauth-filter-.jar /{app-server-home}/shared/
| App developer |
### Deploy the authentication solution
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the frontend application. | Run the following commands to start the frontend application locally:
npm install ng serve --ssl npm start
Adding the `--ssl` flag to the `ng serve` command ensures that the development server uses HTTPS, which is more secure than other protocols and provides a better simulation of a production environment. | App developer |
| Start the backend application. | Start Tomcat server in Eclipse. | App developer |
### Test the application
| Task | Description | Skills required |
| --- | --- | --- |
| Test login functionality. | Access the locally deployed application at `http://localhost:4200` to verify that users are asked to confirm their identity.HTTP is used here for demonstration purposes. In a production or other publicly accessible environment, you must use HTTPS for security. Even for local development, we recommend that you set up HTTPS when possible.The Microsoft login prompt should appear, and users who are configured in Microsoft Entra ID should be allowed to access the application. | App developer |
| Test the authorization header in the request. | The following steps use the [CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo) application as an example. Testing steps for other modern applications will vary.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-entra-id-authentication-in-aws-blu-age-modernized-mainframe-application.html) | App developer |
| Test the logout functionality. | Choose **Quit** to log out, and try to access the application again. It should present a new login prompt. | App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The token issued by Microsoft Entra ID isn’t compatible with Spring Boot OAuth 2.0 security. | For a resolution to the issue, see [Microsoft Entra ID OAuth Flow](https://authguidance.com/azure-ad-troubleshooting/) on the OAuth blog. |
| General token-related questions. | To decode and view the contents of a JWT token, use the [https://jwt.io/](https://jwt.io/) website. |
## Related resources
+ For information about refactoring your application by using AWS Blu Age, see the [AWS Mainframe Modernization documentation](https://docs.aws.amazon.com/m2/latest/userguide/refactoring-m2.html).
+ To understand how OAuth 2.0 works, see the [OAuth 2.0 website](https://oauth.net/2/).
+ For an overview of the Microsoft Authentication Library (MSAL), see the [Microsoft Entra documentation](https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-overview).
+ For information about user profiles on an AS/400 system, see the [IBM i (AS400) tutorial](https://www.go4as400.com/subsystem-jobs-user-profile-in-as400/jobs.aspx?cid=14).
+ For the OAuth 2.0 and OpenID Connect (OIDC) authentication flow in the Microsoft identity platform, see the [Microsoft Entra documentation](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols).
# Integrate Stonebranch Universal Controller with AWS Mainframe Modernization
*Vaidy Sankaran and Pablo Alonso Prieto, Amazon Web Services*
*Robert Lemieux and Huseyin Gomleksizoglu, Stonebranch*
## Summary
Note: AWS Mainframe Modernization Service (Managed Runtime Environment experience) is no longer open to new customers. For capabilities similar to AWS Mainframe Modernization Service (Managed Runtime Environment experience) explore AWS Mainframe Modernization Service (Self-Managed Experience). Existing customers can continue to use the service as normal. For more information, see [AWS Mainframe Modernization availability change](https://docs.aws.amazon.com/m2/latest/userguide/mainframe-modernization-availability-change.html).
This pattern explains how to integrate [Stonebranch Universal Automation Center (UAC) workload orchestration](https://www.stonebranch.com/stonebranch-platform/universal-automation-center) with [Amazon Web Services (AWS) Mainframe Modernization service](https://aws.amazon.com/mainframe-modernization/). AWS Mainframe Modernization service migrates and modernizes mainframe applications to the AWS Cloud. It offers two patterns: [AWS Mainframe Modernization Replatform](https://aws.amazon.com/mainframe-modernization/patterns/replatform/) with Micro Focus Enterprise technology and [AWS Mainframe Modernization Automated Refactor](https://aws.amazon.com/mainframe-modernization/patterns/refactor/?mainframe-blogs.sort-by=item.additionalFields.createdDate&mainframe-blogs.sort-order=desc) with AWS Blu Age.
Stonebranch UAC is a real-time IT automation and orchestration platform. UAC is designed to automate and orchestrate jobs, activities, and workflows across hybrid IT systems, from on-premises to AWS. Enterprise clients using mainframe systems are transitioning to cloud-centric modernized infrastructures and applications. Stonebranch’s tools and professional services facilitate the migration of existing schedulers and automation capabilities to the AWS Cloud.
When you migrate or modernize your mainframe programs to the AWS Cloud using AWS Mainframe Modernization service, you can use this integration to automate batch scheduling, increase agility, improve maintenance, and decrease costs.
This pattern provides instructions for integrating [Stonebranch scheduler](https://www.stonebranch.com/) with mainframe applications migrated to the AWS Mainframe Modernization service Micro Focus Enterprise runtime. This pattern is for solutions architects, developers, consultants, migration specialists, and others working in migrations, modernizations, operations, or DevOps.
**Targeted outcome**
This pattern focuses on providing the following target outcomes:
+ The ability to schedule, automate, and run mainframe batch jobs running in AWS Mainframe Modernization service (Microfocus runtime) from Stonebranch Universal Controller.
+ Monitor the application’s batch processes from the Stonebranch Universal Controller.
+ Start/Restart/Rerun/Stop batch processes automatically or manually from the Stonebranch Universal Controller.
+ Retrieve the results of the AWS Mainframe Modernization batch processes.
+ Capture the [AWS CloudWatch](https://aws.amazon.com/cloudwatch/) logs of the batch jobs in Stonebranch Universal Controller.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A Micro Focus [Bankdemo](https://d1vi4vxke6c2hu.cloudfront.net/demo/bankdemo_runtime.zip) application with job control language (JCL) files, and a batch process deployed in a AWS Mainframe Modernization service (Micro Focus runtime) environment
+ Basic knowledge of how to build and deploy a mainframe application that runs on Micro Focus [Enterprise Server](https://www.microfocus.com/media/data-sheet/enterprise_server_ds.pdf)
+ Basic knowledge of Stonebranch Universal Controller
+ Stonebranch trial license (contact [Stonebranch](https://www.stonebranch.com/))
+ Windows or Linux Amazon Elastic Compute Cloud (Amazon EC2) instances (for example, xlarge) with a minimum of four cores, 8 GB memory, and 2 GB disk space
+ Apache Tomcat version 8.5.x or 9.0.x
+ Oracle Java Runtime Environment (JRE) or OpenJDK version 8 or 11
+ [Amazon Aurora MySQL–Compatible Edition](https://aws.amazon.com/rds/aurora/)
+ [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) bucket for export repository
+ [Amazon Elastic File System (Amaon EFS)](https://aws.amazon.com/efs/) for agent Stonebranch Universal Message Service (OMS) connections for high availability (HA)
+ Stonebranch Universal Controller 7.2 Universal Agent 7.2 Installation Files
+ AWS Mainframe Modernization [task scheduling template](https://github.com/aws-samples/aws-mainframe-modernization-stonebranch-integration/releases) (latest released version of the .zip file)
**Limitations**
+ The product and solution has been tested and compatibility validated only with OpenJDK 8 and 11.
+ The [aws-mainframe-modernization-stonebranch-integration](https://github.com/aws-samples/aws-mainframe-modernization-stonebranch-integration/releases) task scheduling template will work only with AWS Mainframe Modernization service.
+ This task scheduling template will work on only a Unix, Linux, or Windows edition of Stonebranch agents.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
## Architecture
**Target state architecture**
The following diagram shows the example AWS environment that is required for this pilot.
![\[Stonebranch UAC interacting with AWS Mainframe Modernization environment.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/01c6f9fa-87e6-459a-b694-5e03dd7f7952/images/4a7bea37-0a5b-4663-902b-9b051e92f0cb.png)
1. Stonebranch Universal Automation Center (UAC) includes two main components: Universal Controller and Universal Agents. Stonebranch OMS is used as a message bus between the controller and individual agents.
1. Stonebranch UAC Database is used by Universal Controller. The database can be MySQL, Microsoft SQL Server, Oracle, or Aurora MySQL–Compatible.
1. AWS Mainframe Modernization service – Micro Focus runtime environment with the [BankDemo application deployed](https://aws.amazon.com/blogs/aws/modernize-your-mainframe-applications-deploy-them-in-the-cloud/). The BankDemo application files will be stored in an S3 bucket. This bucket also contains the mainframe JCL files.
1. Stonebranch UAC can run the following functions for the batch run:
1. Start a batch job using the JCL file name that exists in the S3 bucket linked to the AWS mainframe modernization service.
1. Get the status of the batch job run.
1. Wait until the batch job run is completed.
1. Fetch logs of the batch job run.
1. Rerun the failed batch jobs.
1. Cancel the batch job while the job is running.
1. Stonebranch UAC can run the following functions for the application:
1. Start Application
1. Get Status of the Application
1. Wait until the Application is started or stopped
1. Stop Application
1. Fetch Logs of Application operation
**Stonebranch jobs conversion**
The following diagram represents Stonebranch’s job conversion process during the modernization journey. It describes how the job schedules and tasks definitions are converted into a compatible format that can run AWS Mainframe Modernization batch tasks.
![\[Process from the mainframe to conversion to job scheduler on Amazon EC2 with JCL files in Amazon S3.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/01c6f9fa-87e6-459a-b694-5e03dd7f7952/images/4d2ed890-f143-455e-8180-4d967b71c494.png)
1. For the conversion process, the job definitions are exported from the existing mainframe system.
1. JCL files can be uploaded to the S3 bucket for the Mainframe Modernization application so that these JCL files can be deployed by the AWS Mainframe Modernization service.
1. The conversion tool converts the exported job definitions to UAC tasks.
1. After all the task definitions and job schedules are created, these objects will be imported to the Universal Controller. The converted tasks then run the processes in the AWS Mainframe Modernization service instead of running them on the mainframe.
**Stonebranch UAC architecture**
The following architecture diagram represents an active-active-passive model of high availability (HA) Universal Controller. Stonebranch UAC is deployed in multiple Availability Zones to provide high availability and support disaster recovery (DR).
![\[Multi-AZ environment with DR and controllers, Amazon EFS, Aurora, and an S3 bucket for backups.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/01c6f9fa-87e6-459a-b694-5e03dd7f7952/images/3f94b855-c146-4fcb-902c-5d343438a558.png)
*Universal Controller*
Two Linux servers are provisioned as Universal Controllers. Both connect to the same database endpoint. Each server houses a Universal Controller application and OMS. The most recent version of Universal Controller is used at the time it is provisioned.
The Universal Controllers are deployed in the Tomcat webapp as the document ROOT and are served on port 80. This deployment eases the configuration of the frontend load balancer.
HTTP over TLS or HTTPS is enabled using the Stonebranch wildcard certificate (for example, `https://customer.stonebranch.cloud`). This secures communication between the browser and the application.
*OMS*
A Universal Agent and OMS (Opswise Message Service) reside on each Universal Controller server. All deployed Universal Agents from the customer end are set up to connect to both OMS services. OMS acts as a common messaging service between the Universal Agents and the Universal Controller.
Amazon EFS mounts a spool directory on each server. OMS uses this shared spool directory to keep the connection and task information from controllers and agents. OMS works in a high-availability mode. If the active OMS goes down, the passive OMS has access to all the data, and it resumes active operations automatically. Universal Agents detect this change and automatically connect to the new active OMS.
*Database*
Amazon Relational Database Service (Amazon RDS) houses the UAC database, with Amazon Aurora MySQL–Compatible as its engine. Amazon RDS is helps in managing and offering scheduled backups at regular intervals. Both Universal Controller instances connect to the same database endpoint.
*Load balancer*
An Application Load Balancer is set-up for each instance. The load balancer directs traffic to the active controller at any given moment. Your instance domain names point to the respective load balancer endpoints.
*URLs*
Each of your instances has a URL, as shown in the following example.
|
|
| Environment | Instance |
| --- |--- |
| **Production** | `customer.stonebranch.cloud` |
| **Development (non-production)** | `customerdev.stonebranch.cloud` |
| **Testing (non-production)** | `customertest.stonebranch.cloud` |
**Note**
Non-production instance names can be set based on your needs.
*High availability*
High availability (HA) is the ability of a system to operate continuously without failure for a designated period of time. Such failures include, but are not limited to, storage, server communication response delays caused by CPU or memory issues, and networking connectivity.
To meet HA requirements:
+ All EC2 instances, databases, and other configurations are mirrored across two separate Availability Zones within the same AWS Region.
+ The controller is provisioned through an Amazon Machine Image (AMI) on two Linux servers in the two Availability Zones. For example, if you are provisioned in the Europe eu-west-1 Region, you have a Universal Controller in Availability Zone eu-west-1a and Availability Zone eu-west-1c.
+ No jobs are allowed to run directly on the application servers and no data is allowed to be stored on these servers.
+ The Application Load Balancer runs health checks on each Universal Controller to identify the active one and direct traffic to it. In the event that one server incurs issues, the load balancer automatically promotes the passive Universal Controller to an active state. The load balancer then identifies the new active Universal Controller instance from the health checks and starts directing traffic. The failover happens within four minutes with no job loss, and the frontend URL remains the same.
+ The Aurora MySQL–Compatible database service stores Universal Controller data. For production environments, a database cluster is built with two database instances in two different Availability Zones within a single AWS Region. Both Universal Controllers use a Java Database Connectivity (JDBC) interface that points to a single database cluster endpoint. In the event that one database instance incurs issues, the database cluster endpoint dynamically points to the healthy instance. No manual intervention is required.
*Backup and purge*
Stonebranch Universal Controller is set to back up and purge old data following the schedule shown in the table.
|
|
| Type | Schedule |
| --- |--- |
| **Activity** | 7 days |
| **Audit** | 90 days |
| **History** | 60 days |
Backup data older than the dates shown is exported to .xml format and stored in the file system. After the backup process is complete, older data is purged from the database and archived in an S3 bucket for up to one year for production instances.
You can adjust this schedule in your Universal Controller interface. However, increasing these time-frames might cause a longer downtime during maintenance.
## Tools
**AWS services**
+ [AWS Mainframe Modernization](https://docs.aws.amazon.com/m2/latest/userguide/what-is-m2.html) is an AWS cloud-native platform that helps you modernize your mainframe applications to AWS managed runtime environments. It provides tools and resources to help you plan and implement migration and modernization.
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) provides block-level storage volumes for use with Amazon EC2 instances.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. This pattern uses Amazon Aurora MySQL–Compatible Edition.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon EC2 instances, containers, and IP addresses in one or more Availability Zones. This pattern uses an Application Load Balancer.
**Stonebranch**
+ [Universal Automation Center (UAC)](https://stonebranchdocs.atlassian.net/wiki/spaces/SD/pages/239239169/Universal+Automation+Center) is a system of enterprise workload automation products. This pattern uses the following UAC components:
+ [Universal Controller](https://www.stonebranch.com/documentation-universal-controller), a Java web application running in a Tomcat web container, is the enterprise job scheduler and workload automation broker solution of Universal Automation Center. The Controller presents a user interface for creating, monitoring, and configuring Controller information; handles the scheduling logic; processes all messages to and from Universal Agents; and synchronizes much of the high availability operation of Universal Automation Center.
+ [Universal Agent](https://www.stonebranch.com/documentation-universal-agent) is a vendor-independent scheduling agent that collaborates with existing job scheduler on all major computing platforms, both legacy and distributed. All schedulers that run on z/Series, i/Series, Unix, Linux, or Windows are supported.
+ [Universal Agent](https://www.stonebranch.com/documentation-universal-agent) is a vendor-independent scheduling agent that collaborates with existing job scheduler on all major computing platforms, both legacy and distributed. All schedulers that run on z/Series, i/Series, Unix, Linux, or Windows are supported.
+ [Stonebranch aws-mainframe-modernization-stonebranch-integration AWS Mainframe Modernization Universal Extension](https://github.com/aws-samples/aws-mainframe-modernization-stonebranch-integration/releases) is the integration template to run, monitor and rerun batch jobs in AWS Mainframe Modernization platform.
**Code**
The code for this pattern is available in the [aws-mainframe-modernization-stonebranch-integration](https://github.com/aws-samples/aws-mainframe-modernization-stonebranch-integration/releases/) GitHub repository.
## Epics
### Install Universal Controller and Universal Agent on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Download the installation files. | Download the installation from Stonebranch servers. To get the installation files, contact with Stonebranch. | Cloud architect |
| Launch the EC2 instance. | You will need about 3 GB of extra space for the Universal Controller and Universal Agent installations. So provide at least 30 GB of disk space for the instance.Add port 8080 to the security group so that it’s accessible. | Cloud architect |
| Check prerequisites. | Before the installation, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Cloud administrator, Linux administrator |
| Install Universal Controller. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Cloud architect, Linux administrator |
| Install Universal Agent. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Cloud administrator, Linux administrator |
| Add OMS to Universal Controller. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
### Import AWS Mainframe Modernization Universal Extension and create a task
| Task | Description | Skills required |
| --- | --- | --- |
| Import Integration Template. | For this step, you need the [AWS Mainframe Modernization Universal Extension](https://github.com/aws-samples/aws-mainframe-modernization-stonebranch-integration/releases). Ensure the latest released version of the .zip file is downloaded.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html)After the Integration Template is imported, you will see **AWS Mainframe Modernization Tasks** under **Available Services**. | Universal Controller administrator |
| Enable resolvable credentials. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
| Launch the task. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
### Test starting a batch job
| Task | Description | Skills required |
| --- | --- | --- |
| Create a task for the batch job. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
| Launch the task. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
### Create a workflow for multiple tasks
| Task | Description | Skills required |
| --- | --- | --- |
| Copy the tasks. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
| Update tasks. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
| Create a workflow. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
| Check the status of the workflow. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Univeral Controller administrator |
### Troubleshoot failed batch jobs and rerun
| Task | Description | Skills required |
| --- | --- | --- |
| Fix the failed task and rerun. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
### Create Start Application and Stop Application tasks
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Start Application action. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | Universal Controller administrator |
### Create a Cancel Batch Execution task
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Cancel Batch action. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/integrate-stonebranch-universal-controller-with-aws-mainframe-modernization.html) | |
## Related resources
+ [Universal Controller](https://stonebranchdocs.atlassian.net/wiki/spaces/UC77/overview)
+ [Universal Agent](https://stonebranchdocs.atlassian.net/wiki/spaces/UA77/overview)
+ [LDAP Settings](https://stonebranchdocs.atlassian.net/wiki/spaces/UC77/pages/794552355/LDAP+Settings)
+ [SAML Single Sign-On](https://stonebranchdocs.atlassian.net/wiki/spaces/UC77/pages/794553130/SAML+Single+Sign-On)
+ [Xpress Conversion Tool](https://www.stonebranch.com/resources/xpress-conversion-windows)
## Additional information
**Icons in the Workflow Editor**
![\[RUNHELLO task at the top, FOOBAR in the middle, and the remaining tasks at the third level.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/01c6f9fa-87e6-459a-b694-5e03dd7f7952/images/837430ee-3159-4fe2-8e17-65168294ef1e.png)
**All tasks connected**
![\[RUNHELLO connects to FOOBAR, which connects to the three remaining tasks.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/01c6f9fa-87e6-459a-b694-5e03dd7f7952/images/fe483348-9a6f-450b-87e6-ceae6b2bdaad.png)
**Workflow status**
![\[FOOBAR task fails and the remaining three tasks are waiting.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/01c6f9fa-87e6-459a-b694-5e03dd7f7952/images/5ea4e239-fbbe-4fa4-9ffa-b7a9443b7975.png)
# Migrate and replicate VSAM files to Amazon RDS or Amazon MSK using Connect from Precisely
*Prachi Khanna and Boopathy GOPALSAMY, Amazon Web Services*
## Summary
This pattern shows you how to migrate and replicate Virtual Storage Access Method (VSAM) files from a mainframe to a target environment in the AWS Cloud by using [Connect](https://www.precisely.com/product/precisely-connect/connect) from Precisely. The target environments covered in this pattern include Amazon Relational Database Service (Amazon RDS) and Amazon Managed Streaming for Apache Kafka (Amazon MSK). Connect uses [change data capture (CDC)](https://www.precisely.com/resource-center/productsheets/change-data-capture-with-connect) to continuously monitor updates to your source VSAM files and then transfer these updates to one or more of your AWS target environments. You can use this pattern to meet your application modernization or data analytics goals. For example, you can use Connect to migrate your VSAM application files to the AWS Cloud with low latency, or migrate your VSAM data to an AWS data warehouse or data lake for analytics that can tolerate synchronization latencies that are higher than required for application modernization.
## Prerequisites and limitations
**Prerequisites**
+ [IBM z/OS V2R1](https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zosv2r1-pdf-download?OpenDocument) or later
+ [CICS Transaction Server for z/OS (CICS TS) V5.1](https://www.ibm.com/support/pages/cics-transaction-server-zos-51-detailed-system-requirements) or later (CICS/VSAM data capture)
+ [IBM MQ 8.0](https://www.ibm.com/support/pages/downloading-ibm-mq-80) or later
+ Compliance with [z/OS security requirements](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Installation/Install-Connect-CDC-SQData-on-zOS/Prerequisites-for-z/OS/Security-authorization-requirements-for-z/OS) (for example, APF authorization for SQData load libraries)
+ VSAM recovery logs turned on
+ (Optional) [CICS VSAM Recovery Version (CICS VR)](https://www.ibm.com/docs/en/cics-vr/5.1?topic=started-introducing-cics-vr) to automatically capture CDC logs
+ An active AWS account
+ An [Amazon Virtual Private Cloud (VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-getting-started.html) with a subnet that’s reachable by your legacy platform
+ A VSAM Connect license from Precisely
**Limitations**
+ Connect doesn’t support automatic target table creation based on source VSAM schemas or copybooks. You must define the target table structure for the first time.
+ For non-streaming targets such as Amazon RDS, you must specify the conversion source to target mapping in the Apply Engine configuration script.
+ Logging, monitoring, and alerting functions are implemented through APIs and require external components (such as Amazon CloudWatch) to be fully operational.
**Product versions**
+ SQData 40134 for z/OS
+ SQData 4.0.43 for the Amazon Linux Amazon Machine Image (AMI) on Amazon Elastic Compute Cloud (Amazon EC2)
## Architecture
**Source technology stack**
+ Job Control Language (JCL)
+ z/OS Unix shell and Interactive System Productivity Facility (ISPF)
+ VSAM utilities (IDCAMS)
** Target technology stack**
+ Amazon EC2
+ Amazon MSK
+ Amazon RDS
+ Amazon VPC
**Target architecture**
*Migrating VSAM files to Amazon RDS*
The following diagram shows how to migrate VSAM files to a relational database, such as Amazon RDS, in real time or near real time by using the CDC agent/publisher in the source environment (on-premises mainframe) and the [Apply Engine](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Apply-engine) in the target environment (AWS Cloud).
![\[Diagram showing VSAM file migration from on-premises mainframe to AWS Cloud using CDC and Apply Engine.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4ee183bd-1c0d-449d-8cdc-eb6e2c41a695/images/47cefbde-e0c8-4c36-ba48-cccc2c443074.png)
The diagram shows the following batch workflow:
1. Connect captures changes to a file by comparing VSAM files from backup files to identify changes and then sends the changes to the logstream.
1. The publisher consumes data from the system logstream.
1. The publisher communicates captured data changes to a target engine through TCP/IP. The Controller Daemon authenticates communication between the source and target environments.
1. The Apply Engine in the target environment receives the changes from the Publisher agent and applies them to a relational or non-relational database.
The diagram shows the following online workflow:
1. Connect captures changes in the online file by using a log replicate and then streams captured changes to a logstream.
1. The publisher consumes data from the system logstream.
1. The publisher communicates captured data changes to the target engine through TCP/IP. The Controller Daemon authenticates communication between the source and target environments.
1. The Apply Engine in the target environment receives the changes from the Publisher agent and then applies them to a relational or non-relational database.
*Migrating VSAM files to Amazon MSK*
The following diagram shows how to stream VSAM data structures from a mainframe to Amazon MSK in high-performance mode and automatically generate JSON or AVRO schema conversions that integrate with Amazon MSK.
![\[Diagram showing data flow from on-premises mainframe to AWS Cloud services via Amazon VPC.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4ee183bd-1c0d-449d-8cdc-eb6e2c41a695/images/13eb27ad-c0d2-489b-91e1-5b2a729fb8dd.png)
The diagram shows the following batch workflow:
1. Connect captures changes to a file by using CICS VR or by comparing VSAM files from backup files to identify changes. Captured changes are sent to the logstream.
1. The publisher consumes data from the system logstream.
1. The publisher communicates captured data changes to the target engine through TCP/IP. The Controller Daemon authenticates communication between the source and target environments.
1. The Replicator Engine that’s operating in parallel processing mode splits the data to a unit of work cache.
1. Worker threads capture the data from the cache.
1. Data is published to Amazon MSK topics from the worker threads.
1. Users apply changes from Amazon MSK to targets such as Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), or Amazon OpenSearch Service by using [connectors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-connectors.html).
The diagram shows the following online workflow:
1. Changes in the online file are captured by using a log replicate. Captured changes are streamed to the logstream.
1. The publisher consumes data from the system logstream.
1. The publisher communicates captured data changes to the target engine through TCP/IP. The Controller Daemon authenticates communication between the source and target environments.
1. The Replicator Engine that’s operating in parallel processing mode splits the data to a unit of work cache.
1. Worker threads capture the data from the cache.
1. Data is published to Amazon MSK topics from the worker threads.
1. Users apply changes from Amazon MSK to targets such as DynamoDB, Amazon S3, or OpenSearch Service by using [connectors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-connectors.html).
## Tools
+ [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) is a fully managed service that helps you build and run applications that use Apache Kafka to process streaming data.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
## Epics
### Prepare the source environment (mainframe)
| Task | Description | Skills required |
| --- | --- | --- |
| Install Connect CDC 4.1. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | IBM Mainframe Developer/Admin |
| Set up the zFS directory. | To set up a zFS directory, follow the instructions from [zFS variable directories](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Installation/Install-Connect-CDC-SQData-on-zOS/Prerequisites-for-z/OS/Security-authorization-requirements-for-z/OS/zFS-variable-directories) in the Precisely documentation.Controller Daemon and Capture/Publisher agent configurations are stored in the z/OS UNIX Systems Services file system (referred to as zFS). The Controller Daemon, Capture, Storage, and Publisher agents require a predefined zFS directory structure for storing a small number of files. | IBM Mainframe Developer/Admin |
| Configure TCP/IP ports. | To configure TCP/IP ports, follow the instructions from [TCP/IP ports](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Installation/Install-Connect-CDC-SQData-on-UNIX/Prerequisites-for-UNIX/Security-authorization-requirements-for-UNIX/TCP/IP-ports) in the Precisely documentation.The Controller Daemon requires TCP/IP ports on source systems. The ports are referenced by the engines on the target systems (where captured change data is processed). | IBM Mainframe Developer/Admin |
| Create a z/OS logstream. | To create a [z/OS logstream](https://www.ibm.com/docs/en/was/8.5.5?topic=SSEQTP_8.5.5/com.ibm.websphere.installation.zseries.doc/ae/cins_logstrm.html), follow the instructions from [Create z/OS system logStreams](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-sources/IMS-z/OS/IMS-TM-EXIT-capture/Prepare-environment/Create-z/OS-system-logStreams?tocId=wy6243SXlIiEczwR8JE8WA) in the Precisely documentation.Connect uses the logstream to capture and stream data between your source environment and target environment during migration.For an example JCL that creates a z/OS LogStream, see [Create z/OS system logStreams](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-sources/IMS-z/OS/IMS-TM-EXIT-capture/Prepare-environment/Create-z/OS-system-logStreams?tocId=wy6243SXlIiEczwR8JE8WA) in the Precisely documentation. | IBM Mainframe Developer |
| Identify and authorize IDs for zFS users and started tasks. | Use RACF to grant access to the OMVS zFS file system. For an example JCL, see [Identify and authorize zFS user and started task IDs](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-sources/IMS-z/OS/IMS-log-reader-capture/Prepare-environment/Identify-and-authorize-zFS-user-and-started-task-IDs?tocId=MrBXpFu~N0iAy~8VTrH0tQ) in the Precisely documentation. | IBM Mainframe Developer/Admin |
| Generate z/OS public/private keys and the authorized key file. | Run the JCL to generate the key pair. For an example, see *Key pair example* in the *Additional information* section of this pattern.For instructions, see [Generate z/OS public and private keys and authorized key file](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-sources/Db2-z/OS/Prepare-the-environment/Generate-z/OS-public-and-private-keys-and-authorized-key-file?tocId=fceE77dWT8smZsSaE~FeMQ) in the Precisely documentation. | IBM Mainframe Developer/Admin |
| Activate the CICS VSAM Log Replicate and attach it to the logstream. | Run the following JCL script:
| IBM Mainframe Developer/Admin |
| Activate the VSAM File Recovery Log through an FCT. | Modify the File Control Table (FCT) to reflect the following parameter changes:
Configure FCT Parms CEDA ALT FILE(name) GROUP(groupname) DSNAME(data set name) RECOVERY(NONE|BACKOUTONLY|ALL) FWDRECOVLOG(NO|1–99) BACKUPTYPE(STATIC|DYNAMIC) RECOVERY PARAMETERS RECOVery : None | Backoutonly | All Fwdrecovlog : No | 1-99 BAckuptype : Static | Dynamic
| IBM Mainframe Developer/Admin |
| Set up CDCzLog for the Publisher agent. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | IBM Mainframe Developer/Admin |
| Activate the Controller Daemon. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | IBM Mainframe Developer/Admin |
| Activate the publisher. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | IBM Mainframe Developer/Admin |
| Activate the logstream. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | IBM Mainframe Developer/Admin |
### Prepare the target environment (AWS)
| Task | Description | Skills required |
| --- | --- | --- |
| Install Precisely on an EC2 instance. | To install Connect from Precisely on the Amazon Linux AMI for Amazon EC2, follow the instructions from [Install Connect CDC (SQData) on UNIX](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Installation/Install-Connect-CDC-SQData-on-UNIX) in the Precisely documentation. | General AWS |
| Open TCP/IP ports. | To modify the security group to include the Controller Daemon ports for inbound and outbound access, follow the instructions from [TCP/IP](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-sources/Change-data-capture/Transient-storage-and-publishing/TCP/IP) in the Precisely documentation. | General AWS |
| Create file directories. | To create file directories, follow the instructions from [Prepare target apply environment](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-targets/Kafka/Prepare-target-apply-environment) in the Precisely documentation. | General AWS |
| Create the Apply Engine configuration file. | Create the Apply Engine configuration file in the working directory of the Apply Engine. The following example configuration file shows Apache Kafka as the target:
For more information, see [Security](https://kafka.apache.org/documentation/#security) in the Apache Kafka documentation. | General AWS |
| Create scripts for Apply Engine processing. | Create the scripts for the Apply Engine to process source data and replicate source data to the target. For more information, see [Create an apply engine script](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Apply-engine/Apply-engine-script-development/Create-an-apply-engine-script) in the Precisely documentation. | General AWS |
| Run the scripts. | Use the `SQDPARSE` and `SQDENG` commands to run the script. For more information, see [Parse a script for zOS](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Apply-engine/Apply-engine-script-development/Parse-a-script/Parse-a-script-for-zOS) in the Precisely documentation. | General AWS |
### Validate the environment
| Task | Description | Skills required |
| --- | --- | --- |
| Validate the list of VSAM files and target tables for CDC processing. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | General AWS, Mainframe |
| Verify that the Connect CDC SQData product is linked. | Run a testing job and verify that the return code from this job is 0 (Successful).Connect CDC SQData Apply Engine status messages should show active connection messages. | General AWS, Mainframe |
### Run and validate test cases (Batch)
| Task | Description | Skills required |
| --- | --- | --- |
| Run the batch job in the mainframe. | Run the batch application job using a modified JCL. Include steps in the modified JCL that do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | General AWS, Mainframe |
| Check the logstream. | Check the logstream to confirm that you can see the change data for the completed mainframe batch job. | General AWS, Mainframe |
| Validate the counts for the source delta changes and target table. | To confirm the records are tallied, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | General AWS, Mainframe |
### Run and validate test cases (Online)
| Task | Description | Skills required |
| --- | --- | --- |
| Run the online transaction in a CICS region. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-and-replicate-vsam-files-to-amazon-rds-or-amazon-msk-using-connect-from-precisely.html) | IBM Mainframe Developer |
| Check the logstream. | Confirm that the logstream is populated with specific record level changes. | AWS Mainframe Developer |
| Validate the count in the target database. | Monitor the Apply Engine for record level counts. | Precisely, Linux |
| Validate the record counts and data records in the target database. | Query the target database to validate the record counts and data records. | General AWS |
## Related resources
+ [VSAM z/OS](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Setup-and-configure-sources/VSAM-z/OS) (Precisely documentation)
+ [Apply engine](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Apply-engine) (Precisely documentation)
+ [Replicator engine](https://help.precisely.com/r/Connect-CDC-SQData/4.1.43/en-US/Connect-CDC-SQData-Help/Source-and-Target-Configuration/Replicator-engine) (Precisely documentation)
+ [The log stream](https://www.ibm.com/docs/en/zos/2.3.0?topic=logger-log-stream) (IBM documentation)
## Additional information
**Configuration file example**
This is an example configuration file for a logstream where the source environment is a mainframe and the target environment is Amazon MSK:
```
-- JOBNAME -- PASS THE SUBSCRIBER NAME
-- REPORT progress report will be produced after "n" (number) of Source records processed.
JOBNAME VSMTOKFK;
--REPORT EVERY 100;
-- Change Op has been ‘I’ for insert, ‘D’ for delete , and ‘R’ for Replace. For RDS it is 'U' for update
-- Character Encoding on z/OS is Code Page 1047, on Linux and UNIX it is Code Page 819 and on Windows, Code Page 1252
OPTIONS
CDCOP('I', 'U', 'D'),
PSEUDO NULL = NO,
USE AVRO COMPATIBLE NAMES,
APPLICATION ENCODING SCHEME = 1208;
-- SOURCE DESCRIPTIONS
BEGIN GROUP VSAM_SRC;
DESCRIPTION COBOL ../copybk/ACCOUNT AS account_file;
END GROUP;
-- TARGET DESCRIPTIONS
BEGIN GROUP VSAM_TGT;
DESCRIPTION COBOL ../copybk/ACCOUNT AS account_file;
END GROUP;
-- SOURCE DATASTORE (IP & Publisher name)
DATASTORE cdc://10.81.148.4:2626/vsmcdct/VSMTOKFK
OF VSAMCDC
AS CDCIN
DESCRIBED BY GROUP VSAM_SRC ACCEPT ALL;
-- TARGET DATASTORE(s) - Kafka and topic name
DATASTORE 'kafka:///MSKTutorialTopic/key'
OF JSON
AS CDCOUT
DESCRIBED BY GROUP VSAM_TGT FOR INSERT;
-- MAIN SECTION
PROCESS INTO
CDCOUT
SELECT
{
SETURL(CDCOUT, 'kafka:///MSKTutorialTopic/key')
REMAP(CDCIN, account_file, GET_RAW_RECORD(CDCIN, AFTER), GET_RAW_RECORD(CDCIN, BEFORE))
REPLICATE(CDCOUT, account_file)
}
FROM CDCIN;
```
**Key pair example**
This an example of how to run the JCL to generate the key pair:
```
//SQDUTIL EXEC PGM=SQDUTIL //SQDPUBL DD DSN=&USER..NACL.PUBLIC, // DCB=(RECFM=FB,LRECL=80,BLKSIZE=21200), // DISP=(,CATLG,DELETE),UNIT=SYSDA, // SPACE=(TRK,(1,1)) //SQDPKEY DD DSN=&USER..NACL.PRIVATE, // DCB=(RECFM=FB,LRECL=80,BLKSIZE=21200), // DISP=(,CATLG,DELETE),UNIT=SYSDA, // SPACE=(TRK,(1,1)) //SQDPARMS DD keygen //SYSPRINT DD SYSOUT= //SYSOUT DD SYSOUT=* //SQDLOG DD SYSOUT=* //*SQDLOG8 DD DUMMY
```
# Modernize the CardDemo mainframe application by using AWS Transform
*Santosh Kumar Singh and Cheryl du Preez, Amazon Web Services*
## Summary
[AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/what-is-service.html) is designed to accelerate the modernization of mainframe applications. It uses generative AI to streamline the mainframe modernization process. It automates complex tasks, such as: legacy code analysis, mainframe documentation, business rules extraction, monolithic applications decomposition into business domain, and code refactoring. It accelerates modernization projects by automating complex tasks, such as application analysis and migration sequence planning. When decomposing monolithic applications, AWS Transform intelligently sequences the mainframe application transformation, which helps you to transform business functions in parallel. AWS Transform can accelerate decision making and enhance operational agility and migration efficiency.
This pattern offers step-by-step instructions to help you test the mainframe modernization capabilities of AWS Transform by using [CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo), which is a sample open source mainframe application.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ AWS IAM Identity Center, [enabled](https://docs.aws.amazon.com/singlesignon/latest/userguide/enable-identity-center.html)
+ [Permissions](https://docs.aws.amazon.com/transform/latest/userguide/security_iam_id-based-policy-examples.html#id-based-policy-examples-admin-enable-transform) that allow administrators to enable AWS Transform
+ [Permissions](https://docs.aws.amazon.com/transform/latest/userguide/security_iam_id-based-policy-examples.html#id-based-policy-examples-admin-connector) that allow administrators to accept Amazon Simple Storage Service (Amazon S3) connection requests for the AWS Transform web application
** Limitations**
+ AWS Transform is available only in some AWS Regions. For a complete list of supported Regions, see [Supported Regions for AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/regions.html).
+ AWS Transform supports code analysis, document generation, business rules extraction, decomposition, and refactoring from Common Business-Oriented Language (COBOL) to Java. For more information, see [Capabilities and key features](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe.html#transform-app-mainframe-features) and [Supported file types for transformation of mainframe applications](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe.html#transform-app-mainframe-supported-files).
+ There is a service quota for mainframe transformation capabilities in AWS Transform. For more information, see [Quotas for AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/transform-limits.html).
+ In order to collaborate on a shared workspace, all users must be registered users of the same instance of AWS IAM Identity Center that is associated with your instance of the AWS Transform web application.
+ The Amazon S3 bucket and AWS Transform must be in the same AWS account and Region.
## Architecture
The following diagram shows the architecture that you set up in this pattern.
![\[Using AWS Transform to modernize a mainframe application that is stored in an Amazon S3 bucket.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0e539474-b733-452d-b0fb-6b3f4cbd5075/images/75be6d78-5b43-448c-ad07-bf74b9ae14ad.png)
The diagram shows the following workflow:
1. AWS Transform uses a connector to access the CardDemo mainframe application, which is stored in an Amazon S3 bucket.
1. AWS Transform uses AWS IAM Identity Center to manage user access and authentication. The system implements multiple layers of security controls for authentication, authorization, encryption, and access management to help protect code and artifacts during processing. Users interact with the AWS Transform agent through a chat interface. You can provide instructions to the AI agent for specific tasks in English. For more information, see [Human in the loop (HITL)](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe.html#transform-app-mainframe-hitl) in the AWS Transform documentation.
1. The AI agent interprets the user's instructions, creates a job plan, divides the job into executable tasks, and actions it autonomously. Users can review and approve the transformation. Transformation tasks include the following:
+ **Code analysis** – AWS Transform analyzes the code in each file for details such as file name, file type, lines of code, and their paths. The agent analyzes the source code, runs classifications, creates dependency mappings, and identifies any missing artifacts. It also identifies duplicate components.
+ **Document generation** – AWS Transform generates documentation for the mainframe application. By analyzing the code, it can automatically create detailed documentation of the application programs, including descriptions of the business logic, flows, integrations, and dependencies present in your legacy systems.
+ **Business logic extraction**– AWS Transform analyzes COBOL programs to document their core business logic, to help you understand the fundamental business logic.
+ **Code decomposition** – AWS Transform decomposes the code into domains that account for dependencies between programs and components. Grouping related files and programs within the same domain improves organization and helps preserve the application's logical structure when breaking it down into smaller components.
+ **Migration wave planning** – Based on the domains you created during the decomposition phase, AWS Transform generates a migration wave plan with recommended modernization order.
+ **Code refactoring** – AWS Transform refactors the code in all or selected domain files into Java code. The goal of this step is to preserve the critical business logic of the application while refactoring it to a modernized, cloud-optimized Java application.
1. AWS Transform stores the refactored code, generated documents, associated artifacts, and runtime libraries in your Amazon S3 bucket. You can do the following:
+ Access the runtime folder in your Amazon S3 bucket.
+ Build and deploy the application by following the [Build and deploy your modernized application post-refactoring](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow-build-deploy.html) in the AWS Transform documentation.
+ Through the chat interface, request and download a sample AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), or Hashicorp Terraform template. These templates can help you deploy the AWS resources that are necessary to support the refactored application.
+ Use [Reforge](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-refactor-code-reforge) to improve the quality of refactored code by using large language models (LLMs). The refactoring engine preserves the functional equivalence of COBOL while transforming it into Java code. Reforge is an optional step that is available after the transformation. This step uses LLMs to restructure the code to closely resemble native Java, which can improve readability and maintainability. Reforge also adds human-readable comments to help you understand the code, and it implements modern coding patterns and best practices.
## Tools
**AWS services**
+ [AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/what-is-service.html) uses agentic AI to help you accelerate the modernization of legacy workloads, such as .NET, mainframe, and VMware workloads.
+ [AWS IAM Identity Center](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) helps you centrally manage single sign-on (SSO) access to your AWS accounts and cloud applications.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
**Code repository**
You can use the open source AWS [CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo) mainframe application as a sample application to get started with mainframe modernization.
## Best practices
+ **Start small** – Begin with small, less complex code (15,000–20,000 lines of code) to get an understanding of how AWS Transform analyzes and transforms mainframe applications.
+ **Combine with human expertise** – Use AWS Transform as an accelerator while applying human expertise for optimal results.
+ **Review and test thoroughly** – Always review the transformed code carefully and run comprehensive tests to validate the functional equivalency after transformation.
+ **Provide feedback** – To provide feedback and suggestions for improvement, use the **Send feedback** button in the AWS Management Console or create a case with [AWS Support](https://support.console.aws.amazon.com/). For more information, see [Creating a support case](https://docs.aws.amazon.com/awssupport/latest/user/case-management.html). Your input is valuable for service enhancements and future development.
## Epics
### Prepare the mainframe application
| Task | Description | Skills required |
| --- | --- | --- |
| Create a bucket. | Create an Amazon S3 bucket in the same AWS account and Region where AWS Transform is enabled. You use this bucket to store the mainframe application code, and AWS Transform uses this bucket to store the generated documents, refactored code and other files associated with the transformation. For instructions, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the Amazon S3 documentation. | General AWS |
| Prepare the sample mainframe application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html) | App developer, DevOps engineer |
### Configure IAM Identity Center and AWS Transform
| Task | Description | Skills required |
| --- | --- | --- |
| Add users to IAM Identity Center. | Add your prospective users to IAM Identity Center. Follow the instructions in [Adding users in IAM Identity Center](https://docs.aws.amazon.com/transform/latest/userguide/transform-user-management.html#transform-add-idc-users) in the AWS Transform documentation. | AWS administrator |
| Enable AWS Transform and add users. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html) | AWS administrator |
| Configure user access to the AWS Transform web application. | Each user must accept the invitation to access the AWS Transform web application. Follow the instructions in [Accepting the invitation](https://docs.aws.amazon.com/transform/latest/userguide/transform-user-onboarding.html#transform-user-invitation) in the AWS Transform documentation. | App developer, App owner |
| Log in to the AWS Transform web application. | Follow the instructions in [Signing in to AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/transform-user-onboarding.html#transform-user-signin). | App developer, App owner |
| Set up a workspace. | Set up a workspace where users can collaborate in the AWS Transform web application. Follow the instructions in [Start your project](https://docs.aws.amazon.com/transform/latest/userguide/transform-environment.html#start-workflow) in the AWS Transform documentation. | AWS administrator |
### Transform the mainframe application
| Task | Description | Skills required |
| --- | --- | --- |
| Create a transformation job. | Create a transformation job to modernize the CardDemo mainframe application. For instructions, see [Create and start a job](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-start-job) in the AWS Transform documentation. When you are asked to set the objectives in the AWS Transform chat interface, choose **Perform mainframe modernization (IBM z/OS to AWS)** and then choose **Analyze code, Generate technical documentation, Business logic, Decompose code, Plan Migration sequence and Transform code to Java**. | App developer, App owner |
| Set up the connector. | Establish a connector to the Amazon S3 bucket that contains the CardDemo mainframe application. This connector allows AWS Transform to access resources in the bucket and perform consecutive transformation functions. For instructions, see [Set up a connector](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-setup-connector) in the AWS Transform documentation. | AWS administrator |
| Perform code analysis. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)For more information, see [Code analysis](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-code-analysis) in the AWS Transform documentation. | App developer, App owner |
| Generate technical documentation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)For more information, see [Generate technical documentation](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-generate-documentation) in the AWS Transform documentation. | App developer, App owner |
| Extract the business logic. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)For more information, see [Extract business logic](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-extract-business-logic) in the AWS Transform documentation. | App developer, App owner |
| Decompose the code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)For more information about decomposition and seeds, see [Decomposition](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-decomposition) in the AWS Transform documentation. | App developer, App owner |
| Plan the migration waves. | Plan the migration waves for the CardDemo application. Follow the instructions in [Migration wave planning](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-wave-planning) in the AWS Transform documentation to review and edit the wave plan. | App developer, App owner |
| Refactor the code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html) | App developer, App owner |
| (Optional) Use Reforge to improve the Java code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)For more information, see [Reforge](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-refactor-code-reforge) in the AWS Transform documentation. | App developer, App owner |
| Streamline the deployment. | AWS Transform can provide infrastructure as code (IaC) templates for CloudFormation, AWS CDK, or Terraform. These templates help you deploy core components, including compute, database, storage, and security resources.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)For more information, see [Deployment capabilities](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-features-deployment) in the AWS Transform documentation. | App developer, App owner |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| You are unable to view the source code or generated document in the AWS Transform web application. | Add a policy to the CORS permission for the Amazon S3 bucket to allow AWS Transform as an origin. For more information, see [S3 bucket CORS permissions](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-setup-connector-s3) in the AWS Transform documentation. |
## Related resources
**AWS documentation**
+ [Transformation of mainframe applications](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html) (AWS Transform documentation)
**Other AWS resources**
+ [Accelerate Your Mainframe Modernization Journey using AI Agents with AWS Transform](https://aws.amazon.com/blogs/migration-and-modernization/accelerate-your-mainframe-modernization-journey-using-ai-agents-with-aws-transform/) (AWS blog post)
+ [AWS Transform FAQs](https://aws.amazon.com/transform/faq/)
+ [AWS IAM Identity Center FAQs](https://aws.amazon.com/iam/identity-center/faqs/)
**Videos and tutorials**
+ [Introduction to Amazon Q Developer: Transform](https://explore.skillbuilder.aws/learn/courses/21893/aws-flash-introduction-to-amazon-q-developer-transform) (AWS Skill Builder)
+ [AWS re:Invent 2024 - Modernize mainframe applications faster using Amazon Q Developer](https://www.youtube.com/watch?v=pSi0XtYfY4o) (YouTube)
+ [AWS re:Invent 2024 - Automating migration and modernization to accelerate transformation](https://www.youtube.com/watch?v=9FjxnEoH5wg) (YouTube)
+ [AWS re:Invent 2024 - Toyota drives innovation & enhances operational efficiency with gen AI](https://www.youtube.com/watch?v=_NXc1MJenw4) (YouTube)
**Note**
AWS Transform was previously known as *Amazon Q Developer transform for mainframe*.
# Modernize and deploy mainframe applications using AWS Transform and Terraform
*Mason Cahill, Polaris Jhandi, Prachi Khanna, Sivasubramanian Ramani, and Santosh Kumar Singh, Amazon Web Services*
## Summary
[AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/what-is-service.html) can accelerate large-scale modernization of .NET, mainframe, and VMware workloads. It deploys specialized AI agents that automate complex tasks like assessments, code analysis, refactoring, decomposition, dependency mapping, validation, and transformation planning. This pattern demonstrates how to use AWS Transform to modernize a mainframe application and then deploy it to AWS infrastructure by using [Hashicorp Terraform](https://developer.hashicorp.com/terraform/intro). These step-by-step instructions help you transform [CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo), which is a sample open source mainframe application, from COBOL to a modern Java application.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ Administrative permissions to create AWS resources and deploy applications
+ Terraform version 1.5.7 or higher, [configured](https://developer.hashicorp.com/terraform/tutorials/aws-get-started?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS)
+ AWS Provider for Terraform, [configured](https://registry.terraform.io/providers/hashicorp/aws/2.36.0/docs#authentication)
+ AWS IAM Identity Center, [enabled](https://docs.aws.amazon.com/singlesignon/latest/userguide/enable-identity-center.html)
+ AWS Transform, [enabled](https://docs.aws.amazon.com/transform/latest/userguide/getting-started.html)
+ A user, [onboarded](https://docs.aws.amazon.com/transform/latest/userguide/transform-user-management.html) to an AWS Transform workspace with a contributor role that can run transformation jobs
**Limitations**
+ AWS Transform is available only in some AWS Regions. For a complete list of supported Regions, see [Supported Regions for AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/regions.html).
+ There is a service quota for mainframe transformation capabilities in AWS Transform. For more information, see [Quotas for AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/transform-limits.html).
+ To collaborate on a shared workspace, all users must be registered users of the same instance of AWS IAM Identity Center that is associated with your instance of the AWS Transform web application.
+ The Amazon Simple Storage Service (Amazon S3) bucket and AWS Transform must be in the same AWS account and Region.
## Architecture
The following diagram shows the end-to-end modernization of the legacy application and deployment to the AWS Cloud. Application and database credentials are stored in AWS Secrets Manager, and Amazon CloudWatch provides monitoring and logging capabilities.
![\[AWS Transform modernizing a mainframe application and deployment through Terraform.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/78bc1e6e-cd3d-4c6d-ae4b-0675a6898fd9/images/332ccf35-f55a-449e-a05d-7e321b3867b7.png)
The diagram shows the following workflow:
1. Through AWS IAM Identity Center, the user authenticates and accesses AWS Transform in the AWS account.
1. The user uploads the COBOL mainframe code to the Amazon S3 bucket and initiates the transformation in AWS Transform.
1. AWS Transform modernizes the COBOL code into cloud-native Java code and stores the modernized code in the Amazon S3 bucket.
1. Terraform creates the AWS infrastructure to deploy the modernized application, including an Application Load Balancer, Amazon Elastic Compute Cloud (Amazon EC2) instance, and Amazon Relational Database Service (Amazon RDS) database. Terraform deploys the modernized code to the Amazon EC2 instance.
1. The VSAM files are uploaded to Amazon EC2 and are migrated from Amazon EC2 to the Amazon RDS database.
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down. In this pattern, SQL Server failover cluster instances are installed on Amazon EC2 instances.
+ [AWS IAM Identity Center](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) helps you centrally manage single sign-on (SSO) access to your AWS accounts and cloud applications.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/what-is-service.html) uses agentic AI to help you accelerate the modernization of legacy workloads, such as .NET, mainframe, and VMware workloads.
**Other tools**
+ [Apache Maven](https://maven.apache.org/) is an open source software project management and build automation tool for Java projects.
+ [Apache Tomcat](https://tomcat.apache.org/) is an open source Servlet container and web server for Java code.
+ [HashiCorp Terraform](https://www.terraform.io/docs) is an infrastructure as code (IaC) tool that helps you use code to provision and manage cloud infrastructure and resources.
+ [Spring Boot](https://spring.io/projects/spring-boot) is an open source framework built on top of the Spring Framework in Java.
**Code repository**
The code for this pattern is available in the GitHub [Mainframe Transformation E2E](https://github.com/aws-samples/sample-mainframe-transformation-e2e) repository. This pattern uses the open source AWS [CardDemo](https://github.com/aws-samples/aws-mainframe-modernization-carddemo) mainframe application as a sample application.
## Best practices
+ Assign full ownership of code and resources targeted for migration.
+ Develop and test a proof of concept before scaling to a full migration.
+ Secure commitment from all stakeholders.
+ Establish clear communication channels.
+ Define and document minimum viable product (MVP) requirements.
+ Set clear success criteria.
## Epics
### Prepare and upload the mainframe application code
| Task | Description | Skills required |
| --- | --- | --- |
| Create a bucket. | Create an Amazon S3 bucket in the same AWS account and Region where AWS Transform is enabled. You use this bucket to store the mainframe application code, data and additional scripts required to build and run the application. AWS Transform uses this bucket to store the refactored code and other files associated with the transformation. For instructions, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the Amazon S3 documentation. | General AWS, AWS administrator |
| Set the CORS permissions for the bucket. | When setting up your bucket for AWS Transform access, you need to configure cross-origin resource sharing (CORS) for the bucket. If this is not set up correctly, you might not be able to use the inline viewing or file comparison functionalities of AWS Transform. For instructions about how to configure CORS for a bucket, see [Using cross-origin resource sharing](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cors.html) in the Amazon S3 bucket. For the policy, see [S3 bucket CORS permissions](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-setup-connector-s3) in the AWS Transform documentation. | General AWS, AWS administrator |
| Prepare the sample mainframe application code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | General AWS, App developer |
### Transform the mainframe application
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the AWS Transform job. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, App owner |
| Set up a connector. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, App owner |
| Transform the code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, App owner |
### Deploy the infrastructure through Terraform
| Task | Description | Skills required |
| --- | --- | --- |
| Update the templates. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html)For production or production-like environments, configure additional security components. For example, enable [AWS WAF protections for your Application Load Balancer](https://aws.amazon.com/about-aws/whats-new/2024/02/aws-application-load-balancer-one-click-waf-integrations/). | General AWS, AWS administrator |
| Deploy the infrastructure. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | Terraform |
### Install and configure Apache Tomcat on the Amazon EC2 instance
| Task | Description | Skills required |
| --- | --- | --- |
| Install the required software. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
| Verify software installation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
### Compile and package the modernized application code
| Task | Description | Skills required |
| --- | --- | --- |
| Download and extract the generated code. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
| Build the modernized application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
### Migrate the database
| Task | Description | Skills required |
| --- | --- | --- |
| Create the database and JICS schemas. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
| Validate database creation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
| Migrate data to the JICS database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
### Install the modernized application
| Task | Description | Skills required |
| --- | --- | --- |
| Install the modernized application on the Amazon EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Cloud architect |
| Restart the Tomcat server. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Cloud architect |
| Migrate the VSAM dataset. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Migration engineer |
| Update the parameters in the Groovy scripts. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer |
### Test the application
| Task | Description | Skills required |
| --- | --- | --- |
| Test the modernized application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Test engineer |
| Verify the batch scripts. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | App developer, Test engineer |
### Clean up
| Task | Description | Skills required |
| --- | --- | --- |
| Prepare to delete the infrastructure. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | General AWS |
| Delete the infrastructure. | These steps will permanently delete your resources. Make sure you have backed up any important data before proceeding.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) | General AWS |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Terraform authentication | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) |
| Tomcat-related errors | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-app-transform-terraform.html) |
| URL name not loading | Make sure that the Application Load Balancer security group has your IP address in the inbound rule as a source. |
| Authentication issue in Tomcat log | Confirm that the database secret password in AWS Secrets Manager and the password in **server.xml** match. |
## Related resources
**AWS Prescriptive Guidance**
+ [Modernize the CardDemo mainframe application by using AWS Transform](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-carddemo-mainframe-app.html)
**AWS service documentation**
+ [AWS Blu Age Blusam Adminstration Console](https://docs.aws.amazon.com/m2/latest/userguide/ba-shared-bac-userguide.html)
+ [Infrastructure setup requirements for AWS Blu Age Runtime (non-managed)](https://docs.aws.amazon.com/m2/latest/userguide/ba-infrastructure-setup.html)
+ [Onboarding AWS Blu Age Runtime](https://docs.aws.amazon.com/m2/latest/userguide/ba-runtime-setup-onboard.html)
+ [Modernization of mainframe applications](https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/transform-app-mainframe.html)
+ [Set up configuration for AWS Blu Age Runtime](https://docs.aws.amazon.com/m2/latest/userguide/ba-runtime-config.html)
**AWS blog posts**
+ [Accelerate Your Mainframe Modernization Journey using AI Agents with AWS Transform](https://aws.amazon.com/blogs/migration-and-modernization/accelerate-your-mainframe-modernization-journey-using-ai-agents-with-aws-transform/)
# Modernize mainframe output management on AWS by using Rocket Enterprise Server and LRS PageCenterX
*Shubham Roy, Amazon Web Services*
*Abraham Rondon, Micro Focus*
*Guy Tucker, Levi, Ray and Shoup Inc*
## Summary
By modernizing your mainframe output management, you can achieve cost savings, mitigate the technical debt of maintaining legacy systems, and improve resiliency and agility through DevOps and Amazon Web Services (AWS) cloud-native technologies. This pattern shows you how to modernize your business-critical mainframe output-management workloads on the AWS Cloud. The pattern uses [Rocket Enterprise Server](https://www.rocketsoftware.com/en-us/products/enterprise-suite/enterprise-server) as a runtime for a modernized mainframe application, with Levi, Ray & Shoup, Inc. (LRS) VPSX/MFI (Micro Focus Interface) as a print server and LRS PageCenterX as an archive server. LRS PageCenterX provides output-management solutions for viewing, indexing, searching, archiving, and securing access to business outputs.
The pattern is based on the [replatform](https://aws.amazon.com/blogs/apn/demystifying-legacy-migration-options-to-the-aws-cloud/) mainframe modernization approach. Mainframe applications are migrated by [AWS Mainframe Modernization](https://docs.aws.amazon.com/m2/latest/userguide/what-is-m2.html) on Amazon Elastic Compute Cloud (Amazon EC2). Mainframe output-management workloads are migrated to Amazon EC2, and a mainframe database, such as IBM Db2 for z/OS, is migrated to Amazon Relational Database Service (Amazon RDS). The LRS Directory Integration Server (LRS/DIS) works with AWS Directory Service for Microsoft Active Directory for output-management workflow authentication and authorization.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ A mainframe output-management workload.
+ Basic knowledge of how to rebuild and deliver a mainframe application that runs on Rocket Enterprise Server. For more information, see the [Rocket Enterprise Server](https://www.rocketsoftware.com/sites/default/files/resource_files/enterprise-server.pdf) data sheet in the Rocket Software documentation.
+ Basic knowledge of LRS cloud printing solutions and concepts. For more information, see *Output Modernization* in the LRS documentation.
+ Rocket Enterprise Server software and license. For more information, contact [Rocket Software](https://www.rocketsoftware.com/products/enterprise-suite/request-contact).
+ LRS VPSX/MFI, LRS PageCenterX, LRS/Queue, and LRS/DIS software and licenses. For more information, [contact LRS](https://www.lrsoutputmanagement.com/about-us/contact-us/). You must provide the hostnames of the EC2 instances where the LRS products will be installed.
|
|
| Note: For more information about configuration considerations for mainframe output-management workloads, see *Considerations* in the [Additional information](#modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx-additional) section of this pattern. |
| --- |
**Product versions**
+ [Rocket Enterprise Server 10.0](https://www.rocketsoftware.com/products/enterprise-suite/enterprise-test-server)
+ [LRS VPSX/MFI](https://www.lrsoutputmanagement.com/products/modernization-products/)
+ [LRS PageCenterX](https://www.lrsoutputmanagement.com/products/content-management/pagecenterx-for-open-systems/) V1R3 or later
## Architecture
**Source technology stack**
+ Operating system – IBM z/OS
+ Programming language – Common business-oriented language (COBOL), job control language (JCL), and Customer Information Control System (CICS)
+ Database – IBM Db2 for z/OS, IBM Information Management System (IMS) database, and Virtual Storage Access Method (VSAM)
+ Security – Resource Access Control Facility (RACF), CA Top Secret for z/OS, and Access Control Facility 2 (ACF2)
+ Print and archive solutions – IBM mainframe z/OS output and printing products (IBM Infoprint Server for z/OS, LRS, and CA Deliver) and archiving solutions (CA Deliver, ASG Mobius, or CA Bundle)
**Source architecture**
The following diagram shows a typical current state architecture for a mainframe output-management workload.
![\[Mainframe output process in seven steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f9ad041d-b9f0-4a9a-aba7-40fdc3088b27/images/d170394a-c9b2-43c0-a3d4-677b5f7c2473.png)
The diagram shows the following workflow:
1. Users perform business transactions on a system of engagement (SoE) that’s built on an IBM CICS application written in COBOL.
1. The SoE invokes the mainframe service, which records the business transaction data in a system-of-records (SoR) database such as IBM Db2 for z/OS.
1. The SoR persists the business data from the SoE.
1. The batch job scheduler initiates a batch job to generate print output.
1. The batch job extracts data from the database. It formats the data according to business requirements, and then it generates business output such as billing statements, ID cards, or loan statements. Finally, the batch job routes the output to output management for format, publish, and storage of the output based on the business requirements.
1. Output management receives output from the batch job. Output management indexes, arranges, and publishes the output to a specified destination in the output-management system, such as LRS PageCenterX solutions (as demonstrated in this pattern) or CA View.
1. Users can view, search, and retrieve the output.
**Target technology stack**
+ Operating system – Windows Server running on Amazon EC2
+ Compute – Amazon EC2
+ Storage – Amazon Elastic Block Store (Amazon EBS) and Amazon FSx for Windows File Server
+ Programming language – COBOL, JCL, and CICS
+ Database – Amazon RDS
+ Security – AWS Managed Microsoft AD
+ Printing and archiving – LRS printing (VPSX) and archiving (PageCenterX) solution on AWS
+ Mainframe runtime environment – Rocket Enterprise Server
**Target architecture**
The following diagram shows an architecture for a mainframe output-management workload that’s deployed in the AWS Cloud.
![\[Target architecture for batch app and output management in seven steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f9ad041d-b9f0-4a9a-aba7-40fdc3088b27/images/3e25ab03-bf3a-4fea-b5eb-38cea9e50138.png)
The diagram shows the following workflow:
1. The batch job scheduler initiates a batch job to create output, such as billing statements, ID cards, or loan statements.
1. The mainframe batch job ([replatformed to Amazon EC2](https://aws.amazon.com/blogs/apn/demystifying-legacy-migration-options-to-the-aws-cloud/)) uses the Rocket Enterprise Server runtime to extract data from the application database, apply business logic to the data, and format the data. It then sends the data to an output destination by using the [Rocket Software printer exit module](https://www.microfocus.com/documentation/enterprise-developer/ed100/ED-Eclipse/HCOMCMJCLOU020.html) (OpenText Micro Focus documentation).
1. The application database (an SoR that runs on Amazon RDS) persists data for print output.
1. The LRS VPSX/MFI printing solution is deployed on Amazon EC2, and its operational data is stored in Amazon EBS. LRS VPSX/MFI uses the TCP/IP-based LRS/Queue transmission agent to collect output data through the Rocket Software JES Print Exit API.
LRS VPSX/MFI does data preprocessing, such as EBCDIC to ASCII translation. It also does more complex tasks, including converting mainframe-exclusive data streams such as IBM Advanced Function Presentation (AFP) and Xerox Line Conditioned Data Stream (LCDS) into more common viewing and printing data streams such as Printer Command Language (PCL) and PDF.
During the maintenance window of LRS PageCenterX, LRS VPSX/MFI persists the output queue and serves as backup for the output queue. LRS VPSX/MFI connects and sends output to LRS PageCenterX by using the LRS/Queue protocol. LRS/Queue performs an exchange of both readiness and completion for the jobs to help ensure that the data transfer occurs.
**Notes:**
For more information on print data passed from Rocket Software Print Exit to LRS/Queue and LRS VPSX/MFI supported mainframe batch mechanisms, see *Print data capture* in the [Additional information](#modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx-additional) section.
LRS VPSX/MFI can perform health checks at the printer-fleet level. For more information, see *Printer-fleet health checks* in the [Additional information](#modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx-additional) section of this pattern.
1. The LRS PageCenterX output-management solution is deployed on Amazon EC2, and its operational data is stored in Amazon FSx for Windows File Server. LRS PageCenterX provides a central report management system of all files imported into LRS PageCenterX along with all users able to access the files. Users can view specific file content or perform searches across multiple files for matching criteria.
The LRS/NetX component is a multi-threaded web application server that provides a common runtime environment for the LRS PageCenterX application and other LRS applications. The LRS/Web Connect component is installed on your web server and provides a connector from the web server to the LRS/NetX web application server.
1. LRS PageCenterX provides storage for file system objects. The operational data of LRS PageCenterX is stored in Amazon FSx for Windows File Server.
1. Output-management authentication and authorization are performed by AWS Managed Microsoft AD with LRS/DIS.
**Note**
The target solution typically doesn’t require application changes to accommodate mainframe formatting languages, such as IBM AFP or Xerox LCDS.
**AWS infrastructure architecture**
The following diagram shows a highly available and secure AWS infrastructure architecture for a mainframe output-management workload.
![\[Multi-AZ AWS infrastructure with a workflow in seven steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/f9ad041d-b9f0-4a9a-aba7-40fdc3088b27/images/8d8aa995-b576-4ecd-8a7c-5f566740a515.png)
The diagram shows the following workflow:
1. The batch scheduler initiates the batch process and is deployed on Amazon EC2 across multiple [Availability Zones](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) for high availability (HA).
**Note**
This pattern doesn’t cover the implementation of the batch scheduler. For more information about implementation, see the software vendor documentation for your scheduler.
1. The mainframe batch job (written in a programming language such as JCL or COBOL) uses core business logic to process and generate print output, such as billing statements, ID cards, and loan statements. The batch job is deployed on Amazon EC2 across two Availability Zones for HA. It uses the Rocket Software Print Exit API to route print output to LRS VPSX/MFI for data preprocessing.
1. The LRS VPSX/MFI print server is deployed on Amazon EC2 across two Availability Zones for HA (active-standby redundant pair). It uses [Amazon EBS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) as an operational data store. The Network Load Balancer performs a health check on the LRS VPSX/MFI EC2 instances. If an active instance is in an unhealthy state, the load balancer routes traffic to hot standby instances in the other Availability Zone. The print requests are persisted in the LRS Job Queue locally in each of the EC2 instances. In the event of a failure, a failed instance must be restarted before the LRS services can resume processing the print request.
**Note**
LRS VPSX/MFI can also perform health checks at the printer-fleet level. For more information, see *Printer-fleet health checks* in the [Additional information](#modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx-additional) section of this pattern.
1. LRS PageCenterX output management is deployed on Amazon EC2 across two Availability Zones for HA (active-standby redundant pair). It uses [Amazon FSx for Windows File Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/what-is.html) as an operational data store. If an active instance is in an unhealthy state, the load balancer performs a health check on the LRS PageCenterX EC2 instances and routes traffic to standby instances in the other Availability Zone.
1. A [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) provides a DNS name to integrate the LRS VPSX/MFI Server with LRS PageCenterX.
**Note**
LRS PageCenterX supports a Layer 4 load balancer.
1. LRS PageCenterX uses Amazon FSx for Windows File Server as an operational data store deployed across two Availability Zones for HA. LRS PageCenterX understands only files that are in the file share, not in an external database.
1. [AWS Managed Microsoft AD](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) is used with LRS/DIS to perform output-management workflow authentication and authorization. For more information, see *Print output authentication and authorization* in the [Additional information](#modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx-additional) section.
## Tools
**AWS services**
+ [AWS Directory Service for Microsoft Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) enables your directory-aware workloads and AWS resources to use Microsoft Active Directory in the AWS Cloud.
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) provides block-level storage volumes for use with Amazon Elastic Compute Cloud (Amazon EC2) instances.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon EC2 instances, containers, and IP addresses in one or more Availability Zones. This pattern uses a Network Load Balancer.
+ [Amazon FSx](https://docs.aws.amazon.com/fsx/?id=docs_gateway) provides file systems that support industry-standard connectivity protocols and offer high availability and replication across AWS Regions. This pattern uses Amazon FSx for Windows File Server.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
**Other tools**
+ [LRS PageCenterX](https://www.lrsoutputmanagement.com/products/content-management/pagecenterx-for-open-systems/) software provides a scalable document and report content management solution that helps users obtain maximum value from information through automated indexing, encryption, and advanced search features.
+ [LRS VPSX/MFI (Micro Focus Interface)](https://www.lrsoutputmanagement.com/products/modernization-products/), codeveloped by LRS and Rocket Software, captures output from a Rocket Software JES spool and reliably delivers it to a specified print destination.
+ LRS/Queue is a transmission agent that’s TCP/IP based. LRS VPSX/MFI uses LRS/Queue to collect or capture print data through the Rocket Software JES Print Exit programming interface.
+ LRS Directory Integration Server (LRS/DIS) is used for authentication and authorization during the print workflow.
+ [Rocket Enterprise Server](https://www.microfocus.com/documentation/enterprise-developer/ed80/ES-WIN/GUID-F7D8FD6E-BDE0-4169-8D8C-96DDFFF6B495.html) is an application deployment environment for mainframe applications. It provides the runtime environment for mainframe applications that are migrated or created by using any version of Rocket Enterprise Developer.
## Epics
### Set up the Rocket runtime and deploy a mainframe batch application
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the runtime and deploy a demo application. | To set up Rocket Enterprise Server on Amazon EC2 and deploy the Rocket Software BankDemo demonstration application, follow the instructions in AWS Mainframe Modernization [user-guide](https://docs.aws.amazon.com/m2/latest/userguide/mf-runtime-setup.html).The BankDemo application is a mainframe batch application that creates and then initiates print output. | Cloud architect |
### Set up an LRS print server on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon EC2 Windows instance. | To launch an Amazon EC2 Windows instance, follow the instructions in [Launch an Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html) in the Amazon EC2 documentation. Use the same hostname that you used for your LRS product license.Your instance must meet the following hardware and software requirements for LRS VPSX/MFI:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html)The preceding hardware and software requirements are intended for a small printer fleet (around 500-1000). To get the full requirements, consult with your LRS and AWS contacts.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Install LRS VPSX/MFI on the EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Install LRS/Queue. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Install LRS/DIS. | The LRS/DIS product often is included in LRS VPSX installation. However, if LRS/DIS wasn't installed along with LRS VPSX, use the following steps to install it:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a target group. | Create a target group by following the instructions in [Create a target group for your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html). When you create the target group, register the LRS VPSX/MFI EC2 instance as the target:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a Network Load Balancer. | To create the Network Load Balancer, follow the instructions in the [Elastic Load Balancing documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html). Your Network Load Balancer routes traffic from Rocket Enterprise Server to the LRS VPSX/MFI EC2 instance.When you create the Network Load Balancer, choose the following values on the **Listeners and Routing** page:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Integrate Rocket Enterprise Server with LRS/Queue and LRS VPSX/MFI
| Task | Description | Skills required |
| --- | --- | --- |
| Configure Rocket Enterprise Server for LRS/Queue integration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Configure Rocket Enterprise Server for LRS VPSX/MFI integration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Set up the print queue and the print users
| Task | Description | Skills required |
| --- | --- | --- |
| Associate the Rocket Software Print Exit module with the Rocket Enterprise Server batch printer Server Execution Process. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a print output queue in LRS VPSX/MFI and integrate it with LRS PageCenterX. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a print user in LRS VPSX/MFI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Set up an LRS PageCenterX server on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon EC2 Windows instance. | Launch an Amazon EC2 Windows instance by following the instructions from [Step 1: Launch an instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2_GetStarted.html#ec2-launch-instance) in the Amazon EC2 documentation. Use the same hostname that you used for your LRS product license.Your instance must meet the following hardware and software requirements for LRS PageCenterX:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html)The preceding hardware and software requirements are intended for a small printer fleet (around 500–1000). To get the full requirements, consult with your LRS and AWS contacts.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Install LRS PageCenterX on the EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Install LRS/DIS. | The LRS/DIS product often is included in LRS VPSX installation. However, if LRS/DIS wasn't installed along with LRS VPSX, use the following steps to install it:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a target group. | Create a target group by following the instructions in [Create a target group for your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html). When you create the target group, register the LRS PageCenterX EC2 instance as the target:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a Network Load Balancer. | To create the Network Load Balancer, follow the instructions in the [Elastic Load Balancing documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html). Your Network Load Balancer routes traffic from LRS VPSX/MFI to the LRS PageCenterX EC2 instance.When you create the Network Load Balancer, choose the following values on the **Listeners and Routing** page:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Set up output-management features in LRS PageCenterX
| Task | Description | Skills required |
| --- | --- | --- |
| Enable the Import function in LRS PageCenterX. | You can use the LRS PageCenterX Import function to recognize the outputs landing on LRS PageCenterX by criteria such as Job name or Form ID. You can then route the outputs to specific folders in LRS PageCenterX.To enable the Import function, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Configure the document retention policy. | LRS PageCenterX uses a document retention policy to decide how long to keep a document in LRS PageCenterX.To configure the document retention policy, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a rule to route the output document to a specific folder in LRS PageCenterX. | In LRS PageCenterX, **Destination **determines the folder path where output will be sent when this destination is invoked by **Report Definition**. For this example, create a folder based on the **Form ID** folder in the report definition, and save the output to that folder.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a report definition. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Set up authentication and authorization for output management
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS Managed Microsoft AD domain with users and groups. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Join the EC2 instances to an AWS Managed Microsoft AD domain. | Join the LRS VPSX/MFI and LRS PageCenterX EC2 instances to your AWS Managed Microsoft AD domain [automatically](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-systems-manager-dx-domain/) (AWS Knowledge Center documentation) or [manually](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/join_windows_instance.html) (AWS Directory Service documentation). | Cloud architect |
| Configure and integrate LRS/DIS with AWS Managed Microsoft AD for the LRS PageCenterX EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Configure an Import group to import output from LRS VPSX to LRS PageCenterX. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Add a security rule to the Import group. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Create a user in LRS PageCenterX to perform output import from LRS VPSX/MFI. | When you create a user in LRS PageCenterX to perform output import, the username should be the same as the **VPSX ID** of the print output queue in LRS VPSX/MFI. In this example, the VPSX ID is **VPS1**.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Add the LRS PageCenterX Import user to the Import only group. | To provide necessary permission for document import from LRS VPSX to LRS PageCenterX, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
| Configure LRS/DIS with AWS Managed Microsoft AD for the LRS VPSX/MFI EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Configure Amazon FSx for Windows File Server as the operational data store for LRS PageCenterX
| Task | Description | Skills required |
| --- | --- | --- |
| Create a file system for LRS PageCenterX. | To use Amazon FSx for Windows File Server as an operational data store for LRS PageCenterX in a Multi-AZ environment, follow the instructions in [Step 1: Create your file system](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/getting-started-step1.html). | Cloud architect |
| Map the file share to the LRS PageCenterX EC2 instance. | To map the file share created in previous step to the LRS PageCenterX EC2 instance, follow the instructions in [Step 2: Map your file share to an EC2 instance running Windows Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/getting-started-step2.html). | Cloud architect |
| Map LRS PageCenterX Control Directory and Master Folder Directory to the Amazon FSx network share drive. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Cloud architect |
### Test an output-management workflow
| Task | Description | Skills required |
| --- | --- | --- |
| Initiate a batch print request from the Rocket Software BankDemo app. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Test engineer |
| Check the print output in LRS PageCenterX. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-output-management-on-aws-by-using-rocket-enterprise-server-and-lrs-pagecenterx.html) | Test engineer |
## Related resources
+ [LRS](https://www.lrsoutputmanagement.com/products/modernization-products)
+ [Advanced Function Presentation data stream](https://www.ibm.com/docs/en/i/7.4?topic=streams-advanced-function-presentation-data-stream) (IBM documentation)
+ [Line Conditioned Data Stream (LCDS)](https://www.compart.com/en/lcds) (Compart documentation)
+ [Empowering Enterprise Mainframe Workloads on AWS with Micro Focus](https://aws.amazon.com/blogs/apn/empowering-enterprise-grade-mainframe-workloads-on-aws-with-micro-focus/) (blog post)
+ [Modernize your mainframe online printing workloads on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) (AWS Prescriptive Guidance)
+ [Modernize your mainframe batch printing workloads on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) (AWS Prescriptive Guidance)
## Additional information
**Considerations**
During your modernization journey, you might consider a wide variety of configurations for mainframe batch and online processes and the output they generate. The mainframe platform has been customized by every customer and vendor that uses it with particular requirements that directly affect print. For example, your current platform might incorporate the IBM AFP data stream or Xerox LCDS into the current workflow. Additionally, [mainframe carriage control characters](https://www.ibm.com/docs/en/cmofz/10.5.0?topic=tips-ansi-machine-carriage-controls) and [channel command words](https://www.ibm.com/docs/en/zos/3.1.0?topic=devices-channel-command-words) can affect the look of the printed page and might need special handling. As part of the modernization planning process, we recommend that you assess and understand the configurations in your specific print environment.
**Print data capture**
Rocket Software Print Exit passes the necessary information for LRS VPSX/MFI to effectively process the spool file. The information consists of fields passed in the relevant control blocks, such as the following:
+ JOBNAME
+ OWNER (USERID)
+ DESTINATION
+ FORM
+ FILENAME
+ WRITER
LRS VPSX/MFI supports the following mainframe batch mechanisms for capturing data from Rocket Enterprise Server:
+ BATCH COBOL print/spool processing using standard z/OS JCL SYSOUT DD/OUTPUT statements.
+ BATCH COBOL print/spool processing using standard z/OS JCL CA-SPOOL SUBSYS DD statements.
+ IMS/COBOL print/spool processing using the CBLTDLI interface. For a full list of supported methods and programming examples, see the LRS documentation that’s included with your product license.
**Printer-fleet health checks**
LRS VPSX/MFI (LRS LoadX) can perform deep dive health checks, including device management and operational optimization. Device management can detect failure in a printer device and route the print request to a healthy printer. For more information about deep-dive health checks for printer fleets, see the LRS documentation that’s included with your product license.
**Print authentication and authorization**
LRS/DIS enables LRS applications to authenticate user IDs and passwords by using Microsoft Active Directory or a Lightweight Directory Access Protocol (LDAP) server. In addition to basic print authorization, LRS/DIS can also apply granular-level print security controls in the following use cases:
+ Manage who can browse the printer job.
+ Manage the browsing level of other user's jobs.
+ Manage operational tasks—for example, command-level security such as hold or release, purge, modify, copy, and reroute. Security can be set up by either the user-ID or the group, similar to an Active Directory security group or an LDAP group.
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/f9ad041d-b9f0-4a9a-aba7-40fdc3088b27/attachments/attachment.zip)
# Modernize mainframe batch printing workloads on AWS by using Rocket Enterprise Server and LRS VPSX/MFI
*Shubham Roy and Kevin Yung, Amazon Web Services*
*Abraham Rondon, Micro Focus*
*Guy Tucker, Levi, Ray and Shoup Inc*
## Summary
This pattern shows you how to modernize your business-critical mainframe batch printing workloads on the Amazon Web Services (AWS) Cloud by using Rocket Enterprise Server as a runtime for a modernized mainframe application and LRS VPSX/MFI (Micro Focus Interface) as a print server. The pattern is based on the [replatform](https://aws.amazon.com/blogs/apn/demystifying-legacy-migration-options-to-the-aws-cloud/) mainframe modernization approach. In this approach, you migrate your mainframe batch jobs to Amazon Elastic Compute Cloud (Amazon EC2) and migrate your mainframe database, such as IBM DB2 for z/OS, to Amazon Relational Database Service (Amazon RDS). The authentication and authorization for the modernized print workflow is performed by AWS Directory Service for Microsoft Active Directory, also known as AWS Managed Microsoft AD. The LRS Directory Information Server (LRS/DIS) is integrated with AWS Managed Microsoft AD. By modernizing your batch printing workloads, you can reduce IT infrastructure costs, mitigate the technical debt of maintaining legacy systems, remove data silos, increase agility and efficiency with a DevOps model, and take advantage of on-demand resources and automation in the AWS Cloud.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A mainframe printing or output management workload
+ Basic knowledge of how to rebuild and deliver a mainframe application that runs on Rocket Enterprise Server (For more information, see the [Rocket Enterprise Server](https://www.rocketsoftware.com/sites/default/files/resource_files/enterprise-server.pdf) data sheet in the Rocket documentation.)
+ Basic knowledge of [LRS cloud printing](https://www.lrsoutputmanagement.com/solutions/solutions-cloud-printing/) solutions and concepts
+ Rocket Enterprise Server software and license (For more information, contact [Rocket sales](https://www.rocketsoftware.com/en-us/products/enterprise-suite/request-contact).)
+ LRS VPSX/MFI, LRS/Queue, and LRS/DIS software and licenses (For more information, contact [LRS sales](https://www.lrsoutputmanagement.com/about-us/contact-us/).)
**Note**
For more information about configuration considerations for mainframe batch printing workloads, see *Considerations* in the [Additional information](#modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi-additional) section of this pattern.
**Product versions**
+ [Rocket Enterprise Server](https://www.microfocus.com/en-us/products/enterprise-server/overview?utm_campaign=7018e000000PgfnAAC&utm_content=SCH-BR-AMC-AppM-AMS&gclid=EAIaIQobChMIoZCQ6fvS9wIVxQN9Ch2MzAOlEAAYASAAEgKx2fD_BwE) 6.0 (product update 7)
+ [LRS VPSX/MFI](https://www.lrsoutputmanagement.com/products/vpsx-enterprise/) V1R3 or higher
## Architecture
**Source technology stack**
+ Operating system – IBM z/OS
+ Programming language – Common Business-Oriented Language (COBOL), job control language (JCL), and Customer Information Control System (CICS)
+ Database – IBM DB2 for z/OS and Virtual Storage Access Method (VSAM)
+ Security – Resource Access Control Facility (RACF), CA Top Secret for z/OS, and Access Control Facility 2 (ACF2)
+ Printing and output management – IBM mainframe z/OS printing products (IBM Tivoli Output Manager for z/OS, LRS, and CA View)
**Target technology stack**
+ Operating system – Microsoft Windows Server running on Amazon EC2
+ Compute – Amazon EC2
+ Programming language – COBOL, JCL, and CICS
+ Database – Amazon RDS
+ Security – AWS Managed Microsoft AD
+ Printing and output management – LRS printing solution on AWS
+ Mainframe runtime environment – Rocket Enterprise Server
**Source architecture**
The following diagram shows a typical current state architecture for a mainframe batch printing workload:
![\[From user to mainframe service, Db2 for z/OS, job scheduler, batch job, and output in six steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36de7312-4860-4702-a325-c01cf74c4f33/images/83d82435-0aa6-4eb8-a5c8-0920102afb09.png)
The diagram shows the following workflow:
1. Users perform business transactions on a system of engagement (SoE) that’s built on an IBM CICS application written in COBOL.
1. The SoE invokes the mainframe service, which records the business transaction data in a system-of-records (SoR) database such as IBM DB2 for z/OS.
1. The SoR persists the business data from the SoE.
1. The batch job scheduler initiates a batch job to generate print output.
1. The batch job extracts data from the database, formats the data according to business requirements, and then generates business output such as billing statements, ID cards, or loan statements. Finally, the batch job routes the output to printing output management for processing and output delivery, based on the business requirements.
1. Printing output management receives print output from the batch job, and then delivers that output to a specified destination, such as email, a file share that uses secure FTP, a physical printer that uses LRS printing solutions (as demonstrated in this pattern), or IBM Tivoli.
**Target architecture**
The following diagram shows an architecture for a mainframe batch printing workload that’s deployed in the AWS Cloud:
![\[Batch application on AWS with scheduler, Rocket Enterprise Server, and database in four steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36de7312-4860-4702-a325-c01cf74c4f33/images/8cdd4ef7-3cbd-476a-9aa4-c1c0924f17c6.png)
The diagram shows the following workflow:
1. The batch job scheduler initiates a batch job to create print output, such as billing statements, ID cards, or loan statements.
1. The mainframe batch job ([replatformed to Amazon EC2](https://aws.amazon.com/blogs/apn/demystifying-legacy-migration-options-to-the-aws-cloud/)) uses the Rocket Enterprise Server runtime to extract data from the application database, apply business logic to the data, format the data, and then send the data to a print destination by using [Rocket Software Print Exit](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/HCOMCMJCLOU020.html) (Micro Focus documentation).
1. The application database (an SoR that runs on Amazon RDS) persists data for print output.
1. The LRS VPSX/MFI printing solution is deployed on Amazon EC2 and its operational data is stored in Amazon Elastic Block Store (Amazon EBS). LRS VPSX/MFI uses the TCP/IP-based LRS/Queue transmission agent to collect print data through the Rocket Software JES Print Exit API and deliver the data to a specified printer destination.
**Note**
The target solution typically doesn’t require application changes to accommodate mainframe formatting languages, such as IBM Advanced Function Presentation (AFP) or Xerox Line Condition Data Stream (LCDS). For more information about using Rocket Software for mainframe application migration and modernization on AWS, see the [Empowering Enterprise Mainframe Workloads on AWS with Micro Focus](https://aws.amazon.com/blogs/apn/empowering-enterprise-grade-mainframe-workloads-on-aws-with-micro-focus/) blog post.
**AWS infrastructure architecture**
The following diagram shows a highly available and secure AWS infrastructure architecture for a mainframe batch printing workload:
![\[Multi-AZ deployment on AWS with Rocket Software and LRS components in seven steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/36de7312-4860-4702-a325-c01cf74c4f33/images/287dd143-338c-4d83-a9b2-8e39214a81b0.png)
The diagram shows the following workflow:
1. The batch scheduler initiates the batch process and is deployed on Amazon EC2 across multiple [Availability Zones](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) for high availability (HA).
**Note**
This pattern doesn’t cover the implementation of the batch scheduler. For more information about implementation, see the software vendor documentation for your scheduler.
1. The mainframe batch job (written on a programming language such as JCL or COBOL) uses core business logic to process and generate print output, such as billing statements, ID cards, and loan statements. The job is deployed on Amazon EC2 across two Availability Zones for HA and uses Rocket Software Print Exit to route print output to LRS VPSX/MFI for end-user printing.
1. LRS VPSX/MFI uses a TCP/IP-based LRS/Queue transmission agent to collect or capture print data from the Rocket Software JES Print Exit programming interface. Print Exit passes the necessary information to enable LRS VPSX/MFI to effectively process the spool file and dynamically build LRS/Queue commands. The commands are then run using a standard built-in function from Rocket Software.
**Note**
For more information on print data passed from Rocket Software Print Exit to LRS/Queue and LRS VPSX/MFI supported mainframe batch mechanisms, see *Print data capture* in the [Additional information](#modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi-additional) section of this pattern.
1.
**Note**
A [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) provides a DNS name to integrate Rocket Enterprise Server with LRS VPSX/MFI. : LRS VPSX/MFI supports a Layer 4 load balancer. The Network Load Balancer also does a basic health check on LRS VPSX/MFI and routes traffic to the registered targets that are healthy.
1.
**Note**
The LRS VPSX/MFI print server is deployed on Amazon EC2 across two Availability Zones for HA and uses [Amazon EBS](https://docs.aws.amazon.com/ebs/latest/userguide/what-is-ebs.html) as an operational data store. LRS VPSX/MFI supports both the active-active and active-passive service modes. This architecture uses multiple AZs in an active-passive pair as an active and hot standby. The Network Load Balancer performs a health check on LRS VPSX/MFI EC2 instances and routes traffic to hot standby instances in the other AZ if an active instance is in an unhealthy state. The print requests are persisted in the LRS Job Queue locally in each of the EC2 instances. In the event of recovery, a failed instance has to be restarted for the LRS services to resume processing the print request. : LRS VPSX/MFI can also perform health checks at the printer fleet level. For more information, see *Printer fleet health checks* in the [Additional information](#modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi-additional) section of this pattern.
1. [AWS Managed Microsoft AD](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) integrates with LRS/DIS to perform print workflow authentication and authorization. For more information, see *Print authentication and authorization *in the [Additional information](#modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi-additional) section of this pattern.
1. LRS VPSX/MFI uses Amazon EBS for block storage. You can back up Amazon EBS data from active EC2 instances to Amazon S3 as point-in-time snapshots and restore them to hot standby EBS volumes. To automate the creation, retention, and deletion of Amazon EBS volume snapshots, you can use [Amazon Data Lifecycle Manager](https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/) to set the frequency of automated snapshots and restore them based on your [RTO/RPO requirements](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html).
## Tools
**AWS services**
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/ebs/latest/userguide/what-is-ebs.html) provides block level storage volumes for use with EC2 instances. EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can use Amazon EC2 to launch as many or as few virtual servers as you need, and you can scale out or scale in.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for a relational database and manages common database administration tasks.
+ [AWS Directory Service for Microsoft Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html), also known as AWS Managed Microsoft AD, enables your directory-aware workloads and AWS resources to use Microsoft Active Directory in the AWS Cloud.
**Other tools**
+ [LRS VPSX/MFI (Micro Focus Interface)](https://www.lrsoutputmanagement.com/products/vpsx-enterprise/), co-developed by LRS and Rocket Software, captures output from a Rocket Enterprise Server JES spool and reliably delivers it to a specified print destination.
+ LRS Directory Information Server (LRS/DIS) is used for authentication and authorization during the print workflow.
+ TCP/IP-based LRS/Queue transmission agent is used by LRS VPSX/MFI to collect or capture print data through the Rocket Software JES Print Exit programming interface.
+ [Rocket Enterprise Server](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-A2F23243-962B-440A-A071-480082DF47E7.html) is an application deployment environment for mainframe applications. It provides the execution environment for mainframe applications that are migrated or created by using any version of Rocket Software Enterprise Developer.
## Epics
### Set up Rocket Enterprise Server on Amazon EC2 and deploy a mainframe batch application
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Rocket Enterprise Server and deploy a demo application. | Set up Rocket Enterprise Server on Amazon EC2, and then deploy the Rocket Software BankDemo demonstration application on Amazon EC2.The BankDemo application is a mainframe batch application that creates and then initiates print output. | Cloud architect |
### Set up an LRS print server on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Get an LRS product license for printing. | To get an LRS product license for LRS VPSX/MFI, LRS/Queue, and LRS/DIS, contact the [LRS Output Management team](https://www.lrsoutputmanagement.com/about-us/contact-us/). You must provide the host names of the EC2 instances where the LRS products will be installed. | Build lead |
| Create an Amazon EC2 Windows instance to install LRS VPSX/MFI. | Launch an Amazon EC2 Windows instance by following the instructions from [Launch an Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html) in the Amazon EC2 documentation. Your instance must meet the following hardware and software requirements for LRS VPSX/MFI: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html)The preceding hardware and software requirements are intended for a small printer fleet (around 500–1000). To get the full requirements, consult with your LRS and AWS contacts.When you create your Windows instance, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Install LRS VPSX/MFI on the EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Install LRS/Queue. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Install LRS/DIS. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Create a target group and register LRS VPSX/MFI EC2 as the target. | Create a target group by following the instructions from [Create a target group for your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) in the Elastic Load Balancing documentation.When you create the target group, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Create a Network Load Balancer. | Follow the instructions from [Create a Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html) in the Elastic Load Balancing documentation. Your Network Load Balancer routes traffic from Rocket Enterprise Server to LRS VPSX/MFI EC2.When you create the Network Load Balancer, do the following on the **Listeners and Routing** page:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Integrate Rocket Enterprise Server with LRS VPSX/MFI and LRS/Queue
| Task | Description | Skills required |
| --- | --- | --- |
| Configure Rocket Enterprise Server for LRS/Queue integration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html)LRS currently supports a maximum character limit of 50 for DNS names, but this is subject to change in the future. If your DNS name is greater than 50, then you can use the IP address of the Network Load Balancer as an alternative. | Cloud architect |
| Configure Rocket Enterprise Server for LRS VPSX/MFI integration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Set up printers and print users in Rocket Enterprise Server and LRS VPSX/MFI
| Task | Description | Skills required |
| --- | --- | --- |
| Associate the Rocket Software Print Exit module to the Rocket Enterprise Server batch printer Server Execution Process. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html)For more information about configuration, see [Using the Exit](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/HCOMCMJCLOS025.html) in the Micro Focus documentation. | Cloud architect |
| Add a printer in LRS VPSX/MFI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Create a print user in LRS VPSX/MFI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Set up print authentication and authorization
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS Managed Microsoft AD domain with users and groups. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Join LRS VPSX/MFI EC2 to an AWS Managed Microsoft AD domain. | Join LRS VPSX/MFI EC2 to your AWS Managed Microsoft AD domain [automatically](https://repost.aws/knowledge-center/ec2-systems-manager-dx-domain) (AWS Knowledge Center documentation) or [manually](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/launching_instance.html) (AWS Directory Service documentation). | Cloud architect |
| Configure and integrate LRS/DIS with AWS Managed Microsoft AD. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Test a print workflow
| Task | Description | Skills required |
| --- | --- | --- |
| Initiate a batch print request from the Rocket Software BankDemo app. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html) | Test engineer |
| Check the print output in LRS VPSX/MFI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-batch-printing-workloads-on-aws-by-using-rocket-enterprise-server-and-lrs-vpsx-mfi.html)You can now see the print output of an account statement with columns for **Account No.**, **Description**, **Date**, **Amount**, and **Balance**. For an example, see the **batch\$1print\$1output **attachment for this pattern. | Test engineer |
## Related resources
+ [LRS Output Modernization](https://www.lrsoutputmanagement.com/) (LRS documentation)
+ [ANSI and machine carriage controls](https://www.ibm.com/docs/en/cmofz/9.5.0?topic=tips-ansi-machine-carriage-controls) (IBM documentation)
+ [Channel command words](https://www.ibm.com/docs/en/zos/2.3.0?topic=devices-channel-command-words) (IBM documentation)
+ [Empowering Enterprise Mainframe Workloads on AWS with Micro Focus](https://aws.amazon.com/blogs/apn/empowering-enterprise-grade-mainframe-workloads-on-aws-with-micro-focus/) (AWS Partner Network Blog)
+ [Build a Micro Focus Enterprise Server PAC with Amazon EC2 Auto Scaling and Systems Manager](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-micro-focus-enterprise-server-pac-with-amazon-ec2-auto-scaling-and-systems-manager.html) (AWS Prescriptive Guidance documentation)
+ [Advanced Function Presentation (AFP) data stream](https://www.ibm.com/docs/en/i/7.4?topic=streams-advanced-function-presentation-data-stream) (IBM documentation)
+ [Line Conditioned Data Stream (LCDS)](https://www.compart.com/en/lcds) (Compart documentation)
## Additional information
**Considerations**
During your modernization journey, you may consider a wide variety of configurations for both mainframe batch processes and the output they generate. The mainframe platform has been customized by every customer and vendor that uses it with particular requirements that directly affect print. For example, your current platform may incorporate the IBM Advanced Function Presentation (AFP) or the Xerox Line Condition Data Stream (LCDS) into the current workflow. Additionally, [mainframe carriage control characters](https://www.ibm.com/docs/en/cmofz/9.5.0?topic=tips-ansi-machine-carriage-controls) and [channel command words](https://www.ibm.com/docs/en/zos/2.3.0?topic=devices-channel-command-words) can affect the look of the printed page and may need special handling. As part of the modernization planning process, we recommend that you assess and understand the configurations in your specific print environment.
**Print data capture**
Rocket Software Print Exit passes the necessary information to enable LRS VPSX/MFI to effectively process the spool file. The information consists of fields passed in the relevant control blocks, such as:
+ JOBNAME
+ OWNER (USERID)
+ DESTINATION
+ FORM
+ FILENAME
+ WRITER
LRS VPSX/MFI supports the following mainframe batch mechanisms for capturing data from Rocket Enterprise Server.
+ BATCH COBOL print/spool processing using standard z/OS JCL SYSOUT DD/OUTPUT statements
+ BATCH COBOL print/spool processing using standard z/OS JCL CA-SPOOL SUBSYS DD statements
+ IMS/COBOL print/spool processing using the CBLTDLI interface (For a full list of supported methods and programming examples, see the LRS documentation that’s included with your product license.)
**Printer fleet health checks**
LRS VPSX/MFI (LRS LoadX) can perform deep dive health checks, including device management and operational optimization. Device management can detect failure in a printer device and route the print request to a healthy printer. For more information about deep dive health checks for printer fleets, see the LRS documentation that’s included with your product license.
**Print authentication and authorization **
LRS/DIS enables LRS applications to authenticate user IDs and passwords by using Microsoft Active Directory or an LDAP server. In addition to basic print authorization, LRS/DIS can also apply granular-level print security controls in the following use cases:
+ Manage who can browse the printer job.
+ Manage the browsing level of other user's jobs.
+ Manage operational tasks. For example, command-level security such as hold/release, purge, modify, copy, and reroute. Security can be set up by either the User-ID or Group (similar to AD group or LDAP group).** **
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/36de7312-4860-4702-a325-c01cf74c4f33/attachments/attachment.zip)
# Mainframe modernization: DevOps on AWS with Rocket Software Enterprise Suite
*Kevin Yung, Amazon Web Services*
## Summary
**Customer challenges**
Organizations that run core applications on mainframe hardware usually encounter a few challenges when the hardware needs to scale up to meet the demands of digital innovations. These challenges include the following constraints.
+ Mainframe development and test environments are unable to scale due to the inflexibility of mainframe hardware components and the high cost of changing.
+ Mainframe development is facing skill shortages, because new developers are not familiar and not interested in the traditional mainframe development tools. Modern technology such as containers, continuous integration/continuous delivery (CI/CD) pipelines, and modern test frameworks are not available in mainframe development.
**Pattern outcomes**
To address these challenges, Amazon Web Services (AWS) and Rocket Software Micro Focus, an AWS Partner Network (APN) Partner, have collaborated to create this pattern. The solution is designed to help you achieve the following outcomes.
+ Improved developer productivity. Developers can be given new mainframe development instances within minutes.
+ Use of the AWS Cloud to create new mainframe test environments with virtually unlimited capacity.
+ Rapid provisioning of new mainframe CI/CD infrastructure. Provisioning on AWS can be completed within an hour by using AWS CloudFormation and AWS Systems Manager.
+ Native use of AWS DevOps tools for mainframe development, including AWS CodeBuild, AWS CodeCommit, AWS CodePipeline, AWS CodeDeploy, and Amazon Elastic Container Registry (Amazon ECR).
+ Transform traditional waterfall development to agile development in mainframe projects.
**Technologies summary**
In this pattern, the target stack contains the following components.
|
|
| Logical components | Implementation solutions | Description |
| --- |--- |--- |
| Source code repositories | Rocket Software AccuRev Server, CodeCommit, Amazon ECR | Source code management – The solution uses two types of source code: Mainframe source code, for example, COBOL and JCL. AWS infrastructure templates and automation scripts Both types of source code need version control, but they are managed in different SCMs. Source code deployed into mainframe or Rocket Software Enterprise Servers is managed in Rocket Software Micro Focus AccuRev Server. AWS templates and automation scripts are managed in CodeCommit. Amazon ECR is used for the Docker image repositories. |
| Enterprise developer instances | Amazon Elastic Compute Cloud (Amazon EC2), Rocket Software Enterprise Developer for Eclipse | Mainframe developers can develop code in Amazon EC2 by using Rocket Software Enterprise Developer for Eclipse. This eliminates the need to rely on mainframe hardware to write and test code. |
| Rocket Software Enterprise Suite license management | Rocket Software Enterprise Suite License Manager | For centralized Rocket Software Enterprise Suite license management and governance, the solution uses Rocket Software Enterprise Suite License Manager to host the required license. |
| CI/CD pipelines | CodePipeline, CodeBuild, CodeDeploy, Rocket Software Enterprise Developer in a container, Rocket Software Enterprise Test Server in a container, Rocket Software Micro Focus Enterprise Server | Mainframe development teams need CI/CD pipelines to perform code compilation, integration tests, and regression tests. In AWS, CodePipeline and CodeBuild can work with Rocket Software Enterprise Developer and Enterprise Test Server in a container natively. |
## Prerequisites and limitations
**Prerequisites **
|
|
| Name | Description |
| --- |--- |
| py3270 | py3270 is a Python interface to x3270, an IBM 3270 terminal emulator. It provides an API to a x3270 or s3270 subprocess. |
| x3270 | x3270 is an IBM 3270 terminal emulator for the X Window System and Windows. This can be used by developer for unit testing locally. |
| Robot-Framework-Mainframe-3270-Library | Mainframe3270 is a library for Robot Framework based on py3270 project. |
| Rocket Software Verastream | Rocket Software Verastream is an integration platform that enables testing mainframe assets the way that mobile apps, web applications, and SOA web services are tested. |
| Rocket Software Unified Functional Testing (UFT) installer and license | Rocket Software Unified Functional Testing is software that provides functional and regression test automation for software applications and environments. |
| Rocket Software Enterprise Server installer and license | Enterprise Server provides the runtime environment for mainframe applications. |
| Rocket Software Enterprise Test Server installer and license | Rocket Software Enterprise Test Server is an IBM mainframe application test environment. |
| Rocket Software AccuRev installer and license for Server, and Rocket Software Micro Focus AccuRev installer and license for Windows and Linux operating systems | AccuRev provides source code management (SCM). The AccuRev system is designed for use by a team of people who are developing a set of files. |
| Rocket Software Enterprise Developer for Eclipse installer, patch and license | Enterprise Developer provide mainframe developer a platform to develop and maintain the core mainframe online and batch applications. |
**Limitations **
+ Building a Windows Docker image is not supported in CodeBuild. This [reported issue](https://github.com/docker-library/docker/issues/49) needs support from Windows Kernel/HCS and Docker teams. The work-around is to create a Docker image build runbook by using Systems Manager. This pattern uses the work-around to build Rocket Software Enterpise Developer for Eclipse and Rocket Software Micro Focus Enterprise Test Server Container images.
+ Virtual private cloud (VPC) connectivity from CodeBuild is not supported in Windows yet, so the pattern does not use Rocket Software License Manager to manage licenses in OpenText Rocket Software Enterprise Developer and Rocket Software Enterprise Test Server containers.
**Product versions**
+ Rocket Software Enterprise Developer 5.5 or later
+ Rocket Software Enterprise Test Server 5.5 or later
+ Rocket Software Enterprise Server 5.5 or later
+ Rocket Software AccuRev 7.x or later
+ Windows Docker base image for Rocket Software Enterprise Developer and Enterprise Test Server: **microsoft/dotnet-framework-4.7.2-runtime**
+ Linux Docker base image for AccuRev client: **amazonlinux:2**
## Architecture
**Mainframe environment**
In conventional mainframe development, the developers need to use mainframe hardware to develop and test programs. They face capacity limitations, for example restricted million instructions per second (MIPS) for the dev/test environment, and they must rely on the tools that are available on the mainframe computers.
In many organizations, mainframe development follows the waterfall development methodology, with teams relying on long cycles to release changes. These release cycles are usually longer than digital product development.
The following diagram shows multiple mainframe projects sharing mainframe hardware for their development. In mainframe hardware, it is expensive to scale out a development and test environment for more projects.
![\[Diagram showing mainframe architecture with z/OS, databases, programming languages, and user groups.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/84e717fc-5aea-41a6-977a-d7e7a7ca5da7.png)
*AWS*** architecture **
This pattern extends mainframe development to the AWS Cloud. First, it uses AccuRev SCM to host the mainframe source code on AWS. Then it makes Enterprise Developer and Enterprise Test Server available for building and testing the mainframe code on AWS.
The following sections describe the pattern's three major components.
**1. SCM **
In AWS, the pattern uses AccuRev to create a set of SCM workspaces and version control for the mainframe source code. Its stream-based architecture enables parallel mainframe development for multiple teams. To merge a change, AccuRev uses the promote concept. To add that change to other workspaces, AccuRev uses the update concept.
At the project level, each team can create one or more streams in AccuRev to track project level changes. These are called project streams. These project streams are inherited from the same parent stream. The parent stream is used to merge the changes from different project streams.
Each project stream can promote code to AccuRev, and a promote post trigger is set up to initiate the AWS CI/CD pipeline. The successful build for a project stream change can be promoted to its parent stream for more regression tests.
Usually, the parent stream is called the system integration stream. When there is a promotion from a project stream to a system integration stream, a post promotion trigger initiates another CI/CD pipeline to run regression tests.
In addition to mainframe code, this pattern includes AWS CloudFormation templates, Systems Manager Automation documents, and scripts. Following infrastructure-as-code best practices, they are version-controlled in CodeCommit.
If you need to synchronize mainframe code back to a mainframe environment for deployment, Rocket Software provides the Enterprise Sync solution, which synchronizes code from the AccuRev SCM back to the mainframe SCM.
**2. Developer and test environments**
In a large organization, scaling more than a hundred or even more than a thousand mainframe developers is challenging. To address this constraint, the pattern uses Amazon EC2 Windows instances for development. On the instances, Enterprise Developer for Eclipse tools are installed. The developer can perform all mainframe code test and debugging locally on the instance.
AWS Systems Manager State Manager and Automation documents are used to automate the developer instance provisioning. The average time to create a developer instance is within 15 minutes. The following software and configurations are prepared:
+ AccuRev Windows client for checking out and committing source code into AccuRev
+ Enterprise Developers for Eclipse tool, for writing, testing, and debugging mainframe code locally
+ Open source testing frameworks Python behavior-driven development (BDD) test framework Behave, py3270, and the x3270 emulator for creating scripts to test applications
+ A Docker developer tool for building the Enterprise Test Server Docker image and testing the application in the Enterprise Test Server Docker container
In the development cycle, developers use the EC2 instance to develop and test mainframe code locally. When the local changes are tested successfully, developers promote the change into the AccuRev server.
**3. CI/CD pipelines**
In the pattern, CI/CD pipelines are used for integration tests and regression tests before deployment to the production environment.
As explained in the SCM section, AccuRev uses two types of streams: a project stream and an integration stream. Each stream is hooked up with CI/CD pipelines. To perform the integration between the AccuRev server and AWS CodePipeline, the pattern uses AccuRev post promotion script to create an event to initiate CI/CD.
For example, when a developer promotes a change to a project stream in AccuRev, it initiates a post promotion script to run in AccuRev Server. Then the script uploads the metadata of the change into an Amazon Simple Storage Service (Amazon S3) bucket to create an Amazon S3 event. This event will initiate a CodePipeline configured pipeline to run.
The same event-initiating mechanism is used for the integration stream and its associated pipelines.
In the CI/CD pipeline, CodePipeline uses CodeBuild with the AccuRev Linux client container to check out the latest code from the AccuRev streams. Then the pipeline starts CodeBuild to use the Enterprise Developer Windows container to compile the source code, and to use the Enterprise Test Server Windows container in CodeBuild to test mainframe applications.
The CI/CD pipelines are built using CloudFormation templates, and the blueprint will be used for new projects. By using the templates, it takes less than an hour for a project to create a new CI/CD pipeline in AWS.
To scale your mainframe test capability on AWS, the pattern builds out the Rocket Software DevOps test suite, Verastream and UFT server. By using the modern DevOps tools, you can run as many tests on AWS as you need.
An example mainframe development environment with Rocket Software on AWS is shown in the following diagram.
![\[AWS development pipeline with shared components for multiple project teams.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/27da6a52-4573-44cb-8716-1ac49430f618.png)
*Target technology stack *
This section provides a closer look at the architecture of each component in the pattern.
**1. Source code repository – AccuRev SCM **
AccuRev SCM is set up to manage mainframe source code versions. For high availability, AccuRev supports primary and replica modes. Operators can fail over to the replica when performing maintenance on the primary node.
To speed up the response of the CI/CD pipeline, the pattern uses Amazon CloudWatch Events to detect source code changes and initiate the start of the pipeline.
1. The pipeline is set up to use an Amazon S3 source.
1. A CloudWatch Events rule is set up to capture S3 events from a source S3 bucket.
1. The CloudWatch Events rule sets a target to the pipeline.
1. AccuRev SCM is configured to run a post promotion script locally after promotion is complete.
1. AccuRev SCM generates an XML file that contains the metadata of the promotion, and the script uploads the XML file to the source S3 bucket.
1. After the upload, the source S3 bucket sends events to match the CloudWatch Events rule, and the CloudWatch Events rule initiates the pipeline to run.
When the pipeline runs, it kicks off a CodeBuild project to use an AccuRev Linux client container to check out the latest mainframe code from an associated AccuRev stream.
The following diagram shows an AccuRev Server setup.
![\[AWS Cloud diagram showing AccuRev setup with primary and replica instances across availability zones.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/e60345cc-2283-4b03-8f57-e3dac1770978.png)
**2. Enterprise Developer template**
The pattern uses Amazon EC2 templates to simplify creation of the developer instance. By using State Manager, it can apply software and license settings to EC2 instances consistently.
The Amazon EC2 template builds in its VPC context settings and default instance settings, and it follows enterprise tagging requirements. By using a template, a team can create their own new development instances.
When a developer instance starts, by associating with tags, Systems Manager uses State Manager to apply automation. The automation includes the following general steps.
1. Install Enterprise Developer software and install patches.
1. Install the AccuRev client for Windows.
1. Install the pre-configured script for developers to join the AccuRev stream. Initialize Eclipse workspaces.
1. Install development tools, including x3270, py3270, and Docker.
1. Configure license settings to point to a License Manager load balancer.
The following diagram shows an Enterprise developer instance created by the Amazon EC2 template, with software and configuration applied to the instance by State Manager. Enterprise developer instances connect to AWS License Manager to activate their license.
![\[AWS Cloud diagram showing Enterprise Developer Instance setup with License Manager and Systems Manager components.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/7ca8f538-8362-4a11-a842-7ecff6fa0248.png)
**3. CI/CD pipelines**
As explained in the AWS architecture section, in the pattern, there are project-level CI/CD pipelines and system integration pipelines. Each mainframe project team creates a pipeline or multiple CI/CD pipelines for building the programs that they are developing in a project. These project CI/CD pipelines check out source code from an associated AccuRev stream.
In a project team, developers promote their code in the associated AccuRev stream. Then the promotion initiates the project pipeline to build the code and run integration tests.
Each project CI/CD pipeline uses CodeBuild projects with the Enterprise Developer tool Amazon ECR image and Enterprise Test Server tool Amazon ECR image.
CodePipeline and CodeBuild are used to create the CI/CD pipelines. Because CodeBuild and CodePipeline have no upfront fees or commitments, you pay only for what you use. Compared to mainframe hardware, the AWS solution greatly reduces hardware provisioning lead time and lowers the cost of your testing environment.
In modern development, multiple test methodologies are used. For example, test-driven development (TDD), BDD, and Robot Framework. With this pattern, developers can use these modern tools for mainframe testing. For example, by using x3270, py3270 and the Behave python test tool, you can define an online application's behavior. You can also use build mainframe 3270 robot framework in these CI/CD pipelines.
The following diagram shows the team stream CI/CD pipeline.
![\[AWS Cloud CI/CD pipeline showing CodeCommit, CodePipeline, and CodeBuild with Micro Focus tools integration.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/da59f837-2f23-404f-948b-41402cc6fe0c.png)
The following diagram shows the project CI/CD test report produced by CodePipeline in Mainframe3270 Robot Framework.
![\[Test report summary showing 100% pass rate for 3 test cases in 2.662 seconds.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/4752321a-c60d-455c-ac2f-6f0e2bc3dca0.png)
The following diagram shows the project CI/CD test report produced by CodePipeline in Py3270 and Behave BDD.
![\[Test report summary showing 100% pass rate for 2 test cases in a pipeline.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/d005466e-aeb8-4fd6-8342-743ed049f98a.png)
After project level tests are passed successfully, the tested code is manually promoted to the integration stream in AccuRev SCM. You can automate this step after the teams have a confidence on the tests coverage of their project pipeline.
When code is promoted, the system integration CI/CD pipeline checks out the merged code and performs regression tests. The merged code is promoted from all parallel project streams.
Depending on how fine grain the test environment is required, customers can have more system integration CI/CD pipelines in a different environment, for example UAT, Pre-Production.
In the pattern, the tools used in the system integration pipeline are Enterprise Test Server, UFT Server, and Verastream. All these tools can be deployed into the Docker container and used with CodeBuild.
After successfully testing of the mainframe programs, the artifact is stored, with version control, in an S3 bucket.
The following diagram shows a system integration CI/CD pipeline.
![\[CI/CD pipeline showing AWS services and Micro Focus tools for source, build, test, and promote stages.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/693212e5-1cd0-4f82-a910-39b00d977c38.png)
After the artifact has been successfully tested in the system integration CI/CD pipelines, it can be promoted for production deployment.
If you need to deploy source code back to the mainframe, Rocket Software offers the Enterprise Sync solution to synchronize source code from AccuRev back to Mainframe Endeavour.
The following diagram shows a production CI/CD pipeline deploying the artifact into Enterprise Servers. In this example, CodeDeploy orchestrates the deployment of the tested mainframe artifact into Enterprise Server.
![\[CI/CD pipeline diagram showing CodePipeline, CodeBuild, and CodeDeploy stages for artifact deployment.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2359db4c-f351-45a6-8516-88a3b62e61f9/images/56749c2a-e038-4e56-9487-b2ff83894725.png)
In addition to the architecture walkthrough of the CI/CD pipeline, see the AWS DevOps blog post [Automate thousands of mainframe tests on AWS with the Micro Focus Enterprise Suite](https://aws.amazon.com/blogs/devops/automate-mainframe-tests-on-aws-with-micro-focus/) for more information on testing mainframe applications in CodeBuild and CodePipeline. (Micro Focus is now Rocket Software.) Refer to the blog post for the best practices and details of doing mainframe tests on AWS.
## Tools
**AWS automation tools**
+ [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)
+ [Amazon CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html)
+ [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html)
+ [AWS CodeDeploy](https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html)
+ [AWS CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html)
+ [Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html)
+ [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html)
+ [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)
+ [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html)
**Rocket Software tools**
+ [Rocket Enterprise Developer for Eclipse](https://www.microfocus.com/documentation/enterprise-developer/ed60/ED-Eclipse/GUID-8D6B7358-AC35-4DAF-A445-607D8D97EBB2.html)
+ [Rocket Enterprise Test Server](https://www.microfocus.com/documentation/enterprise-developer/ed60/ETS-help/GUID-ECA56693-D9FE-4590-8798-133257BFEBE7.html)
+ [Rocket Enterprise Server](https://www.microfocus.com/documentation/enterprise-developer/es_60/) (production deployment)
+ [Rocket Software AccuRev](https://supportline.microfocus.com/documentation/books/AccuRev/AccuRev/6.2/webhelp/wwhelp/wwhimpl/js/html/wwhelp.htm)
+ [Rocket Software Enterprise Suite License Manager](https://www.microfocus.com/documentation/slm/)
+ [Rocket Software Verastream Host Integrator](https://www.microfocus.com/documentation/verastream-host-integrator/)
+ [Rocket Software UFT One](https://admhelp.microfocus.com/uft/en/24.4/UFT_Help/Content/User_Guide/Ch_UFT_Intro.htm)
**Other tools**
+ x3270
+ [py3270](https://pypi.org/project/py3270/)
+ [Robot-Framework-Mainframe-3270-Library](https://github.com/Altran-PT-GDC/Robot-Framework-Mainframe-3270-Library)
## Epics
### Create the AccuRev SCM infrastructure
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy a primary AccuRev SCM server by using CloudFormation. | | AWS CloudFormation |
| Create the AccuRev Administrator user. | Log in to AccuRev SCM Server, and run the CLI command to create an Administrator user. | AccuRev SCM Server Administrator |
| Create AccuRev streams. | Create AccuRev streams that inherit from upper streams in sequence: Production, System Integration, Team streams. | AccuRev SCM Administrator |
| Create the developer AccuRev login accounts. | Use AccuRev SCM CLI commands to create AccuRev users login accounts for mainframe developers. | AccuRev SCM Administrator |
### Create the Enterprise Developer Amazon EC2 launch template
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the Amazon EC2 launch template by using CloudFormation. | Use CloudFormation to deploy an Amazon EC2 launch template for Enterprise Developer instances. The template includes a Systems Manager Automation document for the Rocket Enterprise Developer instance. | AWS CloudFormation |
| Create the Enterprise Developer instance from the Amazon EC2 template. | | AWS Console Login and Mainframe Developer Skills |
### Create the Enterprise Developer tool Docker image
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Enterprise Developer tool Docker image. | Use the Docker command and the Enterprise Developer tool Dockerfile to create the Docker image. | Docker |
| Create the Docker repository in Amazon ECR. | On the Amazon ECR console, create the repository for the Enterprise Developer Docker image. | Amazon ECR |
| Push the Enterprise Developer tool Docker image to Amazon ECR. | Run the Docker push command to push the Enterprise Developer tool Docker image to save it in the Docker repository in Amazon ECR. | Docker |
### Create the Enterprise Test Server Docker image
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Enterprise Test Server Docker image. | Use the Docker command and the Enterprise Test Server Dockerfile to create the Docker image. | Docker |
| Create the Docker repository in Amazon ECR. | On the Amazon ECR console, create the Amazon ECR repository for the Enterprise Test Server Docker image. | Amazon ECR |
| Push the Enterprise Test Server Docker image to Amazon ECR. | Run the Docker push command to push and save the Enterprise Test Server Docker image in Amazon ECR. | Docker |
### Create the team stream CI/CD pipeline
| Task | Description | Skills required |
| --- | --- | --- |
| Create the CodeCommit repository. | On the CodeCommit console, create a Git-based repository for infrastructure and CloudFormation code. | AWS CodeCommit |
| Upload the CloudFormation template and the automation code into the CodeCommit repository. | Run the Git push command to upload CloudFormation template and automation code into the repository. | Git |
| Deploy the team stream CI/CD pipeline by using CloudFormation. | Use the prepared CloudFormation template to deploy a team stream CI/CD pipeline. | AWS CloudFormation |
### Create the system integration CI/CD pipeline
| Task | Description | Skills required |
| --- | --- | --- |
| Create the UFT Docker image. | Use the Docker command and the UFT Dockerfile to create the Docker image. | Docker |
| Create the Docker repository in Amazon ECR for the UFT image. | On the Amazon ECR console, create the Docker repository for the UFT image. | Amazon ECR |
| Push the UFT Docker image to Amazon ECR. | Run the Docker push command to push and save the Enterprise Test Server Docker image in Amazon ECR. | Docker |
| Create the Verastream Docker image. | Use the Docker command and the Verastream Dockerfile to create the Docker image. | Docker |
| Create the Docker repository in Amazon ECR for the Verastream image. | On the Amazon ECR console, create the Docker repository for the Verastream image. | Amazon ECR |
| Deploy the system integration CI/CD pipeline by using CloudFormation. | Use the prepared CloudFormation template to deploy a system integration CI/CD pipeline. | AWS CloudFormation |
### Create production deployment CI/CD pipeline
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy Enterprise Server by using the AWS Quick Start. | To deploy Enterprise Server by using CloudFormation, launch the Enterprise Server on AWS Quick Start. | AWS CloudFormation |
| Deploy a production deployment CI/CD pipeline. | On the CloudFormation console, use the CloudFormation template to deploy a production deployment CI/CD pipeline. | AWS CloudFormation |
## Related resources
**References**
+ [AWS DevOps Blog - Automate thousands of mainframe tests on AWS with the Micro Focus Enterprise Suite](https://aws.amazon.com/blogs/devops/automate-mainframe-tests-on-aws-with-micro-focus/) (Micro Focus is now Rocket Software.)
+ [py3270/py3270 GitHub repository](https://github.com/py3270/py3270)
+ [Altran-PT-GDC/Robot-Framework-Mainframe-3270-Library GitHub repository](https://github.com/Altran-PT-GDC/Robot-Framework-Mainframe-3270-Library)
+ [Welcome to behave\$1](https://behave.readthedocs.io/en/latest/index.html)
+ [APN Partner Blog - Tag: Micro Focus](https://aws.amazon.com/blogs/apn/tag/micro-focus/) (Micro Focus is now Rocket Software.)
+ [Launching an instance from a launch template](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html)
**AWS Marketplace**
+ [Rocket Software UFT One](https://aws.amazon.com/marketplace/pp/B01EGCA5OS?ref_=srh_res_product_title)
**AWS Quick Start**
+ [Rocket Enterprise Server on AWS](https://aws.amazon.com/quickstart/architecture/micro-focus-enterprise-server/)
# Modernize mainframe online printing workloads on AWS by using Micro Focus Enterprise Server and LRS VPSX/MFI
*Shubham Roy and Kevin Yung, Amazon Web Services*
*Abraham Rondon, Micro Focus*
*Guy Tucker, Levi, Ray and Shoup Inc*
## Summary
This pattern shows you how to modernize your business-critical mainframe online printing workloads on the Amazon Web Services (AWS) Cloud by using Micro Focus Enterprise Server as a runtime for a modernized mainframe application and LRS VPSX/MFI (Micro Focus Interface) as a print server. The pattern is based on the [replatform](https://aws.amazon.com/blogs/apn/demystifying-legacy-migration-options-to-the-aws-cloud/) mainframe modernization approach. In this approach, you migrate your mainframe online application to Amazon Elastic Compute Cloud (Amazon EC2) and migrate your mainframe database, such as IBM DB2 for z/OS, to Amazon Relational Database Service (Amazon RDS). The authentication and authorization for the modernized print workflow is performed by AWS Directory Service for Microsoft Active Directory, also known as AWS Managed Microsoft AD. The LRS Directory Information Server (LRS/DIS) is integrated with AWS Managed Microsoft AD for print workflow authentication and authorization. By modernizing your online printing workloads, you can reduce IT infrastructure costs, mitigate the technical debt of maintaining legacy systems, remove data silos, increase agility and efficiency with a DevOps model, and take advantage of on-demand resources and automation in the AWS Cloud.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A mainframe online printing or output management workload
+ Basic knowledge of how to rebuild and deliver a mainframe application that runs on Micro Focus Enterprise Server (For more information, see the [Enterprise Server](https://www.microfocus.com/media/data-sheet/enterprise_server_ds.pdf) data sheet in the Micro Focus documentation.)
+ Basic knowledge of LRS cloud printing solutions and concepts (For more information, see [Output Modernization](https://www.lrsoutputmanagement.com/products/modernization-products) in the LRS documentation.)
+ Micro Focus Enterprise Server software and license (For more information, contact [Micro Focus sales](https://www.microfocus.com/en-us/contact/contactme).)
+ LRS VPSX/MFI, LRS/Queue, and LRS/DIS software and licenses (For more information, contact [LRS sales](https://www.lrsoutputmanagement.com/about-us/contact-us/).)
**Note**
For more information about configuration considerations for mainframe online printing workloads, see *Considerations* in the *Additional information* section of this pattern.
**Product versions**
+ [Micro Focus Enterprise Server](https://www.microfocus.com/en-us/products/enterprise-server/overview?utm_campaign=7018e000000PgfnAAC&utm_content=SCH-BR-AMC-AppM-AMS&gclid=EAIaIQobChMIoZCQ6fvS9wIVxQN9Ch2MzAOlEAAYASAAEgKx2fD_BwE) 8.0 or later
+ [LRS VPSX/MFI](https://www.lrsoutputmanagement.com/products/modernization-products/) V1R3 or later
## Architecture
**Source technology stack**
+ Operating system – IBM z/OS
+ Programming language – Common Business-Oriented Language (COBOL) and Customer Information Control System (CICS)
+ Database – IBM DB2 for z/OS IBM Information Management System (IMS) and Virtual Storage Access Method (VSAM)
+ Security – Resource Access Control Facility (RACF), CA Top Secret for z/OS, and Access Control Facility 2 (ACF2)
+ Printing and output management – IBM mainframe z/OS printing products (IBM Infoprint Server for z/OS, LRS, and CA View)
**Target technology stack**
+ Operating system – Microsoft Windows Server running on Amazon EC2
+ Compute – Amazon EC2
+ Programming language – COBOL and CICS
+ Database – Amazon RDS
+ Security – AWS Managed Microsoft AD
+ Printing and output management – LRS printing solution on AWS
+ Mainframe runtime environment – Micro Focus Enterprise Server
**Source architecture**
The following diagram shows a typical current state architecture for a mainframe online printing workload.
![\[Six-step process to produce viewable output.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/924cdae7-9265-4fc9-8e5e-bb2da5368e7e/images/293368f5-d102-4f4e-b290-71da4aeff347.png)
The diagram shows the following workflow:
1. Users perform business transactions on a system of engagement (SoE) that’s built on an IBM CICS application written in COBOL.
1. The SoE invokes the mainframe service, which records the business transaction data in a system-of-records (SoR) database such as IBM DB2 for z/OS.
1. The SoR persists the business data from the SoE.
1. A user initiates a request to generate print output from the CICS SoE, which initiates a print transaction application to process the print request.
1. The print transaction application (such as a CICS and COBOL program) extracts data from the database, formats the data according to business requirements, and generates business output (print data) such as billing statements, ID cards, or loan statements. Then, the application sends a print request by using Virtual Telecommunications Access Method (VTAM). A z/OS print server (such as IBM Infoprint Server) uses NetSpool or a similar VTAM component to intercept the print requests, and then creates print output datasets on the JES spool by using JES output parameters. The JES output parameters specify routing information that the print server uses to transmit the output to a particular network printer. The term *VTAM* refers to the z/OS Communications Server and the System Network Architecture (SNA) services element of z/OS.
1. The printing output transmission component transmits the output print datasets from the JES spool to remote printers or print servers, such as LRS (as demonstrated in this pattern), IBM Infoprint Server, or email destinations.
**Target architecture**
The following diagram shows an architecture for a mainframe online printing workload that’s deployed in the AWS Cloud:
![\[Four-step process from initiate print request to processing on AWS to LRS printing.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/924cdae7-9265-4fc9-8e5e-bb2da5368e7e/images/07c97b6f-1a86-493d-a4e0-b8321b46f9b7.png)
The diagram shows the following workflow:
1. A user initiates a print request from an online (CICS) user-interface to create print output, such as billing statements, ID cards, or loan statements.
1. The mainframe online application ([replatformed to Amazon EC2](https://aws.amazon.com/blogs/apn/demystifying-legacy-migration-options-to-the-aws-cloud/)) uses the Micro Focus Enterprise Server runtime to extract data from the application database, apply business logic to the data, format the data, and then send the data to a print destination by using [Micro Focus CICS Print Exit](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/HCOMCMJCLOU020.html) (DFHUPRNT).
1. The application database (an SoR that runs on Amazon RDS) persists data for print output.
1. The LRS VPSX/MFI printing solution is deployed on Amazon EC2, and its operational data is stored in Amazon Elastic Block Store (Amazon EBS). LRS VPSX/MFI uses a TCP/IP-based LRS/Queue transmission agent to collect print data through the Micro Focus CICS Print Exit API (DFHUPRNT) and deliver the data to a specified printer destination. The original TERMID (TERM) that’s used in the modernized CICS application is used as the VPSX/MFI Queue name.
**Note**
The target solution typically doesn’t require application changes to accommodate mainframe formatting languages, such as IBM Advanced Function Presentation (AFP) or Xerox Line Condition Data Stream (LCDS). For more information about using Micro Focus for mainframe application migration and modernization on AWS, see [Empowering Enterprise Mainframe Workloads on AWS with Micro Focus](https://aws.amazon.com/blogs/apn/empowering-enterprise-grade-mainframe-workloads-on-aws-with-micro-focus/) in the AWS documentation.
**AWS infrastructure architecture**
The following diagram shows a highly available and secure AWS infrastructure architecture for a mainframe online printing workload:
![\[Two Availability Zones with Micro Focus Enterprise server on EC2, Amazon RDS, and LRS printing.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/924cdae7-9265-4fc9-8e5e-bb2da5368e7e/images/093555a1-342c-420c-bb90-e9440d2e8650.png)
The diagram shows the following workflow:
1. The mainframe online application (written on a programming language such as CICS or COBOL) uses core business logic to process and generate print output, such as billing statements, ID cards, and loan statements. The online application is deployed on Amazon EC2 across two [Availability Zones](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) (AZ) for high availability (HA) and uses Micro Focus CICS Print Exit to route print output to LRS VPSX/MFI for end-user printing.
1. LRS VPSX/MFI uses a TCP/IP-based LRS/Queue transmission agent to collect or capture print data from the Micro Focus online Print Exit programming interface. Online Print Exit passes the necessary information to enable LRS VPSX/MFI to effectively process the print file and dynamically build LRS/Queue commands.
**Note**
For more information on various CICS application programing methods for print and how they are supported in Micro Focus Enterprise server and LRS VPSX/MFI, see *Print data capture* in the *Additional information* section of this pattern.
1.
**Note**
A [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) provides a DNS name to integrate Micro Focus Enterprise Server with LRS VPSX/MFI. : LRS VPSX/MFI supports a Layer 4 load balancer. The Network Load Balancer also does a basic health check on LRS VPSX/MFI and routes traffic to the registered targets that are healthy.
1. The LRS VPSX/MFI print server is deployed on Amazon EC2 across two Availability Zones for HA and uses [Amazon EBS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) as an operational data store. LRS VPSX/MFI supports both the active-active and active-passive service modes. This architecture uses multiple Availability Zones in an active-passive pair as an active and hot standby. The Network Load Balancer performs a health check on LRS VPSX/MFI EC2 instances and routes traffic to hot standby instances in another Availability Zone if an active instance is in an unhealthy state. The print requests are persisted in the LRS Job Queue locally in each of the EC2 instances. In the event of recovery, a failed instance has to be restarted for the LRS services to resume processing the print request.
**Note**
LRS VPSX/MFI can also perform health checks at the printer fleet level. For more information, see *Printer fleet health checks* in the *Additional information* section of this pattern.
1. [AWS Managed Microsoft AD](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) integrates with LRS/DIS to perform print workflow authentication and authorization. For more information, see *Print authentication and authorization* in the *Additional information* section of this pattern.
1. LRS VPSX/MFI uses Amazon EBS for block storage. You can back up Amazon EBS data from active EC2 instances to Amazon S3 as point-in-time snapshots and restore them to hot standby EBS volumes. To automate the creation, retention, and deletion of Amazon EBS volume snapshots, you can use [Amazon Data Lifecycle Manager](https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/) to set the frequency of automated snapshots and restore them based on your [RTO/RPO requirements](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html).
## Tools
**AWS services**
+ [Amazon Elastic Block Store (Amazon EBS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html) provides block-level storage volumes for use with Amazon EC2 instances. EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
+ [AWS Directory Service for Microsoft Active Directory (AD)](https://aws.amazon.com/directoryservice/active-directory/), also known as AWS Managed Microsoft Active Directory, enables your directory-aware workloads and AWS resources to use managed Active Directory in AWS.
**Other tools**
+ [LRS VPSX/MFI (Micro Focus Interface)](https://www.lrsoutputmanagement.com/products/modernization-products/), co-developed by LRS and Micro Focus, captures output from a Micro Focus Enterprise Server JES spool and reliably delivers it to a specified print destination.
+ LRS Directory Information Server (LRS/DIS) is used for authentication and authorization during the print workflow.
+ LRS/Queue is a TCP/IP-based LRS/Queue transmission agent, used by LRS VPSX/MFI, to collect or capture print data through the Micro Focus online Print Exit programming interface.
+ [Micro Focus Enterprise Server](https://www.microfocus.com/documentation/enterprise-developer/ed60/ES-WIN/GUID-A2F23243-962B-440A-A071-480082DF47E7.html) is an application deployment environment for mainframe applications. It provides the execution environment for mainframe applications that are migrated or created by using any version of Micro Focus Enterprise Developer.
## Epics
### Set up Micro Focus Enterprise Server on Amazon EC2 and deploy a mainframe online application
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Micro Focus Enterprise Server and deploy a demo online application. | Set up Micro Focus Enterprise Server on Amazon EC2, and then deploy the Micro Focus Account Demo application (ACCT Demo) on Amazon EC2 by following the instructions from [Tutorial: CICS Support](https://www.microfocus.com/documentation/enterprise-developer/ed70/ED-Eclipse/GMWALK00.html) in the Micro Focus documentation.The ACCT Demo application is a mainframe online (CICS) application that creates and then initiates print output. | Cloud architect |
### Set up an LRS print server on Amazon EC2
| Task | Description | Skills required |
| --- | --- | --- |
| Get an LRS product license for printing. | To get an LRS product license for LRS VPSX/MFI, LRS/Queue, and LRS/DIS, contact the [LRS Output Management team](https://www.lrsoutputmanagement.com/about-us/contact-us/). You must provide the host names of the EC2 instances where the LRS products will be installed. | Build lead |
| Create an Amazon EC2 Windows instance to install LRS VPSX/MFI. | Launch an Amazon EC2 Windows instance by following the instructions from [Step 1: Launch an instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2_GetStarted.html#ec2-launch-instance) in the Amazon EC2 documentation. Your instance must meet the following hardware and software requirements for LRS VPSX/MFI: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html)The preceding hardware and software requirements are intended for a small printer fleet (around 500–1000). To get the full requirements, consult with your LRS and AWS contacts.When you create your Windows instance, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Install LRS VPSX/MFI on the EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Install LRS/Queue. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Install LRS/DIS. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Create a target group and register LRS VPSX/MFI EC2 as the target. | Create a target group by following the instructions from [Create a target group for your Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html) in the Elastic Load Balancing documentation.When you create the target group, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Create a Network Load Balancer. | Follow the instructions from [Create a Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html) in the Elastic Load Balancing documentation. Your Network Load Balancer routes traffic from Micro Focus Enterprise Server to LRS VPSX/MFI EC2.When you create the Network Load Balancer, do the following on the **Listeners and Routing** page:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Integrate Micro Focus Enterprise Server with LRS VPSX/MFI and LRS/Queue
| Task | Description | Skills required |
| --- | --- | --- |
| Configure Micro Focus Enterprise Server for LRS/Queue integration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Make CICS Print Exit (DFHUPRNT) available to Micro Focus Enterprise Server initialization. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html)**Validate that Micro Focus Enterprise Server has detected CICS Print Exit (DFHUPRNT)**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Define the CICS printer's terminal ID (TERMIDs) as Micro Focus Enterprise Server. | **Enable 3270 printing in Micro Focus Enterprise Server**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html)**Define the CICS printer's terminal in Micro Focus Enterprise Server**[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Set up printers and print users in Micro Focus Enterprise Server and LRS VPSX/MFI
| Task | Description | Skills required |
| --- | --- | --- |
| Create a print queue in the LRS VPSX. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html)The print queue must be equivalent to the Print TERMIDs created in Micro Focus Enterprise Server. | Cloud architect |
| Create a print user in LRS VPSX/MFI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Set up print authentication and authorization
| Task | Description | Skills required |
| --- | --- | --- |
| Create an AWS Managed Microsoft AD domain with users and groups. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
| Join LRS VPSX/MFI EC2 to an AWS Managed Microsoft AD domain. | Join LRS VPSX/MFI EC2 to your AWS Managed Microsoft AD domain [automatically](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-systems-manager-dx-domain/) (AWS Knowledge Center documentation) or [manually](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/join_windows_instance.html) (AWS Directory Service documentation). | Cloud architect |
| Configure and integrate LRS/DIS with AWS Managed Microsoft AD. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html) | Cloud architect |
### Test an online print workflow
| Task | Description | Skills required |
| --- | --- | --- |
| Initiate an online print request from the Micro Focus ACCT Demo app. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html)The "Print Request Scheduled" message appears at the bottom of the screen. This confirms that an online print request was generated from the ACCT Demo application and sent to LRS VPS/MFI for print processing. | Cloud architect |
| Check the print output in LRS VPSX/MFI. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/modernize-mainframe-online-printing-workloads-on-aws-by-using-micro-focus-enterprise-server-and-lrs-vpsx-mfi.html)You can now see the print output of an account statement with columns for Account No., SURNAME, FIRST, ADDRESS, TELEPHONE, No. Cards Issued, Date issued, Amount, and Balance.For an example, see the **online\$1print\$1output** attachment for this pattern. | Test engineer |
## Related resources
+ [LRS Output Modernization](https://www.lrsoutputmanagement.com/products/modernization-products) (LRS documentation)
+ [VTAM networking concepts](https://www.ibm.com/docs/en/zos/2.1.0?topic=guide-vtam-networking-concepts) (IBM documentation)
+ [Summary of logical unit (LU) types](https://www.ibm.com/docs/en/wsfz-and-o/1.1?topic=installation-summary-logical-unit-lu-types) (IBM documentation)
+ [ANSI and machine carriage controls](https://www.ibm.com/docs/en/cmofz/9.5.0?topic=tips-ansi-machine-carriage-controls) (IBM documentation)
+ [Empowering Enterprise Mainframe Workloads on AWS with Micro Focus](https://aws.amazon.com/blogs/apn/empowering-enterprise-grade-mainframe-workloads-on-aws-with-micro-focus/) (AWS Partner Network Blog)
+ [Build a Micro Focus Enterprise Server PAC with Amazon EC2 Auto Scaling and Systems Manager](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/build-a-micro-focus-enterprise-server-pac-with-amazon-ec2-auto-scaling-and-systems-manager.html) (AWS Prescriptive Guidance documentation)
+ [Advanced Function Presentation (AFP) data stream](https://www.ibm.com/docs/en/i/7.4?topic=streams-advanced-function-presentation-data-stream) (IBM documentation)
+ [Line Conditioned Data Stream (LCDS)](https://www.compart.com/en/lcds) (Compart documentation)
## Additional information
**Considerations**
During your modernization journey, you may consider a wide variety of configurations for mainframe online processes and the output they generate. The mainframe platform has been customized by every customer and vendor that uses it with particular requirements that directly affect print. For example, your current platform may incorporate the IBM Advanced Function Presentation (AFP) or the Xerox Line Condition Data Stream (LCDS) into the current workflow. Additionally, [mainframe carriage control characters](https://www.ibm.com/docs/en/cmofz/9.5.0?topic=tips-ansi-machine-carriage-controls) and [channel command words](https://www.ibm.com/docs/en/zos/2.3.0?topic=devices-channel-command-words) can affect the look of the printed page and may need special handling. As part of the modernization planning process, we recommend that you assess and understand the configurations in your specific print environment.
**Print data capture**
This section summarizes the CICS application programming methods that you can use in an IBM mainframe environment for printing. LRS VPSX/MFI components provide techniques to allow the same application programs to create data in the same way. The following table describes how each application programming method is supported in a modernized CICS application running in AWS and Micro Focus Enterprise Server with an LRS VPSX/MFI print server.
|
|
| Method | Description | Support for the method in a modernized environment |
| --- |--- |--- |
| EXEC CICS SEND TEXT.. or EXEC CICS SEND MAP.. | These CICS and VTAM methods are responsible for creating and delivering 3270/SCS print data streams to LUTYPE0, LUTYPE1, and LUTYPE3 print devices. | A Micro Focus online Print Exit (DFHUPRNT) application program interface (API) enables print data to be processed by VPSX/MFI when 3270/SCS print data streams are created by using either of these methods. |
| EXEC CICS SEND TEXT.. or EXEC CICS SEND MAP.. (with third-party IBM mainframe software) | The CICS and VTAM methods are responsible for creating and delivering 3270/SCS print data streams to LUTYPE0, LUTYPE1, and LUTYPE3 print devices. Third-party software products intercept the print data, convert the data to standard print format data with an ASA/MCH control character, and place the data on the JES spool to be processed by mainframe-based printing systems that use JES. | A Micro Focus online Print Exit (DFHUPRNT) API enables print data to be processed by VPSX/MFI when 3270/SCS print data streams are created by using either of these methods. |
| EXEC CICS SPOOLOPEN | This method is used by CICS application programs to write data directly to the JES spool. The data then becomes available to be processed by mainframe-based printing systems that use JES. | Micro Focus Enterprise Server spools the data to the Enterprise Server spool where it can be processed by the VPSX/MFI Batch Print Exit (LRSPRTE6) that spools the data to VPSX. |
| DRS/API | An LRS-supplied programmatic interface is used for writing print data to JES. | VPSX/MFI supplies a replacement interface that spools the print data directly to VPSX. |
**Printer fleet health checks**
LRS VPSX/MFI (LRS LoadX) can perform deep dive health checks, including device management and operational optimization. Device management can detect failure in a printer device and route the print request to a healthy printer. For more information about deep dive health checks for printer fleets, see the LRS documentation that’s included with your product license.
**Print authentication and authorization **
LRS/DIS enables LRS applications to authenticate user IDs and passwords by using Microsoft Active Directory or an LDAP server. In addition to basic print authorization, LRS/DIS can also apply granular-level print security controls in the following use cases:
+ Manage who can browse the printer job.
+ Manage the browsing level of other user's jobs.
+ Manage operational tasks. For example, command-level security such as hold/release, purge, modify, copy, and reroute. Security can be set up by either the User-ID or Group (similar to AD group or LDAP group).
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/924cdae7-9265-4fc9-8e5e-bb2da5368e7e/attachments/attachment.zip)
# Move mainframe files directly to Amazon S3 using Transfer Family
*Luis Gustavo Dantas, Amazon Web Services*
## Summary
As part of the modernization journey, you can face the challenge of transferring files between your on-premises servers and the Amazon Web Services (AWS) Cloud. Transferring data from mainframes can be a significant challenge because mainframes typically can’t access modern data stores like Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), or Amazon Elastic File System (Amazon EFS).
Many customers use intermediate staging resources, such as on-premises Linux, Unix, or Windows servers, to transfer files to the AWS Cloud. You can avoid this indirect method by using AWS Transfer Family with the Secure Shell (SSH) File Transfer Protocol (SFTP) to upload mainframe files directly to Amazon S3.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A virtual private cloud (VPC) with a subnet that’s reachable by your legacy platform
+ A Transfer Family endpoint for your VPC
+ Mainframe Virtual Storage Access Method (VSAM) files converted to sequential, [fixed-length files](https://www.ibm.com/docs/en/zos/2.1.0?topic=reports-converting-vb-fb) (IBM documentation)
**Limitations**
+ SFTP transfers files in binary mode by default, which means that files are uploaded to Amazon S3 with EBCDIC encoding preserved. If your file doesn't contain binary or packed data, then you can use the **sftp **[ascii subcommand](https://www.ibm.com/docs/en/zos/2.3.0?topic=version-what-zos-openssh-supports) (IBM documentation) to convert your files to text during the transfer.
+ You must [unpack mainframe files](https://apg-library.amazonaws.com/content/f5907bfe-7dff-4cd0-8523-57015ad48c4b) (AWS Prescriptive Guidance) that contain packed and binary content to use these files in your target environment.
+ Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 TB. For more information about Amazon S3 capabilities, see [Amazon S3 FAQs](https://aws.amazon.com/s3/faqs/?nc1=h_ls).
## Architecture
**Source technology stack**
+ Job control language (JCL)
+ z/OS Unix shell and ISPF
+ SFTP
+ VSAM and flat files
**Target technology stack**
+ Transfer Family
+ Amazon S3
+ Amazon Virtual Private Cloud (Amazon VPC)
**Target architecture**
The following diagram shows a reference architecture for using Transfer Family with SFTP to upload mainframe files directly to an S3 bucket.
![\[Using Transfer Family with SFTP to upload mainframe files directly to an S3 bucket\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/1f4fa1fd-b681-41bc-81d8-d556426b14c2/images/110491d5-b58d-4451-8de9-e742756bb192.png)
The diagram shows the following workflow:
1. You use a JCL job to transfer your mainframe files from the legacy mainframe to the AWS Cloud through Direct Connect.
1. Direct Connect enables your network traffic to remain on the AWS global network and bypass the public internet. Direct Connect also enhances the network speed, starting at 50 Mbps and scaling up to 100 Gbps.
1. The VPC endpoint enables connections between your VPC resources and the supported services without using the public internet. Access to Transfer Family and Amazon S3 achieves high availability by taking place through the elastic network interfaces located in two private subnets and Availability Zones.
1. Transfer Family authenticates users and uses SFTP to receive your files from the legacy environment and move them to an S3 bucket.
**Automation and scale**
After the Transfer Family service is in place, you can transfer an unlimited number of files from the mainframe to Amazon S3 by using a JCL job as the SFTP client. You can also automate the file transfer by using a mainframe batch job scheduler to run the SFTP jobs when you’re ready to transfer the mainframe files.
## Tools
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
+ [AWS Transfer Family](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-family.html) enables you to securely scale your recurring business-to-business file transfers to Amazon S3 and Amazon EFS by using SFTP, FTPS, and FTP protocols.
## Epics
### Create the S3 bucket and the access policy
| Task | Description | Skills required |
| --- | --- | --- |
| Create the S3 bucket. | [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to host the files that you transfer from your legacy environment. | General AWS |
| Create the IAM role and policy. | Transfer Family uses your AWS Identity and Access Management (IAM) role to grant access to the S3 bucket that you created earlier.[Create an IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) that includes the following [IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html):
You must choose the Transfer use case when you create the IAM role. | General AWS |
### Define the transfer service
| Task | Description | Skills required |
| --- | --- | --- |
| Create the SFTP server. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/move-mainframe-files-directly-to-amazon-s3-using-transfer-family.html)For more information about how to set up an SFTP server, see [Create an SFTP-enabled server](https://docs.aws.amazon.com/transfer/latest/userguide/create-server-sftp.html) (AWS Transfer Family User Guide). | General AWS |
| Get the server address. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/move-mainframe-files-directly-to-amazon-s3-using-transfer-family.html) | General AWS |
| Create the SFTP client key pair. | Create an SSH key pair for either [Microsoft Windows](https://docs.aws.amazon.com/transfer/latest/userguide/key-management.html#windows-ssh) or [macOS/Linux/UNIX](https://docs.aws.amazon.com/transfer/latest/userguide/key-management.html#macOS-linux-unix-ssh). | General AWS, SSH |
| Create the SFTP user. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/move-mainframe-files-directly-to-amazon-s3-using-transfer-family.html) | General AWS |
### Transfer the mainframe file
| Task | Description | Skills required |
| --- | --- | --- |
| Send the SSH private key to the mainframe. | Use SFTP or SCP to send the SSH private key to the legacy environment.SFTP example:
sftp [USERNAME@mainframeIP] [password] cd [/u/USERNAME] put [your-key-pair-file]
Next, store the SSH key in the z/OS Unix file system under the user name that will later run the file transfer batch job (for example, `/u/CONTROLM`). For more information about z/OS Unix shell, see [An introduction to the z/OS shells](https://www.ibm.com/docs/en/zos/2.2.0?topic=shells-introduction-zos) (IBM documentation). | Mainframe, z/OS Unix shell, FTP, SCP |
| Create the JCL SFTP client. | Because mainframes don't have a native SFTP client, you must use the BPXBATCH utility to run the SFTP client from the z/OS Unix shell.In the ISPF editor, create the JCL SFTP client. For example:
For more information about how to run a command in the z/OS Unix shell, see [The BPXBATCH utility](https://www.ibm.com/docs/en/zos/2.2.0?topic=ispf-bpxbatch-utility) (IBM documentation). For more information about how to create or edit JCL jobs in z/OS, see [What is ISPF?](https://www.ibm.com/docs/en/zos-basic-skills?topic=interfaces-what-is-ispf) and [The ISPF editor](https://www.ibm.com/docs/en/zos-basic-skills?topic=ispf-editor) (IBM documentation). | JCL, Mainframe, z/OS Unix shell |
| Run the JCL SFTP client. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/move-mainframe-files-directly-to-amazon-s3-using-transfer-family.html)For more information about how to check the activity of batch jobs, see [z/OS SDSF User's Guide](https://www.ibm.com/docs/en/zos/2.4.0?topic=sdsf-zos-users-guide) (IBM documentation). | Mainframe, JCL, ISPF |
| Validate the file transfer. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/move-mainframe-files-directly-to-amazon-s3-using-transfer-family.html) | General AWS |
| Automate the JCL SFTP client. | Use job scheduler to automatically trigger the JCL SFTP client.You can use mainframe job schedulers, such as [BMC Control-M](https://www.bmcsoftware.pt/it-solutions/control-m.html) or [CA Workload Automation](https://www.broadcom.com/products/mainframe/workload-automation/ca7), to automate batch jobs for file transfers based on time and other batch job dependencies. | Job scheduler |
## Related resources
+ [How AWS Transfer Family works](https://docs.aws.amazon.com/transfer/latest/userguide/how-aws-transfer-works.html)
# Optimize the performance of your AWS Blu Age modernized application
*Vishal Jaswani, Manish Roy, and Himanshu Sah, Amazon Web Services*
## Summary
Mainframe applications that are modernized with AWS Blu Age require functional and performance equivalence testing before they’re deployed to production. In performance tests, modernized applications can perform more slowly than legacy systems, particularly in complex batch jobs. This disparity exists because mainframe applications are monolithic, whereas modern applications use multitier architectures. This pattern presents optimization techniques to address these performance gaps for applications that are modernized by using [automated refactoring with AWS Blu Age](https://docs.aws.amazon.com/m2/latest/userguide/refactoring-m2.html).
The pattern uses the AWS Blu Age modernization framework with native Java and database tuning capabilities to identify and resolve performance bottlenecks. The pattern describes how you can use profiling and monitoring to identify performance issues with metrics such as SQL execution times, memory utilization, and I/O patterns. It then explains how you can apply targeted optimizations, including database query restructuring, caching, and business logic refinement.
The improvements in batch processing times and system resource utilization help you match mainframe performance levels in your modernized systems. This approach maintains functional equivalence during transition to modern cloud-based architectures.
To use this pattern, set up your system and identify performance hotspots by following the instructions in the [Epics](#optimize-performance-aws-blu-age-modernized-application-epics) section, and apply the optimization techniques that are covered in detail in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section.
## Prerequisites and limitations
**Prerequisites**
+ An AWS Blu Age modernized application
+ A [JProfiler license](https://www.ej-technologies.com/store/jprofiler)
+ Administrative privileges to install database client and profiling tools
+ AWS Blu Age [Level 3 certification](https://bluinsights.aws/certification/)
+ Intermediate-level understanding of the AWS Blu Age framework, generated code structure, and Java programming
**Limitations**
The following optimization capabilities and features are outside the scope of this pattern:
+ Network latency optimization between application tiers
+ Infrastructure-level optimizations through Amazon Elastic Compute Cloud (Amazon EC2) instance types and storage optimization
+ Concurrent user load testing and stress testing
**Product versions**
+ JProfiler version 13.0 or later (we recommend the latest version)
+ pgAdmin version 8.14 or later
## Architecture
This pattern sets up a profiling environment for an AWS Blu Age application by using tools such as JProfiler and pgAdmin. It supports optimization through the DAOManager and SQLExecutionBuilder APIs provided by AWS Blu Age.
The remainder of this section provides detailed information and examples for identifying performance hotspots and optimization strategies for your modernized applications. The steps in the [Epics](#optimize-performance-aws-blu-age-modernized-application-epics) section refer back to this information for further guidance.
**Identifying performance hotspots in modernized mainframe applications**
In modernized mainframe applications, *performance hotspots* are specific areas in the code that cause significant slowdowns or inefficiencies. These hotspots are often caused by the architectural differences between mainframe and modernized applications. To identify these performance bottlenecks and optimize the performance of your modernized application, you can use three techniques: SQL logging, a query `EXPLAIN` plan, and JProfiler analysis.
*Hotspot identification technique: SQL logging*
Modern Java applications, including those that have been modernized by using AWS Blu Age, have built-in capabilities to log SQL queries. You can enable specific loggers in AWS Blu Age projects to track and analyze the SQL statements executed by your application. This technique is particularly useful for identifying inefficient database access patterns, such as excessive individual queries or poorly structured database calls, that could be optimized through batching or query refinement.
To implement SQL logging in your AWS Blu Age modernized application, set the log level to `DEBUG` for SQL statements in the `application.properties` file to capture query execution details:
```
level.org.springframework.beans.factory.support.DefaultListableBeanFactory : WARN
level.com.netfective.bluage.gapwalk.runtime.sort.internal: WARN
level.org.springframework.jdbc.core.StatementCreatorUtils: DEBUG
level.com.netfective.bluage.gapwalk.rt.blu4iv.dao: DEBUG
level.com.fiserv.signature: DEBUG
level.com.netfective.bluage.gapwalk.database.support.central: DEBUG
level.com.netfective.bluage.gapwalk.rt.db.configuration.DatabaseConfiguration: DEBUG
level.com.netfective.bluage.gapwalk.rt.db.DatabaseInteractionLoggerUtils: DEBUG
level.com.netfective.bluage.gapwalk.database.support.AbstractDatabaseSupport: DEBUG
level.com.netfective.bluage.gapwalk.rt: DEBUG
```
Monitor high-frequency and slow-performing queries by using the logged data to identify optimization targets. Focus on queries within batch processes because they typically have the highest performance impact.
*Hotspot identification technique: Query EXPLAIN plan*
This method uses the query planning capabilities of relational database management systems. You can use commands such as `EXPLAIN` in PostgreSQL or MySQL, or `EXPLAIN PLAN` in Oracle, to examine how your database intends to run a given query. The output of these commands provides valuable insights into the query execution strategy, including whether indexes will be used or full table scans will be performed. This information is critical for optimizing query performance, especially in cases where proper indexing can significantly reduce execution time.
Extract the most repetitive SQL queries from the application logs and analyze the execution path of slow-performing queries by using the `EXPLAIN` command that’s specific to your database. Here’s an example for a PostgreSQL database.
Query:
```
SELECT * FROM tenk1 WHERE unique1 < 100;
```
`EXPLAIN` command:
```
EXPLAIN SELECT * FROM tenk1 where unique1 < 100;
```
Output:
```
Bitmap Heap Scan on tenk1 (cost=5.06..224.98 rows=100 width=244)
Recheck Cond: (unique1 < 100)
-> Bitmap Index Scan on tenk1_unique1 (cost=0.00..5.04 rows=100 width=0)
Index Cond: (unique1 < 100)
```
You can interpret the `EXPLAIN` output as follows:
+ Read the `EXPLAIN` plan from the innermost to the outermost (bottom to top) operations.
+ Look for key terms. For example, `Seq Scan` indicates full table scan and `Index Scan` shows index usage.
+ Check cost values: The first number is the startup cost, and the second number is the total cost.
+ See the `rows` value for the estimated number of output rows.
In this example, the query engine uses an index scan to find the matching rows, and then fetches only those rows (`Bitmap Heap Scan`). This is more efficient than scanning the entire table, despite the higher cost of individual row access.
Table scan operations in the output of an `EXPLAIN` plan indicate a missing index. Optimization requires the creation of an appropriate index.
*Hotspot identification technique: JProfiler analysis*
JProfiler is a comprehensive Java profiling tool that helps you resolve performance bottlenecks by identifying slow database calls and CPU-intensive calls. This tool is particularly effective in identifying slow SQL queries and inefficient memory usage.
Example analysis for query:
```
select evt. com.netfective.bluage.gapwalk.rt.blu4iv.dao.Blu4ivTableManager.queryNonTrasactional
```
The JProfiler Hot Spots view provides the following information:
+ **Time** column
+ Shows total execution duration (for example, 329 seconds)
+ Displays percentage of total application time (for example, 58.7%)
+ Helps identify most time-consuming operations
+ **Average Time** column
+ Shows per-execution duration (for example, 2,692 microseconds)
+ Indicates individual operation performance
+ Helps spot slow individual operations
+ **Events** column
+ Shows execution count (for example, 122,387 times)
+ Indicates operation frequency
+ Helps identify frequently called methods
For the example results:
+ High frequency: 122,387 executions indicate potential for optimization
+ Performance concern: 2,692 microseconds for average time suggests inefficiency
+ Critical impact: 58.7% of total time indicates major bottleneck
JProfiler can analyze your application's runtime behavior to reveal hotspots that might not be apparent through static code analysis or SQL logging. These metrics help you identify the operations that need optimization and determine the optimization strategy that would be most effective. For more information about JProfiler features, see the [JProfiler documentation](https://www.ej-technologies.com/resources/jprofiler/help/doc/main/introduction.html).
When you use these three techniques (SQL logging, query `EXPLAIN` plan, and JProfiler) in combination, you can gain a holistic view of your application's performance characteristics. By identifying and addressing the most critical performance hotspots, you can bridge the performance gap between your original mainframe application and your modernized cloud-based system.
After you identify your application’s performance hotspots, you can apply optimization strategies, which are explained in the next section.
**Optimization strategies for mainframe modernization**
This section outlines key strategies for optimizing applications that have been modernized from mainframe systems. It focuses on three strategies: using existing APIs, implementing effective caching, and optimizing business logic.
*Optimization strategy: Using existing APIs*
AWS Blu Age provides several powerful APIs in DAO interfaces that you can use to optimize performance. Two primary interfaces—DAOManager and SQLExecutionBuilder—offer capabilities for enhancing application performance.
**DAOManager**
DAOManager serves as the primary interface for database operations in modernized applications. It offers multiple methods to enhance database operations and improve application performance, particularly for straightforward create, read, update, and delete (CRUD) operations and batch processing.
+ **Use SetMaxResults.** In the DAOManager API, you can use the **SetMaxResults** method to specify the maximum number of records to retrieve in a single database operation. By default, DAOManager retrieves only 10 records at a time, which can lead to multiple database calls when processing large datasets. Use this optimization when your application needs to process a large number of records and is currently making multiple database calls to retrieve them. This is particularly useful in batch processing scenarios where you're iterating through a large dataset. In the following example, the code on the left (before optimization) uses the default data retrieval value of 10 records. The code on the right (after optimization) sets **setMaxResults** to retrieve 100,000 records at a time.
![\[Example of using SetMaxResults to avoid multiple database calls.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/beb9623e-e7a8-45ef-adc6-19a249224b05.png)
**Note**
Choose larger batch sizes carefully and check object size, because this optimization increases the memory footprint.
+ **Replace SetOnGreatorOrEqual with SetOnEqual.** This optimization involves changing the method you use to set the condition for retrieving records. The **SetOnGreatorOrEqual** method retrieves records that are greater than or equal to a specified value, whereas **SetOnEqual** retrieves only records that exactly match the specified value.
Use **SetOnEqual** as illustrated in the following code example, when you know that you need exact matches and you're currently using the **SetOnGreatorOrEqual** method followed by **readNextEqual()**. This optimization reduces unnecessary data retrieval.
![\[Example of using SetOnEqual to retrieve records based on an exact match.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/5ce0dac9-f281-4862-a71f-1614493a83f0.png)
+ **Use batch write and update operations.** You can use batch operations to group multiple write or update operations into a single database transaction. This reduces the number of database calls and can significantly improve performance for operations that involve multiple records.
In the following example, the code on the left performs write operations in a loop, which slows down the application’s performance. You can optimize this code by using a batch write operation: During each iteration of the `WHILE` loop, you add records to a batch until the batch size reaches a predetermined size of 100. You can then flush the batch when it reaches the predetermined batch size, and then flush any remaining records to the database. This is particularly useful in scenarios where you process large datasets that require updates.
![\[Example of grouping multiple operations into a single database transaction.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/e3bd60d4-06f5-4c1c-9cbd-463f6835a1ba.png)
+ **Add indexes.** Adding indexes is a database-level optimization that can significantly improve query performance. An index allows the database to quickly locate rows with a specific column value without scanning the entire table. Use indexing on columns that are frequently used in `WHERE` clauses, `JOIN` conditions, or `ORDER BY` statements. This is particularly important for large tables or when quick data retrieval is crucial.
**SQLExecutionBuilder**
SQLExecutionBuilder is a flexible API that you can use to take control of the SQL queries that will be executed, fetch certain columns only, `INSERT` by using `SELECT`, and use dynamic table names. In the following example, SQLExecutorBuilder uses a custom query that you define.
![\[Example of using SQLExecutorBuilder with a custom query.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/364e9fb1-0cbc-47d0-936d-46fb3b48b608.png)
**Choosing between DAOManager and SQLExecutionBuilder**
The choice between these APIs depends on your specific use case:
+ Use DAOManager when you want AWS Blu Age Runtime to generate the SQL queries instead of writing them yourself.
+ Choose SQLExecutionBuilder when you need to write SQL queries to take advantage of database-specific features or write optimal SQL queries.
*Optimization strategy: Caching*
In modernized applications, implementing effective caching strategies can significantly reduce database calls and improve response times. This helps bridge the performance gap between mainframe and cloud environments.
In AWS Blu Age applications, simple caching implementations use internal data structures such as hash maps or array lists, so you don’t have to set up an external caching solution that requires cost and code restructuring. This approach is particularly effective for data that is accessed frequently but changes infrequently. When you implement caching, consider memory constraints and update patterns to ensure that the cached data remains consistent and provides actual performance benefits.
The key to successful caching is identifying the right data to cache. In the following example, the code on the left always reads data from the table, whereas the code on the right reads data from the table when the local hash map doesn’t have a value for a given key. `cacheMap` is a hash map object that’s created in the context of the program and cleared in the cleanup method of the program context.
Caching with DAOManager:
![\[Example of caching optimizations with DAOManager.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/4efd3d22-c694-4f7d-a543-2bed341d1651.png)
Caching with SQLExecutionBuilder:
![\[Example of caching optimizations with SQLExecutionBuilder.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/c8964804-96eb-4e26-b2bf-8742e62b4c33.png)
*Optimization strategy: Business logic optimization*
Business logic optimization focuses on restructuring automatically generated code from AWS Blu Age to better align with modern architecture capabilities. This becomes necessary when the generated code maintains the same logic structure as the legacy mainframe code, which may not be optimal for modern systems. The goal is to improve performance while maintaining functional equivalence with the original application.
This optimization approach goes beyond simple API tweaks and caching strategies. It involves changes to how the application processes data and interacts with the database. Common optimizations include avoiding unnecessary read operations for simple updates, removing redundant database calls, and restructuring data access patterns to better align with modern application architecture. Here are a few examples:
+ **Updating data directly in the database. **Restructure your business logic by using direct SQL updates instead of multiple DAOManager operations with loops. For example, the following code (left side) makes multiple database calls and uses excessive memory. Specifically, it uses multiple database read and write operations within loops, individual updates instead of batch processing, and unnecessary object creation for each iteration.
The following optimized code (right side) uses a single Direct SQL update operation. Specifically, it uses a single database call instead of multiple calls and doesn’t require loops because all updates are handled in a single statement. This optimization provides better performance and resource utilization, and reduces complexity. It prevents SQL injection, provides better query plan caching, and helps improve security.
![\[Restructuring code by using direct SQL updates instead of DAOManager operations with loops.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/7d0a7879-8db2-4cc5-b41c-ee370b3f22e5.png)
**Note**
Always use parameterized queries to prevent SQL injection and ensure proper transaction management.
+ **Reducing redundant database calls.** Redundant database calls can significantly impact application performance, particularly when they occur within loops. A simple but effective optimization technique is to avoid repeating the same database query multiple times. The following code comparison demonstrates how moving the `retrieve()` database call outside the loop prevents redundant execution of identical queries, which improves efficiency.
![\[alt text not found\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6b42fafd-1535-416d-8abd-1a5f9007ddba/images/da9c15f4-bcf1-4827-b91a-73212fe35cca.png)
+ **Reducing database calls by using the SQL** `JOIN`** clause.** Implement SQLExecutionBuilder to minimize the calls to the database. SQLExecutionBuilder provides more control over SQL generation and is particularly useful for complex queries that DAOManager cannot handle efficiently. For example, the following code uses multiple DAOManager calls:
```
List employees = daoManager.readAll();
for(Employee emp : employees) {
Department dept = deptManager.readById(emp.getDeptId()); // Additional call for each employee
Project proj = projManager.readById(emp.getProjId()); // Another call for each employee
processEmployeeData(emp, dept, proj);
}
```
The optimized code uses a single database call in SQLExecutionBuilder:
```
SQLExecutionBuilder builder = new SQLExecutionBuilder();
builder.append("SELECT e.*, d.name as dept_name, p.name as proj_name");
builder.append("FROM employee e");
builder.append("JOIN department d ON e.dept_id = d.id");
builder.append("JOIN project p ON e.proj_id = p.id");
builder.append("WHERE e.status = ?", "ACTIVE");
List> results = builder.execute(); // Single database call
for(Map result : results) {
processComplexData(result);
}
```
*Using optimization strategies together*
These three strategies work synergistically: APIs provide the tools for efficient data access, caching reduces the need for repeated data retrieval, and business logic optimization ensures that these APIs are used in the most effective way possible. Regular monitoring and adjustment of these optimizations ensure continued performance improvements while maintaining the reliability and functionality of the modernized application. The key to success lies in understanding when and how to apply each strategy based on your application’s characteristics and performance goals.
## Tools
+ [JProfiler](https://www.ej-technologies.com/jprofiler) is a Java profiling tool that’s designed for developers and performance engineers. It analyzes Java applications and helps identify performance bottlenecks, memory leaks, and threading issues. JProfiler offers CPU, memory, and thread profiling as well as database and Java virtual machine (JVM) monitoring to provide insights into application behavior.
**Note**
As an alternative to JProfiler, you can use [Java VisualVM](https://visualvm.github.io/). This is a free, open source performance profiling and monitoring tool for Java applications that offers real-time monitoring of CPU usage, memory consumption, thread management, and garbage collection statistics. Because Java VisualVM is a built-in JDK tool, it is more cost-effective than JProfiler for basic profiling needs.
+ [pgAdmin](https://www.pgadmin.org/) is an open source administration and development tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects. You can use pgAdmin to perform a wide range of tasks, from writing simple SQL queries to developing complex databases. Its features include a syntax highlighting SQL editor, a server-side code editor, a scheduling agent for SQL, shell, and batch tasks, and support for all PostgreSQL features for both novice and experienced PostgreSQL users.
## Best practices
Identifying performance hotspots:
+ Document baseline performance metrics before you start optimizations.
+ Set clear performance improvement targets based on business requirements.
+ When benchmarking, disable verbose logging, because it can affect performance.
+ Set up a performance test suite and run it periodically.
+ Use the latest version of pgAdmin. (Older versions don't support the `EXPLAIN` query plan.)
+ For benchmarking, detach JProfiler after your optimizations are complete because it adds to latency.
+ For benchmarking, make sure to run the server in start mode instead of debug mode, because debug mode adds to latency.
Optimization strategies:
+ Configure **SetMaxResults** values in the `application.yaml` file, to specify right-sized batches according to your system specifications.
+ Configure **SetMaxResults** values based on data volume and memory constraints.
+ Change **SetOnGreatorOrEqual** to **SetOnEqual** only when subsequent calls are `.readNextEqual()`.
+ In batch write or update operations, handle the last batch separately, because it might be smaller than the configured batch size and could be missed by the write or update operation.
Caching:
+ Fields that are introduced for caching in `processImpl`, which mutate with each execution, should always be defined in the context of that `processImpl`. The fields should also be cleared by using the `doReset()` or `cleanUp()` method.
+ When you implement in-memory caching, right-size the cache. Very large caches that are stored in memory can take up all the resources, which might affect the overall performance of your application.
SQLExecutionBuilder:
+ For queries that you are planning to use in SQLExecutionBuilder, use key names such as `PROGRAMNAME_STATEMENTNUMBER`.
+ When you use SQLExecutionBuilder, always check for the `Sqlcod` field. This field contains a value that specifies whether the query was executed correctly or encountered any errors.
+ Use parameterized queries to prevent SQL injection.
Business logic optimization:
+ Maintain functional equivalence when restructuring code, and run regression testing and database comparison for the relevant subset of programs.
+ Maintain profiling snapshots for comparison.
## Epics
### Install JProfiler and pgAdmin
| Task | Description | Skills required |
| --- | --- | --- |
| Install and configure JProfiler. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
| Install and configure pgAdmin. | In this step, you install and configure a DB client to query your database. This pattern uses a PostgreSQL database and pgAdmin as a database client. If you are using another database engine, follow the documentation for the corresponding DB client.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
### Identify hotspots
| Task | Description | Skills required |
| --- | --- | --- |
| Enable SQL query logging in your AWS Blu Age application. | Enable the loggers for SQL query logging in the `application.properties` file of your AWS Blu Age application, as explained in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
| Generate and analyze query `EXPLAIN` plans to identify database performance hotspots. | For details, see the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
| Create a JProfiler snapshot to analyze a slow-performing test case. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
| Analyze the JProfiler snapshot to identify performance bottlenecks. | Follow these steps to analyze the JProfiler snapshot.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html)For more information about using JProfiler, see the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section and the [JProfiler documentation](https://www.ej-technologies.com/jprofiler/docs). | App developer |
### Establish a baseline
| Task | Description | Skills required |
| --- | --- | --- |
| Establish a performance baseline before you implement optimizations. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
### Apply optimization strategies
| Task | Description | Skills required |
| --- | --- | --- |
| Optimize read calls. | Optimize data retrieval by using the DAOManager **SetMaxResults** method. For more information about this approach, see the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer, DAOManager |
| Refactor the business logic to avoid multiple calls to the database. | Reduce database calls by using a SQL `JOIN` clause. For details and examples, see *Business logic optimization* in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer, SQLExecutionBuilder |
| Refactor the code to use caching to reduce the latency of read calls. | For information about this technique, see *Caching* in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
| Rewrite inefficient code that uses multiple DAOManager operations for simple update operations. | For more information about updating data directly in the database, see *Business logic optimization* in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
### Test optimization strategies
| Task | Description | Skills required |
| --- | --- | --- |
| Validate each optimization change iteratively while maintaining functional equivalence. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html)Using baseline metrics as a reference ensures accurate measurement of each optimization's impact while maintaining system reliability. | App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| When you run the modern application, you see an exception with the error `Query_ID not found`. | To resolve this issue:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) |
| You’ve added indexes, but you don’t see any performance improvements. | Follow these steps to ensure that the query engine is using the index:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) |
| You encounter an out-of-memory exception. | Verify that the code releases the memory held by the data structure. |
| Batch write operations result in missing records in the table | Review the code to ensure that an additional write operation is performed when the batch count isn’t zero. |
| SQL logging doesn’t appear in application logs. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) |
## Related resources
+ [Refactoring applications automatically with AWS Blu Age](https://docs.aws.amazon.com/m2/latest/userguide/refactoring-m2.html) (*AWS Mainframe Modernization User Guide*)
+ [pgAdmin documentation](https://www.pgadmin.org/docs/)
+ [JProfiler documentation](https://www.ej-technologies.com/jprofiler/docs)
# Secure and streamline user access in a Db2 federation database on AWS by using trusted contexts
*Sai Parthasaradhi, Amazon Web Services*
## Summary
Many companies are migrating their legacy mainframe workloads to Amazon Web Services (AWS). This migration includes shifting IBM Db2 for z/OS databases to Db2 for Linux, Unix, and Windows (LUW) on Amazon Elastic Compute Cloud (Amazon EC2). During a phased migration from on premises to AWS, users might need to access data in IBM Db2 z/OS and in Db2 LUW on Amazon EC2 until all applications and databases are fully migrated to Db2 LUW. In such remote data-access scenarios, user authentication can be challenging because different platforms use different authentication mechanisms.
This pattern covers how to set up a federation server on Db2 for LUW with Db2 for z/OS as a remote database. The pattern uses a trusted context to propagate a user’s identity from Db2 LUW to Db2 z/OS without re-authenticating on the remote database. For more information about trusted contexts, see the [Additional information](#secure-and-streamline-user-access-in-a-db2-federation-database-on-aws-by-using-trusted-contexts-additional) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A Db2 instance running on an Amazon EC2 instance
+ A remote Db2 for z/OS database running on premises
+ The on-premises network connected to AWS through [AWS Site-to-Site VPN](https://aws.amazon.com/vpn/) or [AWS Direct Connect](https://aws.amazon.com/directconnect/)
## Architecture
**Target architecture**
![\[On-premises mainframe connects through on-premises Db2 server and VPN to the Db2 DB on EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9e04f0fe-bae2-412a-93ac-83da50222017/images/0a384695-7907-4fb8-bb7e-d170dcc114af.png)
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Site-to-Site VPN](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html) helps you pass traffic between instances that you launch on AWS and your own remote network.
**Other tools**
+ [db2cli](https://www.ibm.com/docs/en/db2/11.5?topic=commands-db2cli-db2-interactive-cli) is the Db2 interactive command line interface (CLI) command.
## Epics
### Enable federation on the Db2 LUW database running on AWS
| Task | Description | Skills required |
| --- | --- | --- |
| Enable federation on the DB2 LUW DB. | To enable federation on DB2 LUW, run the following command.
update dbm cfg using federated YES
| DBA |
| Restart the database. | To restart the database, run the following command.
db2stop force; db2start;
| DBA |
### Catalog the remote database
| Task | Description | Skills required |
| --- | --- | --- |
| Catalog the remote Db2 z/OS subsystem. | To catalog the remote Db2 z/OS database on Db2 LUW running on AWS, use the following example command.
catalog TCPIP NODE tcpnode REMOTE mainframehost SERVER mainframeport
| DBA |
| Catalog the remote database. | To catalog the remote database, use the following example command.
catalog db dbnam1 as ndbnam1 at node tcpnode
| DBA |
### Create the remote server definition
| Task | Description | Skills required |
| --- | --- | --- |
| Collect user credentials for the remote Db2 z/OS database. | Before proceeding with the next steps, gather the following information:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/secure-and-streamline-user-access-in-a-db2-federation-database-on-aws-by-using-trusted-contexts.html) | DBA |
| Create the DRDA wrapper. | To create the DRDA wrapper, run the following command.
CREATE WRAPPER DRDA;
| DBA |
| Create the server definition. | To create the server definition, run the following example command.
CREATE SERVER ndbserver TYPE DB2/ZOS VERSION 12 WRAPPER DRDA AUTHORIZATION "dbuser1" PASSWORD "dbpasswd" OPTIONS ( DBNAME 'ndbnam1',FED_PROXY_USER 'ZPROXY' );
In this definition, `FED_PROXY_USER` specifies the proxy user that will be used for establishing trusted connections to the Db2 z/OS database. The authorization user ID and password are required only for creating the remote server object in the Db2 LUW database. They won’t be used later during runtime. | DBA |
### Create user mappings
| Task | Description | Skills required |
| --- | --- | --- |
| Create a user mapping for the proxy user. | To create a user mapping for proxy user, run the following command.
CREATE USER MAPPING FOR ZPROXY SERVER ndbserver OPTIONS (REMOTE_AUTHID 'ZPROXY', REMOTE_PASSWORD 'zproxy');
| DBA |
| Create user mappings for each user on Db2 LUW. | Create user mappings for all the users on the Db2 LUW database on AWS who need to access remote data through the proxy user. To create the user mappings, run the following command.
CREATE USER MAPPING FOR PERSON1 SERVER ndbserver OPTIONS (REMOTE_AUTHID 'USERZID', USE_TRUSTED_CONTEXT 'Y');
The statement specifies that a user on Db2 LUW (`PERSON1`) can establish a trusted connection to the remote Db2 z/OS database (`USE_TRUSTED_CONTEXT 'Y'`). After the connection is established through the proxy user, the user can access the data by using the Db2 z/OS user ID (`REMOTE_AUTHID 'USERZID'`). | DBA |
### Create the trusted context object
| Task | Description | Skills required |
| --- | --- | --- |
| Create the trusted context object. | To create the trusted context object on the remote Db2 z/OS database, use the following example command.
CREATE TRUSTED CONTEXT CTX_LUW_ZOS BASED UPON CONNECTION USING SYSTEM AUTHID ZPROXY ATTRIBUTES ( ADDRESS '10.10.10.10' ) NO DEFAULT ROLE ENABLE WITH USE FOR PUBLIC WITHOUT AUTHENTICATION;
In this definition, `CTX_LUW_ZOS` is an arbitrary name for the trusted context object. The object contains the proxy user ID and the IP address of the server from which the trusted connection must originate. In this example, the server the Db2 LUW database on AWS. You can use the domain name instead of the IP address. The clause `WITH USE FOR PUBLIC WITHOUT AUTHENTICATION` indicates that switching the user ID on a trusted connection is allowed for every user ID. A password doesn't need to be provided. | DBA |
## Related resources
+ [IBM Resource Access Control Facility (RACF)](https://www.ibm.com/products/resource-access-control-facility)
+ [IBM Db2 LUW Federation](https://www.ibm.com/docs/en/db2/11.5?topic=federation)
+ [Trusted contexts](https://www.ibm.com/docs/en/db2-for-zos/13?topic=contexts-trusted)
## Additional information
**Db2 trusted contexts**
A trusted context is a Db2 database object that defines a trust relationship between a federated server and a remote database server. To define a trusted relationship, the trusted context specifies trust attributes. There are three types of trust attributes:
+ The system authorization ID that makes the initial database connection request
+ The IP address or domain name from which the connection is made
+ The encryption setting for data communications between the database server and the database client
A trusted connection is established when all attributes of a connection request match the attributes specified in any trusted context object defined on the server. There are two types of trusted connections: implicit and explicit. After an implicit trusted connection is established, a user inherits a role that isn't available to them outside the scope of that trusted connection definition. After an explicit trusted connection is established, users can be switched on the same physical connection, with or without authentication. In addition, Db2 users can be granted roles that specify privileges that are for use only within the trusted connection. This pattern uses an explicit trusted connection.
*Trusted context in this pattern*
After the pattern is complete, PERSON1 on Db2 LUW accesses remote data from Db2 z/OS by using a federated trusted context. The connection for PERSON1 is established through a proxy user if the connection originates from the IP address or domain name that is specified in the trusted context definition. After the connection is established, PERSON1's corresponding Db2 z/OS user ID is switched without re-authentication, and the user can access the data or objects based on the Db2 privileges set up for that user.
*Benefits of federated trusted contexts*
+ This approach maintains the principle of least privilege by eliminating the use of a common user ID or application ID that would need a superset of all the privileges required by all users.
+ The real identity of the user who performs the transaction on both the federated and remote database is always known and can be audited.
+ Performance improves because the physical connection is being reused across the users without the need for the federated server to re-authenticate.
# Transfer large-scale Db2 z/OS data to Amazon S3 in CSV files
*Bruno Sahinoglu, Abhijit Kshirsagar, and Ivan Schuster, Amazon Web Services*
## Summary
A mainframe is still a system of record in many enterprises, containing a massive amount of data including master data entities with records of current as well as the historical business transactions. It is often siloed and not easily accessed by the distributed systems within the same enterprise. With the emergence of cloud technology and big data democratization, enterprises are interested in using the insights hidden in the mainframe data to develop new business capabilities.
With that objective, enterprises are looking to open their mainframe Db2 data to their Amazon Web Services (AWS) Cloud environment. The business reasons are several and the transfer methods differ from case to case. You might prefer to connect your application directly to the mainframe, or you might prefer to replicate your data in near real time. If the use case is to feed a data warehouse or a data lake, having an up-to-date copy is no longer a concern, and the procedure described in this pattern might suffice, especially if you want to avoid any third-party product licensing costs. Another use case might be the mainframe data transfer for a migration project. In a migration scenario, data is required for performing the functional equivalence testing. The approach described in this post is a cost effective way to transfer the Db2 data to the AWS Cloud environment.
Because Amazon Simple Storage Service (Amazon S3) is one of the most integrated AWS services, you can access the data from there and gather insights directly by using other AWS services such as Amazon Athena, AWS Lambda functions, or Amazon QuickSight . You can also load the data to Amazon Aurora or Amazon DynamoDB by using AWS Glue or AWS Database Migration Service (AWS DMS). With that aim in mind, this describes how to unload Db2 data in CSV files in ASCII format on the mainframe and transfer the files to Amazon S3.
For this purpose, [mainframe scripts](https://github.com/aws-samples/unloaddb2-samples) have been developed to help to generate job control languages (JCLs) to unload and transfer as many Db2 tables as you need.
## Prerequisites and limitations
**Prerequisites**
+ An IBM z/OS operating system user with authorization to run Restructured Extended Executor (REXX) and JCL scripts.
+ Access to z/OS Unix System Services (USS) to generate SSH (Secure Shell) private and public keys.
+ A writable S3 bucket. For more information, see [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) in the Amazon S3 documentation.
+ An AWS Transfer Family SSH File Transfer Protocol (SFTP)-enabled server using **Service managed** as the identity provider and Amazon S3 as the AWS storage service. For more information, see [Create an SFTP-enabled server](https://docs.aws.amazon.com/transfer/latest/userguide/create-server-sftp.html) in the AWS Transfer Family documentation.
**Limitations**
+ This approach isn’t suitable for near real-time or real-time data synchronization.
+ Data can be moved only from Db2 z/OS to Amazon S3, not the other way around.
## Architecture
**Source technology stack**
+ Mainframe running Db2 on z/OS
**Target technology stack**
+ AWS Transfer Family
+ Amazon S3
+ Amazon Athena
+ Amazon QuickSight
+ AWS Glue
+ Amazon Relational Database Service (Amazon RDS)
+ Amazon Aurora
+ Amazon Redshift
**Source and target architecture**
The following diagram shows the process for generating, extracting, and transferring Db2 z/OS data in ASCII CSV format to an S3 bucket.
![\[Data flow from corporate data center to AWS Cloud, showing mainframe extraction and cloud processing steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66e6fa1a-1c7d-4b7a-8404-9ba85e433b24/images/87b13e0d-0be9-4462-bdbf-67342334416c.png)
1. A list of tables is selected for data migration from the Db2 catalog.
1. The list is used to drive the generation of unload jobs with the numeric and data columns in the external format.
1. The data is then transferred over to Amazon S3 by using AWS Transfer Family.
1. An AWS Glue extract, transform, and load (ETL) job can transform the data and load it to a processed bucket in the specified format, or AWS Glue can feed the data directly into the database.
1. Amazon Athena and Amazon QuickSight can be used to query and render the data to drive analytics.
The following diagram shows a logical flow of the entire process.
![\[Flowchart showing JCL process with TABNAME, REXXEXEC, and JCL decks steps, including inputs and outputs.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66e6fa1a-1c7d-4b7a-8404-9ba85e433b24/images/d72f2572-10c9-43f9-b6c9-7e57c9a69d52.png)
1. The first JCL, called TABNAME, will use the Db2 utility DSNTIAUL to extract and generate the list of tables that you plan to unload from Db2. To choose your tables, you must manually adapt the SQL input to select and add filter criteria to include one or more Db2 schemas.
1. The second JCL, called REXXEXEC, will use the a JCL skeleton and the REXX program that is provided to process the Table list created by the JCL TABNAME and generate one JCL per table name. Each JCL will contain one step for unloading the table and another step for sending the file to the S3 bucket by using the SFTP protocol.
1. The last step consists of running the JCL to unload the table and transferring the file to AWS. The entire process can be automated using a scheduler on premises or on AWS.
## Tools
**AWS services**
+ [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is an interactive query service that helps you analyze data directly in Amazon Simple Storage Service (Amazon S3) by using standard SQL.
+ [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL.
+ [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) is a fully managed extract, transform, and load (ETL) service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams.
+ [Amazon QuickSight](https://docs.aws.amazon.com/quicksight/latest/user/welcome.html) is a cloud-scale business intelligence (BI) service that helps you visualize, analyze, and report your data in a single dashboard.
+ [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html) is a managed petabyte-scale data warehouse service in the AWS Cloud.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Transfer Family](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-family.html) is a secure transfer service that enables you to transfer files into and out of AWS storage services.
**Mainframe tools**
+ [SSH File Transfer Protocol (SFTP)](https://www.ssh.com/academy/ssh/sftp-ssh-file-transfer-protocol) is a secure file transfer protocol that allows remote login to and file transfer between servers. SSH provides security by encrypting all traffic.
+ [DSNTIAUL](https://www.ibm.com/docs/en/db2-for-zos/11?topic=dpasp-dsntiaul-sample-program) is a sample program provided by IBM for unloading data.
+ [DSNUTILB](https://www.ibm.com/docs/en/db2-for-zos/11?topic=sharing-recommendations-utilities-in-coexistence) is a utilities batch program provided by IBM for unloading data with different options from DSNTIAUL.
+ [z/OS OpenSSH](https://www.ibm.com/docs/en/zos/2.4.0?topic=zbed-zos-openssh) is a port of Open Source Software SSH running on the Unix System Service under the IBM operating system z/OS. SSH is a secure, encrypted connection program between two computers running on a TCP/IP network. It provides multiple utilities, including ssh-keygen.
+ [REXX (Restructured Extended Executor)](https://www.ibm.com/docs/en/zos/2.1.0?topic=guide-learning-rexx-language) script is used to automate JCL generation with the Db2 Unload and SFTP steps.
**Code**
The code for this pattern is available in the GitHub [unloaddb2](https://github.com/aws-samples/unloaddb2-samples) repository.
## Best practices
For the first unload, the generated JCLs should unload the entire table data.
After the first full unload, perform incremental unloads for better performance and cost savings. pdate the SQL query in the template JCL deck to accommodate any changes to the unload process.
You can convert the schema manually or by using a script on Lambda with the Db2 SYSPUNCH as an input. For an industrial process, [AWS Schema Conversion Tool (SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.DB2zOS.html) is the preferred option.
Finally, use a mainframe-based scheduler or a scheduler on AWS with an agent on the mainframe to help manage and automate the entire process.
## Epics
### Set up the S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create the S3 bucket. | For instructions, see [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). | General AWS |
### Set up the Transfer Family server
| Task | Description | Skills required |
| --- | --- | --- |
| Create an SFTP-enabled server. | To open and create an SFTP server on the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/), do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html) | General AWS |
| Create an IAM role for Transfer Family. | To create an AWS Identity and Access Management (IAM) role for Transfer Family to access Amazon S3, follow the instructions in [Create an IAM role and policy](https://docs.aws.amazon.com/transfer/latest/userguide/requirements-roles.html). | AWS administrator |
| Add an Amazon S3 service-managed user. | To add the Amazon S3 service-managed user, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/transfer/latest/userguide/service-managed-users.html#add-s3-user), and use your mainframe user ID. | General AWS |
### Secure the communication protocol
| Task | Description | Skills required |
| --- | --- | --- |
| Create the SSH key. | Under your mainframe USS environment, run the following command.
ssh-keygen -t rsa
When prompted for a passphrase, keep it empty. | Mainframe developer |
| Give the right authorization levels to the SSH folder and key files. | By default, the public and private keys will be stored in the user directory `/u/home/username/.ssh`.You must give the authorization 644 to the key files and 700 to the folder.
chmod 644 .ssh/id_rsa chmod 700 .ssh
| Mainframe developer |
| Copy the public key content to your Amazon S3 service-managed user. | To copy the USS-generated public key content, open the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/).[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html) | Mainframe developer |
### Generate the JCLs
| Task | Description | Skills required |
| --- | --- | --- |
| Generate the in-scope Db2 table list. | Provide input SQL to create a list of the tables that are scoped for data migration. This step requires you to specify selection criteria quering the Db2 catalog table SYSIBM.SYSTABLES using a SQL where clause. Filters can be customized to include a specific schema or table names that start with a particular prefix or based on a timestamp for incremental unload. Output is captured in a physical sequential (PS) dataset on the mainframe. This dataset will act as input for the next phase of JCL generation.Before you use the JCL TABNAME (You can rename it if necessary), make the following changes:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**Db2 table list extraction job**
//* //* UNLOAD ALL THE TABLE NAMES FOR A PARTICULAR SCHEMA //* //STEP01 EXEC PGM=IEFBR14 //* //DD1 DD DISP=(MOD,DELETE,DELETE), // UNIT=SYSDA, // SPACE=(1000,(1,1)), // DSN=.DSN81210.TABLIST //* //DD2 DD DISP=(MOD,DELETE,DELETE), // UNIT=SYSDA, // SPACE=(1000,(1,1)), // DSN=.DSN81210.SYSPUNCH //* //UNLOAD EXEC PGM=IKJEFT01,DYNAMNBR=20 //SYSTSPRT DD SYSOUT=* //STEPLIB DD DISP=SHR,DSN=DSNC10.DBCG.SDSNEXIT // DD DISP=SHR,DSN=DSNC10.SDSNLOAD // DD DISP=SHR,DSN=CEE.SCEERUN // DD DISP=SHR,DSN=DSNC10.DBCG.RUNLIB.LOAD //SYSTSIN DD * DSN SYSTEM(DBCG) RUN PROGRAM(DSNTIAUL) PLAN(DSNTIB12) PARMS('SQL') - LIB('DSNC10.DBCG.RUNLIB.LOAD') END //SYSPRINT DD SYSOUT=* //* //SYSUDUMP DD SYSOUT=* //* //SYSREC00 DD DISP=(NEW,CATLG,DELETE), // UNIT=SYSDA,SPACE=(32760,(1000,500)), // DSN=.DSN81210.TABLIST //* //SYSPUNCH DD DISP=(NEW,CATLG,DELETE), // UNIT=SYSDA,SPACE=(32760,(1000,500)), // VOL=SER=SCR03,RECFM=FB,LRECL=120,BLKSIZE=12 // DSN=.DSN81210.SYSPUNCH //* //SYSIN DD * SELECT CHAR(CREATOR), CHAR(NAME) FROM SYSIBM.SYSTABLES WHERE OWNER = '' AND NAME LIKE '%' AND TYPE = 'T'; /*
| Mainframe developer |
| Modify the JCL templates. | The JCL templates that are provided with this pattern contain a generic job card and library names. However, most mainframe sites will have their own naming standards for dataset names, library names, and job cards. For example, a specific job class might be required to run Db2 jobs. The Job Entry Subsytem implementations JES2 and JES3 can impose additional changes. Standard load libraries might have a different first qualifier than `SYS1`, which is IBM default. Therefore, customize the templates to account for your site-specific standards before you run them.Make the following changes in the skeleton JCL UNLDSKEL:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**Unload and SFTP JCL skeleton**
| Mainframe developer |
| Generate the Mass Unload JCL. | This step involves running a REXX script under an ISPF environment by using JCL. Provide the list of in-scope tables created on the first step as input for mass JCL generation against the `TABLIST DD` name. The JCL will generate one new JCL per table name in a user-specified partitioned dataset specified against the `ISPFILE DD` name. Allocate this library beforehand. Each new JCL will have two steps: one step to unload the Db2 table into a file, and one step to send the file to the S3 bucket.Make the following changes in the JCL REXXEXEC (you can change the name):[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**Mass JCL generation job**
//RUNREXX JOB (CREATEJCL),'RUNS ISPF TABLIST',CLASS=A,MSGCLASS=A, // TIME=1440,NOTIFY=&SYSUID //* Most of the values required can be updated to your site specific //* values using the command 'TSO ISRDDN' in your ISPF session. //* Update all the lines tagged with //update marker to desired //* site specific values. //ISPF EXEC PGM=IKJEFT01,REGION=2048K,DYNAMNBR=25 //SYSPROC DD DISP=SHR,DSN=USER.Z23D.CLIST //SYSEXEC DD DISP=SHR,DSN=.TEST.REXXLIB //ISPPLIB DD DISP=SHR,DSN=ISP.SISPPENU //ISPSLIB DD DISP=SHR,DSN=ISP.SISPSENU // DD DISP=SHR,DSN=.TEST.ISPSLIB //ISPMLIB DD DSN=ISP.SISPMENU,DISP=SHR //ISPTLIB DD DDNAME=ISPTABL // DD DSN=ISP.SISPTENU,DISP=SHR //ISPTABL DD LIKE=ISP.SISPTENU,UNIT=VIO //ISPPROF DD LIKE=ISP.SISPTENU,UNIT=VIO //ISPLOG DD SYSOUT=*,RECFM=VA,LRECL=125 //SYSPRINT DD SYSOUT=* //SYSTSPRT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSDBOUT DD SYSOUT=* //SYSTSPRT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSDBOUT DD SYSOUT=* //SYSHELP DD DSN=SYS1.HELP,DISP=SHR //SYSOUT DD SYSOUT=* //* Input list of tablenames //TABLIST DD DISP=SHR,DSN=.DSN81210.TABLIST //* Output pds //ISPFILE DD DISP=SHR,DSN=.TEST.JOBGEN //SYSTSIN DD * ISPSTART CMD(ZSTEPS ) /*
Before you use the REXX script, make the following changes:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**ZSTEPS REXX script**
/*REXX - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ /* 10/27/2021 - added new parms to accommodate ftp */ Trace "o" parse arg usrpfx ftpuser ftpsite Say "Start" Say "Ftpuser: " ftpuser "Ftpsite:" ftpsite Say "Reading table name list" "EXECIO * DISKR TABLIST (STEM LINE. FINIS" DO I = 1 TO LINE.0 Say I suffix = I Say LINE.i Parse var LINE.i schema table rest tabname = schema !! "." !! table Say tabname tempjob= "LOD" !! RIGHT("0000" !! i, 5) jobname=tempjob Say tempjob ADDRESS ISPEXEC "FTOPEN " ADDRESS ISPEXEC "FTINCL UNLDSKEL" /* member will be saved in ISPDSN library allocated in JCL */ ADDRESS ISPEXEC "FTCLOSE NAME("tempjob")" END
| Mainframe developer |
### Run the JCLs
| Task | Description | Skills required |
| --- | --- | --- |
| Perform the Db2 Unload step. | After the JCL generation, you will have as many JCLs as you have tables that need to be unloaded.This story uses a JCL-generated example to explain the structure and the most important steps.No action is required on your part. The following information is for reference only. If your intention is to submit the JCLs that you have generated in the previous step, skip to the *Submit the LODnnnnn JCLs* task.When unloading Db2 data using a JCL with the IBM provided DSNUTILB Db2 utility, you must make sure that the unloaded data does not contain compressed numeric data. To accomplish this, use the DSNUTILB `DELIMITED` parameter.The `DELIMITED` parameter supports unloading the data in CSV format by adding a character as the delimiter and double quotation marks for the text field, removing the padding in the VARCHAR column, and converting all the numeric fields into EXTERNAL FORMAT, including the DATE fields.The following example shows what the unload step in the generated JCL looks like, using the comma character as a delimiter.
| Mainframe developer, System engineer |
| Perform the SFTP step. | To use the SFTP protocol from a JCL, use the BPXBATCH utility. The SFTP utility can’t access the MVS datasets directly. You can use the copy command (`cp`) to copy the sequential file `&USRPFX..DB2.UNLOAD.&JOBNAME` to the USS directory, where it becomes `&TABNAME..csv`.Run the `sftp` command using the private key (`id_rsa`) and using the RACF user ID as the user name to connect to the AWS Transfer Family IP address.
| Mainframe developer, System engineer |
| Submit the LODnnnnn JCLs. | The prior JCL has generated all LODnnnnn JCL tables that need to be unloaded, transformed into CSV, and transferred to the S3 bucket.Run the `submit` command on all the JCLs that have been generated. | Mainframe developer, System engineer |
## Related resources
For more information about the different tools and solutions used in this document, see the following:
+ [z/OS OpenSSH User’s Guide](https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc276806/$file/foto100_v2r4.pdf)
+ [Db2 z/OS – Sample UNLOAD control statements](https://www.ibm.com/docs/en/db2-for-zos/11?topic=unload-sample-control-statements)
+ [Db2 z/OS – Unloading delimited files](https://www.ibm.com/docs/en/db2-for-zos/11?topic=unload-unloading-delimited-files)
+ [Transfer Family – Create an SFTP-enabled server](https://docs.aws.amazon.com/transfer/latest/userguide/create-server-sftp.html)
+ [Transfer Family – Working with service-managed users](https://docs.aws.amazon.com/transfer/latest/userguide/service-managed-users.html)
## Additional information
After you have your Db2 data on Amazon S3, you have many ways to develop new insights. Because Amazon S3 integrates with AWS data analytics services, you can freely consume or expose this data on the distributed side. For example, you can do the following:
+ Build a [data lake on Amazon S3](https://aws.amazon.com/products/storage/data-lake-storage/), and extract valuable insights by using query-in-place, analytics, and machine learning tools without moving the data.
+ Initiate a [Lambda function](https://aws.amazon.com/lambda/) by setting up a post-upload processing workflow that is integrated with AWS Transfer Family.
+ Develop new microservices for accessing the data in Amazon S3 or in [fully managed database](https://aws.amazon.com/free/database/?trk=ps_a134p000007CdNEAA0&trkCampaign=acq_paid_search_brand&sc_channel=PS&sc_campaign=acquisition_FR&sc_publisher=Google&sc_category=Database&sc_country=FR&sc_geo=EMEA&sc_outcome=acq&sc_detail=amazon%20relational%20database%20service&sc_content=Relational%20Database_e&sc_matchtype=e&sc_segment=548727697660&sc_medium=ACQ-P|PS-GO|Brand|Desktop|SU|Database|Solution|FR|EN|Text&s_kwcid=AL!4422!3!548727697660!e!!g!!amazon%20relational%20database%20service&ef_id=CjwKCAjwzt6LBhBeEiwAbPGOgcGbQIl1-QsbHfWTgMZSSHEXzSG377R9ZyK3tCcbnHuT45L230FufxoCeEkQAvD_BwE:G:s&s_kwcid=AL!4422!3!548727697660!e!!g!!amazon%20relational%20database%20service) by using [AWS Glue](https://aws.amazon.com/glue/), which is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development.
In a migration use case, because you can transfer any data from the mainframe to S3, you can do the following:
+ Retire physical infrastructure, and create a cost-effective data archival strategy with Amazon S3 Glacier and S3 Glacier Deep Archive.
+ Build scalable, durable, and secure backup and restore solutions with Amazon S3 and other AWS services, such as S3 Glacier and Amazon Elastic File System (Amazon EFS), to augment or replace existing on-premises capabilities.
# Transform Easytrieve to modern languages by using AWS Transform custom
*Shubham Roy, Subramanyam Malisetty, and Harshitha Shashidhar, Amazon Web Services*
## Summary
This pattern provides prescriptive guidance for faster and lower-risk transformation of mainframe Broadcom [Easytrieve Report Generator](https://techdocs.broadcom.com/us/en/ca-mainframe-software/devops/ca-easytrieve-report-generator/11-6.html) (EZT) workloads using [AWS Transform custom](https://aws.amazon.com/transform/custom/) language-to-language transformation. It addresses the challenges of modernizing niche and proprietary mainframe EZT workloads that are commonly used for batch data processing and report generation. The pattern replaces expensive, lengthy, and error-prone migration approaches that rely on proprietary tooling and rare mainframe expertise with an agentic AI automated solution you create on AWS Transform.
This pattern provides a ready-to -use custom transformation definition for EZT transformation. The definition uses multiple transformation inputs:
+ EZT business rules extracted using [AWS Transform for mainframe](https://aws.amazon.com/transform/mainframe/)
+ EZT programming reference documentation
+ EZT source code
+ Mainframe input and output datasets
AWS Transform custom uses these inputs to generate functionally equivalent applications in modern target languages, such as Java or Python.
The transformation process uses intelligent test execution, automated debugging, and iterative fix capabilities to validate functional equivalence against expected outputs. It also supports continual learning, enabling the custom transformation definition to improve accuracy and consistency across successive transformations. Using this pattern, organizations can reduce migration effort and risk, address niche mainframe technical debt, and modernize EZT workloads on AWS to improve agility, reliability, security, and innovation.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A mainframe EZT workload with input and output data
**Limitations**
*Scope limitations *
+ **Language support** – Only EZT-to-Java transformation is supported for this specific transformation pattern.
+ **Out of scope** – Transformation of other mainframe programming languages requires a new custom transformation definition in AWS Transform custom.
*Process limitations *
+ **Validation dependency** – Without baseline output data the transformation cannot be validated.
+ **Proprietary logic** – Highly specific, custom-developed utilities require additional user documentation and reference materials in order to be correctly interpreted by the AI agent.
*Technical limitations *
+ **Service limits** – For AWS Transform custom service limits and quotas see [AWS Transform User Guide - Quotas](https://docs.aws.amazon.com/transform/latest/userguide/transform-limits.html) and the [AWS General Reference - Transform Quotas](https://docs.aws.amazon.com/general/latest/gr/aws-transform.html).
**Product versions**
+ AWS Transform CLI – Latest version
+ Node.js – version 20 or later
+ Git – Latest version
+ Target environment
+ Java – version 17 or later
+ Spring Boot – version 3.x is the primary target for refactored applications
+ Maven – version 3.6 or later
## Architecture
**Source technology stack**
+ **Operating system** – IBM z/OS
+ **Programming language** – Easytrieve, Job control language (JCL)
+ **Database** – IBM DB2 for z/OS, Virtual Storage Access Method (VSAM), Mainframe flat files
**Target technology stack**
+ **Operating system** – Amazon Linux
+ **Compute** – Amazon Elastic Compute Cloud (Amazon EC2)
+ **Programming language** – Java
+ **Database** Amazon Relational Database Service (Amazon RDS)
**Target architecture**
![\[target architecture diagram for using AWS Transform custom to transform EZT to modern code.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/71f15422-42cb-4c7e-94fa-051a4f130445/images/eb89eed0-dd55-485c-a433-9869162eaad9.png)
**Workflow**
This solution uses an AWS Transform custom language-to-language migration transformation pattern to modernize mainframe Easytrieve (EZT) applications to Java through a four-step automated workflow.
*Step 1 – Provide your legacy code to AWS Transform for Mainframe, which:*
+ Analyzes the code
+ Extracts the high-level business logic
+ Extracts the detailed business logic.
*Step 2 – Create a folder with the required inputs:*
+ EZT business rules extracted using AWS Transform for mainframe
+ EZT programming reference documentation
+ EZT source code
+ Mainframe input and output datasets
*Step 3 – Create and run a custom transformation definition*
1. Use the AWS Transform CLI to describe transformation objectives in natural language. AWS Transform custom analyzes the BRE, source code, and EZT programming guides to generate a custom transformation definition for developer review and approval.
1. Then, invoke the AWS Transform CLI with the project source code. AWS Transform custom creates transformation plans, converts EZT to Java upon approval, generates supporting files, builds the executable JAR, and validates exit criteria.
1. Use the validation agent to test the functional equivalence against mainframe output. The Self-Debugger Agent autonomously fixes issues. Final deliverables include validated Java code and HTML validation reports.
**Automation and scale**
+ Agentic AI multi-mode execution architecture – AWS Transform custom leverages agentic AI with 3 execution modes (conversational, interactive, full automation) to automate complex transformation tasks including code analysis, refactoring, transformation planning and testing.
+ Adaptive learning feedback system – The platform implements continuous learning mechanisms through code sample analysis, documentation parsing, and developer feedback integration with versioned transformation definitions.
+ Concurrent application processing architecture – The system enables distributed parallel execution of multiple application transformation operations simultaneously across scalable infrastructure.
## Tools
**AWS services **
+ [AWS Transform custom](https://docs.aws.amazon.com/transform/latest/userguide/custom.html) is an agentic AI service is used to transform legacy EZT applications into modern programming languages.
+ [AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/what-is-service.html) uses agentic AI to help you accelerate the modernization of legacy workloads, such as .NET, mainframe, and VMware workloads.
+ [AWS Transform for mainframe ](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe.html)is used to analyze legacy EZT applications to extract embedded business logic and generate comprehensive business rule documentation, including logic summaries, acronym definitions, and structured knowledge bases. These serve as input data for AWS Transform custom.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. Amazon S3 serves as the primary storage service for AWS Transform custom for storing transformation definitions, code repositories, and processing results.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. IAM provides the security framework for AWS Transform custom, managing permissions and access control for transformation operations.
**Other tools**
+ [AWS Transform CLI](https://docs.aws.amazon.com/transform/latest/userguide/custom-command-reference.html) is the command-line interface for AWS Transform custom, enabling developers to define, execute, and manage custom code transformations through natural language conversations and automated execution modes. AWS Transform custom supports both interactive sessions (atx custom def exec) and autonomous transformations for scalable modernization of codebases.
+ [Git](https://git-scm.com/doc) version control system used for branch protection, change tracking, and rollback capabilities during automated fix application.
+ [Java](https://www.java.com/en/) is the programming language and development environment used in this pattern.
**Code repository**
The code for this pattern is available in [Easytrieve to Modern Languages Transformation with AWS Transform Custom](https://github.com/aws-samples/sample-mainframe-easytrieve-transform?tab=readme-ov-file#easytrieve-to-modern-languages-transformation-with-aws-transform-custom) on GitHub.
## Best practices
+ Establish standardized project structure – Create a four-folder structure (source-code, bre-doc, input-data, output-data), validate completeness, and document contents before transformation.
+ Use baseline files for validation – Use production baseline input files, perform byte-by-byte comparison with baseline output, accept zero tolerance for deviations.
+ Use all available reference documents – To increase accuracy of transformation provide all available reference documents such as business requirements and coding checklists.
+ Provide input for quality improvement – AWS Transform custom automatically extracts learnings from transformation executions (developer feedback, code issues) and creates knowledge items for them. after each successful transformation review knowledge items and approve the one that you would like to be used in future executions. This improves the quality of future transformations.
## Epics
### Generate a business rule extract (BRE)
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS Transform for mainframe. | Set up the environment and required AWS Identity and Access Management (IAM) permissions to support mainframe modernization workflows. For more information see [Transformation of mainframe applications](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html) in AWS documentation. | App developer |
| Generate Business Rule Extract (BRE) documentation. | Extract business logic from source EZT or COBOL code to generate functional documentation. For instructions on how to initiate the extraction process and review the output, see [Extract business logic](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-extract-business-logic) in the AWS Transform documentation. | App developer |
### Set up AWS Transform custom
| Task | Description | Skills required |
| --- | --- | --- |
| Provision the infrastructure for AWS Transform custom. | Deploy the production-ready infrastructure required to host a secure transformation environment. This includes a private Amazon EC2 instance configured with the necessary tools, IAM permissions, and network settings for converting Easytrieve code. To provision the environment using infrastructure as code (IaC), follow the deployment instructions in the [Easytrieve to Modern Languages Transformation with AWS Transform Custom](https://github.com/aws-samples/sample-mainframe-easytrieve-transform) GitHub repository. | App developer, AWS administrator |
| Prepare input materials for transformation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
### Create a custom transformation definition
| Task | Description | Skills required |
| --- | --- | --- |
| Create transformation definition. | Follow these steps to create the custom transformation definition for EZT to Java transformation with functional validation.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
| Publish transformation definition. | After review and validation of the transformation definition you can publish it to the AWS Transform custom registry with a natural language prompt, providing a definition name such as *Easytrieve-to-Java-Migration*. | App developer |
### Prepare baseline data for validation.
| Task | Description | Skills required |
| --- | --- | --- |
| Review the transformation validation summary. | Before executing the AWS Transform custom transformation, validate that the `input-data` folder contains the required data files captured before execution of the mainframe batch job. After the mainframe batch job execution, ensure that the `output-data` folder captures the resulting files. All files are in Sequential/Text/DB2 format using EBCDIC encoding based on execution requirements.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
| Run the custom transformation job. | Execute the AWS Transform CLI command, choosing the non-interactive or the interactive option:
| App developer |
| Enable knowledge items for continuous learning. | Improve future transformation accuracy by promoting suggested knowledge items to your persistent configuration. After a transformation, the agent stores identified patterns and mapping rules in your local session directory. To review and apply these learned items, run these commands on your Amazon EC2 instance:
# List all knowledge items for a specific transformation definition atx custom def list-ki -n
# Retrieve the details of a specific knowledge item atx custom def get-ki -n --id
# Update the status of a knowledge item (ENABLED or DISABLED) atx custom def update-ki-status -n --id --status ENABLED
# Update the knowledge item configuration to enable auto-approval atx custom def update-ki-config -n --auto-enabled TRUE
| App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| *Input and output path configuration*Input files are not being read, or output files are not being written correctly. | Specify the complete directory path where input files are stored and clearly indicate the location where output should be written. Ensure proper access permissions are configured for these directories. Best practices include using absolute paths rather than relative paths to avoid ambiguity and verifying that all specified paths exist with appropriate read/write permissions. |
| *Resuming interrupted executions*Execution was interrupted or needs to be continued from where stopped | You can resume execution from where you left off by providing the conversation ID in the CLI command.Find the conversation ID in the logs of your previous execution attempt. |
| *Resolving memory constraints*Out of memory error occurs during execution. | You can ask AWS Transform to share the current in-memory JVM size and then increase the memory allocation based on this information. This adjustment helps accommodate larger processing requirements.Consider breaking large jobs into smaller batches if memory constraints persist after adjustments. |
| *Addressing output file discrepancies*Output files don't match expectations, and AWS Transform indicates no further changes are possible. | Provide specific feedback and technical reasons explaining why the current output is incorrect. Include additional technical or business documentation to support your requirements. This detailed context helps AWS Transform correct the code to generate the proper output files. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) |
## Related resources
+ [AWS Transform custom documentation](https://docs.aws.amazon.com/transform/latest/userguide/custom.html)
+ [Easytrieve Report Generator 11.6](https://techdocs.broadcom.com/us/en/ca-mainframe-software/devops/ca-easytrieve-report-generator/11-6/getting-started.html)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/71f15422-42cb-4c7e-94fa-051a4f130445/attachments/attachment.zip)
# More patterns
**Topics**
+ [Deploy the Security Automations for AWS WAF solution by using Terraform](deploy-the-security-automations-for-aws-waf-solution-by-using-terraform.md)
+ [Replicate mainframe databases to AWS by using Precisely Connect](replicate-mainframe-databases-to-aws-by-using-precisely-connect.md)