$ dgmgrl RDS_DATAGUARD@RDS_CUSTOM_ORCL_D**CDB**
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Fri Sep 30 22:44:49 2022
Version 19.10.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "ORCL_D"
Connected as SYSDG.
DGMGRL>
DGMGRL> show database orcl_d
Database - orcl_d
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 11.00 KByte/s
Instance(s):
ORCL
SUCCESS
DGMGRL>
-bash-4.2$ dgmgrl C##RDS_DATAGUARD@RDS_CUSTOM_RDSCDB_B| DBA | | Change the log transport setting by connecting to DGMGRL from the primary node. | Change the log transport mode to `FastSync`, corresponding to the redo transport setting `SYNC+NOAFFIRM`. To ensure that you have valid settings after the role switch, change it for both the primary database and the standby database.**Non-CDB**
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Jan 11 20:24:11 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "RDSCDB_B"
Connected as SYSDG.
DGMGRL>
DGMGRL> show database rdscdb_b
Database - rdscdb_b
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 2.00 KByte/s
Real Time Query: OFF
Instance(s):
RDSCDB
Database Status:
SUCCESS
DGMGRL>
DGMGRL>**CDB**
DGMGRL> edit database orcl_d set property logxptmode=fastsync;
Property "logxptmode" updated
DGMGRL> show database orcl_d LogXptMode;
LogXptMode = 'fastsync'
DGMGRL> edit database orcl_a set property logxptmode=fastsync;
Property "logxptmode" updated
DGMGRL> show database orcl_a logxptmode;
LogXptMode = 'fastsync'
DGMGRL>
DGMGRL> edit database rdscdb_b set property logxptmode=fastsync;DGMGRL> edit database rdscdb_b set property logxptmode=fastsync;| DBA | | Change the protection mode to MaxAvailability. | Change the protection mode to `MaxAvailability` by connecting to `DGMGRL` from the primary node.**Non-CDB**
Property "logxptmode" updated
DGMGRL> show database rdscdb_b LogXptMode;
LogXptMode = 'fastsync'
DGMGRL> edit database rdscdb_a set property logxptmode=fastsync;
Property "logxptmode" updated
DGMGRL> show database rdscdb_a logxptmode;
LogXptMode = 'fastsync'
DGMGRL>
DGMGRL> edit configuration set protection mode as maxavailability;**CDB**
Succeeded.
DGMGRL> show configuration;
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_a - Primary database
orcl_d - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 38 seconds ago)
DGMGRL>
DGMGRL> show configuration| DBA | ### Change the replica status from mount to read-only and enable redo apply | Task | Description | Skills required | | --- | --- | --- | | Stop redo apply for the standby database. | The read replica is created in `MOUNT` mode by default. To open it in read-only mode, you first need to turn off redo apply by connecting to `DGMGRL` from the primary or standby node.**Non-CDB**
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_a - Primary database
rdscdb_b - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 57 seconds ago)
DGMGRL>
DGMGRL> show database orcl_dDGMGRL> show database orcl_d**CDB**
Database - orcl_d
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 11.00 KByte/s
Real Time Query: OFF
Instance(s):
ORCL
Database Status:
SUCCESS
DGMGRL> edit database orcl_d set state=apply-off;
Succeeded.
DGMGRL> show database orcl_d
Database - orcl_d
Role: PHYSICAL STANDBY
Intended State: APPLY-OFF
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 42 seconds (computed 1 second ago)
Average Apply Rate: (unknown)
Real Time Query: OFF
Instance(s):
ORCL
Database Status:
SUCCESS
DGMGRL>
DGMGRL> show configurationDGMGRL> show configuration| DBA | | Open the read replica instance in read-only mode. | Connect to the standby database by using the TNS entry, and open it in read-only mode by connecting to it from the primary or standby node.**Non-CDB**
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_a - Primary database
rdscdb_b - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 57 seconds ago)
DGMGRL> show database rdscdb_b;
Database - rdscdb_b
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 2.00 KByte/s
Real Time Query: OFF
Instance(s):
RDSCDB
Database Status:
SUCCESS
DGMGRL> edit database rdscdb_b set state=apply-off;
Succeeded.
DGMGRL> show database rdscdb_b;
Database - rdscdb_b
Role: PHYSICAL STANDBY
Intended State: APPLY-OFF
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: (unknown)
Real Time Query: OFF
Instance(s):
RDSCDB
Database Status:
SUCCESS
$ sqlplus RDS_DATAGUARD@RDS_CUSTOM_ORCL_D as sysdg**CDB**
-bash-4.2$ sqlplus RDS_DATAGUARD@RDS_CUSTOM_ORCL_D as sysdg
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Sep 30 23:00:14 2022
Version 19.10.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Enter password:
Last Successful login time: Fri Sep 30 2022 22:48:27 +00:00
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.10.0.0.0
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED
SQL> alter database open read only;
Database altered.
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY
SQL>
-bash-4.2$ sqlplus C##RDS_DATAGUARD@RDS_CUSTOM_RDSCDB_B as sysdg| DBA | | Activate redo apply on the read replica instance. | Activate redo apply on the read replica instance by using `DGMGR`L from the primary or standby node.**Non-CDB**
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 11 21:14:07 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2022, Oracle. All rights reserved.
Enter password:
Last Successful login time: Wed Jan 11 2023 21:12:05 +00:00
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
SQL> select name,open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
RDSCDB MOUNTED
SQL> alter database open read only;
Database altered.
SQL> select name,open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
RDSCDB READ ONLY
SQL>
$ dgmgrl RDS_DATAGUARD@RDS_CUSTOM_ORCL_D**CDB**
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Fri Sep 30 23:02:16 2022
Version 19.10.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "ORCL_D"
Connected as SYSDG.
DGMGRL>
edit database orcl_d set state=apply-on;
DGMGRL> edit database orcl_d set state=apply-on;
Succeeded.
DGMGRL> show database orcl_d
Database - orcl_d
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 496.00 KByte/s
Real Time Query: ON
Instance(s):
ORCL
Database Status:
SUCCESS
DGMGRL>
-bash-4.2$ dgmgrl C##RDS_DATAGUARD@RDS_CUSTOM_RDSCDB_B-bash-4.2$ dgmgrl C##RDS_DATAGUARD@RDS_CUSTOM_RDSCDB_B| DBA | ## Related resources + [Configuring Amazon RDS as an Oracle PeopleSoft Database](https://d1.awsstatic.com/whitepapers/configuring-amazon-rds-as-peoplesoft-database.pdf) (AWS whitepaper) + [Oracle Data Guard Broker guide ](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/index.html)(Oracle reference documentation) + [Data Guard Concepts and Administration ](https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/index.html)(Oracle reference documentation) ## Additional information **Select your database protection mode** Oracle Data Guard provides three protection modes to configure your Data Guard environment based on your availability, protection, and performance requirements. The following table summarizes these three modes. | | | Protection mode | Redo transport setting | Description | | --- |--- |--- | | ***MAXIMUM PERFORMANCE*** | `ASYNC` | For transactions happening on the primary database, redo data is asynchronously transmitted and written to the standby database redo log. Therefore, the performance impact is minimal.`MaxPerformance` can’t provide RPO=0 because of asynchronous log shipping. | | ***MAXIMUM PROTECTION*** | `SYNC+AFFIRM` | For transactions on the primary database, redo data is synchronously transmitted and written to the standby database redo log on disk before the transaction is acknowledged. If the standby database becomes unavailable, the primary database shuts itself down to ensure transactions are protected. | | ***MAXIMUM AVAILABILITY*** | `SYNC+AFFIRM` | This is similar to `MaxProtection` mode, except when no acknowledgement is received from the standby database. In that case, it operates as if it were in `MaxPerformance` mode to preserve the primary database availability until it’s able to write its redo stream to a synchronized standby database again. | | `SYNC+NOAFFIRM` | For transactions on the primary database, redo is synchronously transmitted to the standby database, and the primary waits only for acknowledgement that the redo has been received on the standby, not that it has been written to the standby disk. This mode, which is also known as `FastSync`, can provide a performance benefit at the expense of potential exposure to data loss in a special case of multiple simultaneous failures. | Read replicas in RDS Custom for Oracle are created with maximum performance protection mode, which is also the default protection mode for Oracle Data Guard. The maximum performance mode provides the lowest performance impact on the primary database, which can help you meet the recovery point objective (RPO) requirement measured in seconds. To work to achieve a zero data loss (RPO=0) objective, you can customize the Oracle Data Guard protection mode to `MaxAvailability` with the `SYNC+NOAFFIRM` setting for redo transport for better performance. Because commits on the primary database are acknowledged only after the corresponding redo vectors are successfully transmitted to the standby database, the network latency between the primary instance and replica can be crucial for commit-sensitive workloads. We recommend performing load testing for your workload to assess the performance impact when the read replica is customized to run in `MaxAvailability` mode. Deploying the read replica in the same Availability Zone as the primary database provides lower network latency compared with deploying the read replica in a different Availability Zone. However, deploying the primary and read replicas in the same Availability Zone might not meet your HA requirements because, in the unlikely event of Availability Zone unavailability, both the primary instance and read replica instance are impacted. # Analyze object dependencies for partial database migrations from Oracle to PostgreSQL *anuradha chintha, Amazon Web Services* ## Summary This pattern describes the importance of systematically identifying and managing system dependencies when migrating a partial Oracle database to either Amazon Relational Database Service (Amazon RDS) or Amazon Aurora PostgreSQL. In a partial migration, only a subset of database objects and data from the original database are migrated, while the source database continues to operate and serve applications that depend on nonmigrated components You must identify and analyze the migration’s scope when handling large-scale databases that have tightly coupled applications with upstream and downstream dependencies. To begin a partial migration, identify scope objects, including tables, triggers, views, stored procedures, functions, and packages. The scope identification process follows a comprehensive approach: + First-level scope objects are identified through direct references in application code and critical module-specific jobs. + Second-level objects are derived through comprehensive dependency analysis. When you understand how different parts of your system interact, you can better plan the correct order for moving database components and reduce the risk of migration failures. The following table lists different types of dependency analysis. | | | Analysis type | Focus areas | Purpose | | --- |--- |--- | | Object dependencies | TablesViewsStored ProceduresFunctionsTriggers | Identifies relationships between database objects and their hierarchical structures | | Segment dependencies | Foreign key relationshipsPrimary key chainsCross-schema references | Maps data relationships and maintains referential integrity | | Security dependencies | User permissionsRole hierarchiesObject privileges | Ensures proper access control migration and security maintenance | | Access patterns | Read operationsWrite operations | Determines database interaction patterns | To maintain consistency between source and target systems, establish data synchronization mechanisms during the transition period. You must also modify application code and functions to handle data distribution across both the source Oracle and target PostgreSQL databases. ## Prerequisites and limitations **Prerequisites** + An active AWS account + An Oracle database (source) + An Amazon RDS or Amazon Aurora PostgreSQL (target) **Product versions** + Oracle 19c or later + PostgreSQL 16 or later ## Architecture **Source technology stack** + Oracle 19c or later **Target technology stack** + Amazon RDS or Amazon Aurora PostgreSQL **Target architecture** The following diagram shows the migration process from an on-premises Oracle database to Amazon RDS for Oracle, which involves the following: + Identifying database dependencies + Migrating database code and objects using AWS Schema Conversion Tool (AWS SCT) + Migrating data using AWS Database Migration Service (AWS DMS) + Replicating ongoing changes through change data capture (CDC) using AWS DMS For more information, see [Integrating AWS Database Migration Service with AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_DMSIntegration.html) in the AWS documentation. ![\[alt text not found\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/90160825-3199-4382-95a8-ad63139c5c89/images/b09c36a4-27fa-412e-877e-57a31bcce0dc.png) ## Tools **AWS services** + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) for Oracle helps you set up, operate, and scale an Oracle relational database in the AWS Cloud. + Amazon Aurora is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL. + AWS Schema Conversion Tool (AWS SCT) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database. + [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups. **Other services** + [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) is an integrated development environment that simplifies the development and management of Oracle databases in both traditional and cloud-based deployments. For this pattern, you can use [SQL\$1Plus](https://docs.oracle.com/cd/B19306_01/server.102/b14357/qstart.htm). ## Best practices For best practices about provisioning and migrating an Oracle database, see [Best practices for migrating to Amazon RDS for Oracle](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/best-practices.html). ## Epics ### Identify object dependencies | Task | Description | Skills required | | --- | --- | --- | | Create an object table. | Identify objects that are essential to the application's functionality, and create a table named `DEPENDENT_ANALYSIS_BASELINE`. Add records for each object to the table. For an example, see the *Additional information* section. | Data engineer, DBA | | Create a database procedure. | Create a stored procedure named `sp_object_dependency_analysis` to analyze object dependencies in both directions (forward and backward) by using data from the `DBA_DEPENDENCIES` table. For an example, see the *Additional information* section. | Data engineer, DBA | | Run the procedure. | Run the scripts at each successive level until no new object dependencies are found. All dependencies and levels are stored in the `DEPENDENT_ANALYSIS_BASELINE` table. | DBA, Data engineer | ### Create a procedure for segment-level dependencies | Task | Description | Skills required | | --- | --- | --- | | Create a dependency table. | Create a segment-level dependency table named `REFERENTIAL_ANALYSIS_BASELINE`. When all object-level dependencies are discovered, check the parent tables of `DEPENDENT_ANALYSIS_BASELINE` by querying the `DBA_CONSTRAINT` table.Exclude dependencies where baseline tables are referenced by other tables. Backfilling handles these relationships. The following is an example script:
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Jan 11 21:21:11 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "RDSCDB_B"
Connected as SYSDG.
DGMGRL> edit database rdscdb_b set state=apply-on;
Succeeded.
DGMGRL> show database rdscdb_b
Database - rdscdb_b
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 35.00 KByte/s
Real Time Query: ON
Instance(s):
RDSCDB
Database Status:
SUCCESS
DGMGRL> show database rdscdb_b
Database - rdscdb_b
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 16.00 KByte/s
Real Time Query: ON
Instance(s):
RDSCDB
Database Status:
SUCCESS
DGMGRL>
CREATE TABLE REFERENTIAL_ANALYSIS_BASELINE| Data engineer, DBA | | Create a database procedure. | Create a procedure called `SP_OBJECT_REFERENTIAL_ANALYSIS`, and generate a referential analysis for all identified objects. For an example, see the *Additional information* section. | Data engineer, DBA | | Run the procedure. | Run the procedure to get referential dependencies. Generate referential analysis object details in `REFERENTIAL_ANALYSIS_BASELINE`. | Data engineer, DBA | ### Identify objects that read and write | Task | Description | Skills required | | --- | --- | --- | | Create tables for read and write objects. | Use the following script to create a read object table named `TABLE_READ_OBJECT_DETAILS` and a write object table named `TABLE_WRITE_OBJECT_DETAILS`:
(CHILD_OWNER VARCHAR2(50 BYTE),
CHILD_NAME VARCHAR2(100 BYTE),
PARENT_OWNER VARCHAR2(50 BYTE),
PARENT_NAME VARCHAR2(50 BYTE),
REFERENCE_PATH VARCHAR2(1000 BYTE));
CREATE TABLE TABLE_READ_OBJECT_DETAILS
(OWNER VARCHAR2(50 BYTE),
TAB_NAME VARCHAR2(50 BYTE),
READER_OWNER VARCHAR2(50 BYTE),
READER_NAME VARCHAR2(50 BYTE),
READER_TYPE VARCHAR2(50 BYTE));
CREATE TABLE TABLE_WRITE_OBJECT_DETAILS| Data engineer, DBA | | Create a procedure for analysis. | Create the procedures `SP_READER_OBJECTS_ANALYSIS` and `SP_WRITER_OBJECTS_ANALYSIS` for analyzing read objects and write objects, respectively. These procedures use pattern matching to find related objects. For an example, see the *Additional information* section. | Data engineer, DBA | | Run the procedures. | Run these procedures to identify dependent objects. | DBA, Data engineer | ### Review database privileges | Task | Description | Skills required | | --- | --- | --- | | Create a table for reviewing privileges. | Create a table for analyzing privileges named `OBJECT_PRIVS_ANALYSIS`. To recursively capture object privileges in the `DEPENDENT_ANALYSIS_BASELINE` table, use the following script:
(TABLE_NAME VARCHAR2(100 BYTE),
WRITEOBJ_OWNER VARCHAR2(100 BYTE),
WRITEOBJ_NAME VARCHAR2(100 BYTE),
WRITEOBJ_TYPE VARCHAR2(100 BYTE),
LINE VARCHAR2(100 BYTE),
TEXT VARCHAR2(4000 BYTE),
OWNER VARCHAR2(50 BYTE));
CREATE TABLE OBJECT_PRIVS_ANALYSIS| Data engineer, DBA | | Create a procedure for reviewing privileges. | Create a procedure named `SP_OBJECT_PRIVS_ANALYSIS`. Generate a privileges analysis for identified objects. For an example, see the *Additional information* section. | DBA, Data engineer | | Run the procedure. | Run the procedure to capture them in the `OBJECT_PRIVS_ANALYSIS` table. | DBA, Data engineer | ## Troubleshooting | Issue | Solution | | --- | --- | | Unable to access dictionary tables | Ensure that the user who created the analysis objects can access the DBA tables. | ## Related resources **AWS documentation** + [Amazon RDS and Aurora documentation](https://docs.aws.amazon.com/rds/) + [Oracle database 19c to Amazon Aurora PostgreSQL migration playbook](https://docs.aws.amazon.com/dms/latest/oracle-to-aurora-postgresql-migration-playbook/chap-oracle-aurora-pg.html) + [What is AWS Database Migration Service?](https://docs.aws.amazon.com/dms/latest/userguide/) + [What is the AWS Schema Conversion Tool?](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/) **Other documentation** + [Oracle database objects](https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/Database-Objects.html) ## Additional information **Script for `DEPENDENT_ANALYSIS_BASELINE`** ``` CREATE TABLE DEPENDENT_ANALYSIS_BASELINE (OWNER VARCHAR2(128 BYTE) NOT NULL ENABLE, OBJECT_NAME VARCHAR2(128 BYTE) NOT NULL ENABLE, OBJECT_TYPE VARCHAR2(20 BYTE), DEPEDNCY_LEVEL NUMBER, PROJECT_NEED VARCHAR2(20 BYTE), CATAGORY VARCHAR2(4000 BYTE), COMMENTS VARCHAR2(4000 BYTE), CATAGORY1 CLOB, COMMENTS1 CLOB, CUSTOMER_COMMENTS VARCHAR2(1000 BYTE), BACKFILL_TO_GUS VARCHAR2(1000 BYTE), BACKFILL_NEAR_REAL_TIME_OR_BATCH VARCHAR2(1000 BYTE), PK_EXISTS VARCHAR2(3 BYTE), UI_EXISTS VARCHAR2(3 BYTE), LOB_EXISTS VARCHAR2(3 BYTE), MASTER_LINK VARCHAR2(100 BYTE), CONSTRAINT PK_DEPENDENT_ANALYSIS_BASELINE PRIMARY KEY (OWNER,OBJECT_NAME,OBJECT_TYPE)); ``` **Procedure for `SP_WRITER_OBJECTS_ANALYSIS`** ``` CREATE OR REPLACE PROCEDURE SP_WRITER_OBJECTS_ANALYSIS IS BEGIN EXECUTE IMMEDIATE 'TRUNCATE TABLE TABLE_WRITE_OBJECT_DETAILS'; FOR I IN (SELECT OWNER, OBJECT_NAME FROM DEPENDENT_ANALYSIS_BASELINE WHERE OBJECT_TYPE = 'TABLE') LOOP INSERT INTO TABLE_WRITE_OBJECT_DETAILS(OWNER, TABLE_NAME, WRITEOBJ_OWNER, WRITEOBJ_NAME, WRITEOBJ_TYPE, LINE, TEXT) SELECT DISTINCT I.OWNER, I.OBJECT_NAME, OWNER WRITEOBJ_OWNER, NAME, TYPE, LINE, TRIM(TEXT) FROM DBA_SOURCE WHERE UPPER(TEXT) LIKE '%' || I.OBJECT_NAME || '%' AND (UPPER(TEXT) LIKE '%INSERT%' || I.OBJECT_NAME || '%' OR UPPER(TEXT) LIKE '%UPDATE%' || I.OBJECT_NAME || '%' OR UPPER(TEXT) LIKE '%DELETE%' || I.OBJECT_NAME || '%' OR UPPER(TEXT) LIKE '%UPSERT%' || I.OBJECT_NAME || '%' OR UPPER(TEXT) LIKE '%MERGE%' || I.OBJECT_NAME || '%') AND UPPER(TEXT) NOT LIKE '%PROCEDURE%' AND UPPER(TEXT) NOT LIKE 'PROCEDURE%' AND UPPER(TEXT) NOT LIKE '%FUNCTION%' AND UPPER(TEXT) NOT LIKE 'FUNCTION%' AND UPPER(TEXT) NOT LIKE '%TRIGGER%' AND UPPER(TEXT) NOT LIKE 'TRIGGER%' AND UPPER(TRIM(TEXT)) NOT LIKE '%AFTER UPDATE%' AND UPPER(TRIM(TEXT)) NOT LIKE 'BEFORE UPDATE%' AND UPPER(TRIM(TEXT)) NOT LIKE 'BEFORE INSERT%' AND UPPER(TRIM(TEXT)) NOT LIKE 'AFTER INSERT%' AND UPPER(TRIM(TEXT)) NOT LIKE 'BEFORE DELETE%' AND UPPER(TRIM(TEXT)) NOT LIKE 'AFTER DELETE%' AND UPPER(TRIM(TEXT)) NOT LIKE '%GGLOGADM.GG_LOG_ERROR%' AND (TRIM(TEXT) NOT LIKE '/*%' AND TRIM(TEXT) NOT LIKE '--%' ) AND (OWNER, NAME, TYPE) IN (SELECT OWNER, NAME, TYPE FROM DBA_DEPENDENCIES WHERE REFERENCED_NAME = I.OBJECT_NAME); END LOOP; END; ``` **Script for `SP_READER_OBJECTS_ANALYSIS`** ``` CREATE OR REPLACE PROCEDURE SP_READER_OBJECTS_ANALYSIS IS BEGIN EXECUTE IMMEDIATE 'TRUNCATE TABLE TABLE_READ_OBJECT_DETAILS'; FOR I IN (SELECT OWNER, OBJECT_NAME FROM DEPENDENT_ANALYSIS_BASELINE WHERE OBJECT_TYPE = 'TABLE') LOOP INSERT INTO TABLE_READ_OBJECT_DETAILS SELECT DISTINCT i.owner, i.object_name, owner, name, type FROM dba_dependencies WHERE referenced_name = I.OBJECT_NAME AND referenced_type = 'TABLE' AND type NOT IN ('SYNONYM', 'MATERIALIZED VIEW', 'VIEW') AND (owner, name, type) NOT IN ( SELECT DISTINCT owner, trigger_name, 'TRIGGER' FROM dba_triggers WHERE table_name = I.OBJECT_NAME AND table_owner = i.owner UNION ALL SELECT DISTINCT owner, name, type FROM dba_source WHERE upper(text) LIKE '%' || I.OBJECT_NAME || '%' AND (upper(text) LIKE '%INSERT %' || I.OBJECT_NAME || '%' OR upper(text) LIKE '%UPDATE% ' || I.OBJECT_NAME || '%' OR upper(text) LIKE '%DELETE %' || I.OBJECT_NAME || '%' OR upper(text) LIKE '%UPSERT %' || I.OBJECT_NAME || '%' OR upper(text) LIKE '%MERGE %' || I.OBJECT_NAME || '%') AND upper(text) NOT LIKE '%PROCEDURE %' AND upper(text) NOT LIKE 'PROCEDURE %' AND upper(text) NOT LIKE '%FUNCTION %' AND upper(text) NOT LIKE 'FUNCTION %' AND upper(text) NOT LIKE '%TRIGGER %' AND upper(text) NOT LIKE 'TRIGGER %' AND upper(trim(text)) NOT LIKE 'BEFORE INSERT %' AND upper(trim(text)) NOT LIKE 'BEFORE UPDATE %' AND upper(trim(text)) NOT LIKE 'BEFORE DELETE %' AND upper(trim(text)) NOT LIKE 'AFTER INSERT %' AND upper(trim(text)) NOT LIKE 'AFTER UPDATE %' AND upper(trim(text)) NOT LIKE 'AFTER DELETE %' AND (trim(text) NOT LIKE '/*%' AND trim(text) NOT LIKE '--%')); END LOOP; END; ``` **Script for `SP_OBJECT_REFERENTIAL_ANALYSIS`** ``` CREATE OR REPLACE PROCEDURE SP_OBJECT_REFERENTIAL_ANALYSIS IS BEGIN EXECUTE IMMEDIATE 'TRUNCATE TABLE REFERENTIAL_ANALYSIS_BASELINE'; INSERT INTO REFERENTIAL_ANALYSIS_BASELINE WITH rel AS ( SELECT DISTINCT c.owner, c.table_name, c.r_owner r_owner, (SELECT table_name FROM dba_constraints WHERE constraint_name = c.r_constraint_name AND owner = c.r_owner) r_table_name FROM dba_constraints c WHERE constraint_type = 'R' AND c.owner NOT IN (SELECT username FROM dba_users WHERE oracle_maintained = 'Y') AND c.r_owner NOT IN (SELECT username FROM dba_users WHERE oracle_maintained = 'Y')), tab_list AS ( SELECT OWNER, object_name FROM DEPENDENT_ANALYSIS_BASELINE WHERE UPPER(OBJECT_TYPE) = 'TABLE') SELECT DISTINCT owner child_owner, table_name child, r_owner parent_owner, r_table_name parent, SYS_CONNECT_BY_PATH(r_table_name, ' -> ') || ' -> ' || table_name PATH FROM rel START WITH (r_owner, r_table_name) IN (SELECT * FROM tab_list) CONNECT BY NOCYCLE (r_owner, r_table_name) = ((PRIOR owner, PRIOR table_name)) UNION SELECT DISTINCT owner child_owner, table_name child, r_owner parent_owner, r_table_name parent, SYS_CONNECT_BY_PATH(table_name, ' -> ') || ' -> ' || r_table_name PATH FROM rel START WITH (owner, table_name) IN (SELECT * FROM tab_list) CONNECT BY NOCYCLE (owner, table_name) = ((PRIOR r_owner, PRIOR r_table_name)); END; ``` **Script for `SP_OBJECT_PRIVS_ANALYSIS`** ``` CREATE OR REPLACE PROCEDURE SP_OBJECT_PRIVS_ANALYSIS IS V_SQL VARCHAR2(4000); V_CNT NUMBER; BEGIN V_SQL := 'TRUNCATE TABLE OBJECT_PRIVS_ANALYSIS'; EXECUTE IMMEDIATE V_SQL; FOR I IN (SELECT OWNER, OBJECT_NAME FROM DEPENDENT_ANALYSIS_BASELINE WHERE OBJECT_TYPE = 'TABLE') LOOP INSERT INTO OBJECT_PRIVS_ANALYSIS(OWNER, OBJECT_NAME, USER_NAME, PRIVS) WITH obj_to_role AS ( SELECT DISTINCT GRANTEE role_name, DECODE(privilege, 'SELECT', 'READ', 'REFERENCE', 'READ', 'INSERT', 'WRITE', 'UPDATE', 'WRITE', 'DELETE', 'WRITE', privilege) privs FROM DBA_TAB_PRIVS t, DBA_ROLES r WHERE OWNER = I.OWNER AND TYPE = 'TABLE' AND TABLE_NAME = I.OBJECT_NAME AND t.GRANTEE = r.ROLE AND r.ROLE IN (SELECT ROLE FROM DBA_ROLES WHERE ORACLE_MAINTAINED = 'N') ) SELECT I.OWNER, I.OBJECT_NAME, grantee, privs FROM ( -- Recursively Role to User mapping with privilege SELECT DISTINCT grantee, privs FROM (SELECT rp.granted_role, rp.grantee, privs, (SELECT DECODE(COUNT(*), 0, 'ROLE', 'USER') FROM (SELECT 'User' FROM DBA_users WHERE username = rp.GRANTEE)) grantee_type FROM DBA_role_privs rp, obj_to_role r WHERE rp.granted_role = r.role_name AND grantee IN ((SELECT USERNAME FROM DBA_USERS WHERE ORACLE_MAINTAINED = 'N') UNION (SELECT ROLE FROM DBA_ROLES WHERE ORACLE_MAINTAINED = 'N')) AND granted_role IN (SELECT ROLE FROM DBA_ROLES WHERE ORACLE_MAINTAINED = 'N') START WITH granted_role IN (SELECT DISTINCT role_name FROM obj_to_role) CONNECT BY granted_role = PRIOR grantee) WHERE grantee_type = 'USER' ) UNION ( -- Direct Object grants to User SELECT I.OWNER, I.OBJECT_NAME, GRANTEE, DECODE(privilege, 'SELECT', 'READ', 'REFERENCE', 'READ', 'INSERT', 'WRITE', 'UPDATE', 'WRITE', 'DELETE', 'WRITE', privilege) privs FROM DBA_TAB_PRIVS, DBA_USERS WHERE GRANTEE = USERNAME AND OWNER = I.OWNER AND TYPE = 'TABLE' AND TABLE_NAME = I.OBJECT_NAME ) ORDER BY 2 DESC; END LOOP; END; ``` **Procedure for `SP_OBJECT_DEPENDENCY_ANALYSIS`** ``` CREATE OR REPLACE PROCEDURE SP_OBJECT_DEPENDENCY_ANALYSIS (v_level NUMBER) IS TYPE typ IS RECORD ( schema VARCHAR2(100), obj_type VARCHAR2(100), obj_name VARCHAR2(100), path VARCHAR2(5000) ); TYPE array IS TABLE OF typ; l_data array; c SYS_REFCURSOR; l_errors NUMBER; l_errno NUMBER; l_msg VARCHAR2(4000); l_idx NUMBER; l_level NUMBER; BEGIN l_level := v_level + 1; OPEN c FOR WITH obj_list AS ( SELECT owner schema_name, object_type, object_name FROM DEPENDENT_ANALYSIS_BASELINE WHERE depedncy_level = v_level ), fw_dep_objects AS ( SELECT level lvl, owner, name, type, referenced_owner, referenced_name, referenced_type, SYS_CONNECT_BY_PATH(name, ' -> ') || ' -> ' || referenced_name PATH FROM dba_dependencies START WITH (owner, CASE WHEN type = 'PACKAGE BODY' THEN 'PACKAGE' ELSE type END, name) IN (SELECT schema_name, object_type, object_name FROM obj_list) CONNECT BY NOCYCLE (owner, type, name) = ((PRIOR referenced_owner, PRIOR referenced_type, PRIOR referenced_name)) ), bw_dep_objects AS ( SELECT level lvl, owner, name, type, referenced_owner, referenced_name, referenced_type, SYS_CONNECT_BY_PATH(name, ' <- ') || ' <- ' || referenced_name PATH FROM dba_dependencies START WITH (referenced_owner, CASE WHEN referenced_type = 'PACKAGE BODY' THEN 'PACKAGE' ELSE referenced_type END, referenced_name) IN (SELECT schema_name, object_type, object_name FROM obj_list) CONNECT BY NOCYCLE (referenced_owner, referenced_type, referenced_name) = ((PRIOR owner, PRIOR type, PRIOR name)) ) SELECT * FROM ( (SELECT DISTINCT referenced_owner schema, referenced_type obj_type, referenced_name obj_name, path FROM fw_dep_objects) UNION (SELECT DISTINCT owner schema, type obj_type, name obj_name, path FROM bw_dep_objects) ) WHERE schema IN (SELECT username FROM all_users WHERE oracle_maintained = 'N') ORDER BY obj_type; LOOP FETCH c BULK COLLECT INTO l_data LIMIT 100; BEGIN FORALL i IN 1..l_data.count SAVE EXCEPTIONS INSERT INTO DEPENDENT_ANALYSIS_BASELINE ( owner, object_name, object_type, catagory, depedncy_level, project_need, comments ) VALUES ( l_data(i).schema, l_data(i).obj_name, CASE WHEN l_data(i).obj_type = 'PACKAGE BODY' THEN 'PACKAGE' ELSE l_data(i).obj_type END, 'level ' || l_level || ' dependency', l_level, '', 'from dependency proc' || l_data(i).path ); EXCEPTION WHEN OTHERS THEN l_errors := sql%bulk_exceptions.count; FOR i IN 1..l_errors LOOP l_errno := sql%bulk_exceptions(i).error_code; l_msg := SQLERRM(-l_errno); l_idx := sql%bulk_exceptions(i).error_index; UPDATE DEPENDENT_ANALYSIS_BASELINE SET catagory1 = catagory1 || ', found in level' || l_level || ' dependent of ' || l_data(l_idx).path, comments1 = comments1 || ', from dependency proc exception ' || l_data(i).path WHERE owner = l_data(l_idx).schema AND object_name = l_data(l_idx).obj_name AND object_type = l_data(l_idx).obj_type; END LOOP; END; EXIT WHEN c%NOTFOUND; END LOOP; CLOSE c; END; ``` # Assess query performance for migrating SQL Server databases to MongoDB Atlas on AWS *Battulga Purevragchaa, Amazon Web Services* *Krishnakumar Sathyanarayana, PeerIslands US Inc* *Babu Srinivasan, MongoDB* ## Summary This pattern provides guidance for loading MongoDB with near real-world data and assessing MongoDB query performance that is as close to the production scenario as possible. The assessment provides input to help you plan your migration to MongoDB from a relational database. The pattern uses [PeerIslands Test Data Generator and Performance Analyzer](https://tools.peerislands.io/) to test query performance. This pattern is particularly useful for Microsoft SQL Server migration to MongoDB, because performing schema transformations and loading data from current SQL Server instances to MongoDB can be very complex. Instead, you can load near real-world data into MongoDB, understand MongoDB performance, and fine-tune the schema design before you start the actual migration. ## Prerequisites and limitations **Prerequisites** + An active AWS account + Familiarity with [MongoDB Atlas](https://www.mongodb.com/docs/atlas/getting-started/) + Target MongoDB schema + Typical query patterns **Limitations** + Data load times and performance will be limited by the MongoDB cluster instance size. We recommend that you choose instances that are recommended for production use to understand real-world performance. + PeerIslands Test Data Generator and Performance Analyzer currently supports only online data loads and queries. Offline batch processing (for example, loading data into MongoDB by using Spark connectors) isn’t yet supported. + PeerIslands Test Data Generator and Performance Analyzer supports field relations within a collection. It doesn’t support relationships across collections. **Product editions** + This pattern supports both [MongoDB Atlas](https://www.mongodb.com/atlas) and [MongoDB Enterprise Advanced](https://www.mongodb.com/products/mongodb-enterprise-advanced). ## Architecture **Target technology stack** + MongoDB Atlas or MongoDB Enterprise Advanced **Architecture** ![\[Architecture to assess query performance for migrating SQL Server database to MongoDB Atlas on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/25f0ab73-d587-4bd0-9fc0-ac675d5aa349/images/717caae4-d52e-4c78-bb7d-2ecb5acccd42.png) PeerIslands Test Data Generator and Performance Analyzer is built by using Java and Angular, and stores its generated data on Amazon Elastic Block Store (Amazon EBS). The tool consists of two workflows: test data generation and performance testing. + In test data generation, you create a template, which is the JSON representation of the data model that has to be generated. After you create the template, you can generate the data in a target collection, as defined by the load generation configuration. + In performance testing, you create a profile. A profile is a multi-stage testing scenario where you can configure create, read, update, and delete (CRUD) operations, aggregation pipelines, the weightage for each operation, and the duration of each stage. After you create the profile, you can run performance testing on the target database based on the configuration. PeerIslands Test Data Generator and Performance Analyzer stores its data on Amazon EBS, so you can connect Amazon EBS to MongoDB by using any MongoDB-supported connection mechanism, including peering, allow lists, and private endpoints. By default, the tool doesn’t include operational components; however, it can be configured with Amazon Managed Service for Prometheus, Amazon Managed Grafana, Amazon CloudWatch, and AWS Secrets Manager if required. ## Tools + [PeerIslands Test Data Generator and Performance Analyzer](https://tools.peerislands.io/) includes two components. The Test Data Generator component helps you generate highly customer-specific, real-world data based on your MongoDB schema. The tool is fully UI-driven with a rich data library and can be used to quickly generate billions of records on MongoDB. The tool also provides capabilities to implement relationships between fields in the MongoDB schema. The Performance Analyzer component helps you generate highly customer-specific queries and aggregations, and perform realistic performance testing on MongoDB. You can use the Performance Analyzer to test MongoDB performance with rich load profiles and parameterized queries for your specific use case. ## Best practices See the following resources: + [MongoDB Schema Design Best Practices](https://www.mongodb.com/developer/products/mongodb/mongodb-schema-design-best-practices/) (MongoDB Developer website) + [Best Practices of Deploying MongoDB Atlas on AWS](https://www.mongodb.com/presentation/best-practices-of-deploying-mongodb-atlas-on-aws) (MongoDB website) + [Connecting Applications Securely to a MongoDB Atlas Data Plane with AWS PrivateLink](https://aws.amazon.com/blogs/apn/connecting-applications-securely-to-a-mongodb-atlas-data-plane-with-aws-privatelink/) (AWS blog post) + [Best Practices Guide for MongoDB Performance ](https://www.mongodb.com/basics/best-practices)(MongoDB website) ## Epics ### Understand your source data | Task | Description | Skills required | | --- | --- | --- | | Understand the database footprint of the current SQL Server source. | Understand your current SQL Server footprint. This can be achieved by running queries against the `INFORMATION` schema of the database. Determine the number of tables and size of each table. Analyze the index associated with each table. For more information about SQL analysis, see the blog post [SQL2Mongo: Data Migration Journey](https://engineering.peerislands.io/sql2mongo-data-migration-journey-fec91a421d60) on the PeerIslands website. | DBA | | Understand the source schema. | Determine the table schema and the business representation of the data (for example, zip codes, names, and currency). Use your existing entity relationship (ER) diagram or generate the ER diagram from the existing database. For more information, see the blog post [SQL2Mongo: Data Migration Journey](https://engineering.peerislands.io/sql2mongo-data-migration-journey-fec91a421d60) on the PeerIslands website. | DBA | | Understand query patterns. | Document the top 10 SQL queries you use. You can use the **performance\$1schema.events\$1statements\$1summary\$1by\$1digest** tables that are available in the database to understand the top queries. For more information, see the blog post [SQL2Mongo: Data Migration Journey](https://engineering.peerislands.io/sql2mongo-data-migration-journey-fec91a421d60) on the PeerIslands website. | DBA | | Understand SLA commitments. | Document the target service-level agreements (SLAs) for database operations. Typical measures include query latency and queries per second. The measures and their targets are typically available in non-functional requirements (NFR) documents. | DBA | ### Define the MongoDB schema | Task | Description | Skills required | | --- | --- | --- | | Define the target schema. | Define various options for the target MongoDB schema. For more information, see [Schemas](https://www.mongodb.com/docs/atlas/app-services/schemas/) in the MongoDB Atlas documentation. Consider the best practices and design patterns based on the table relations. | MongoDB engineer | | Define target query patterns. | Define MongoDB queries and aggregation pipelines. These queries are the equivalent of the top queries you captured for your SQL Server workload. To understand how to construct MongoDB aggregation pipelines, see the [MongoDB documentation](https://www.mongodb.com/docs/manual/core/aggregation-pipeline/). | MongoDB engineer | | Define the MongoDB instance type. | Determine the size of the instance that you plan to use for testing. For guidance, see the [MongoDB documentation](https://www.mongodb.com/docs/atlas/sizing-tier-selection/). | MongoDB engineer | ### Prepare the target database | Task | Description | Skills required | | --- | --- | --- | | Set up the MongoDB Atlas cluster. | To set up a MongoDB cluster on AWS, follow the instructions in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). | MongoDB engineer | | Create users in the target database. | Configure the MongoDB Atlas cluster for access and network security by following the instructions in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/connect-to-database-deployment/). | MongoDB engineer | | Create appropriate roles in AWS and configure role-based access control for Atlas. | If required, set up additional users by following the instructions in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/security/set-up-unified-aws-access/). Configure [authentication and authorization](https://www.mongodb.com/docs/atlas/security/config-db-auth/) through AWS roles. | MongoDB engineer | | Set up Compass for MongoDB Atlas access. | Set up the [MongoDB Compass GUI utility](https://www.mongodb.com/products/compass) for ease of navigation and access. | MongoDB engineer | ### Set up the base load by using Test Data Generator | Task | Description | Skills required | | --- | --- | --- | | Install Test Data Generator. | Install [PeerIsland Test Data Generator](https://github.com/PeerIslands/loadgen_binary) in your environment. | MongoDB engineer | | Configure Test Data Generator to generate the appropriate data. | Create a template by using the data library to generate specific data for each field in the MongoDB schema. For more information, see the [MongoDB Data Generator & Perf. Analyzer](https://vimeo.com/570068857) video. | MongoDB engineer | | Horizontally scale Test Data Generator to generate the required load. | Use the template you created to start the load generation against the target collection by configuring the required parallelism. Determine the time frames and scale to generate the necessary data. | MongoDB engineer | | Validate the load in MongoDB Atlas. | Check the data loaded into MongoDB Atlas. | MongoDB engineer | | Generate required indexes on MongoDB. | Define indexes as required, based on query patterns. For best practices, see the [MongoDB documentation](https://www.mongodb.com/docs/manual/applications/indexes/). | MongoDB engineer | ### Conduct performance testing | Task | Description | Skills required | | --- | --- | --- | | Set up load profiles in Performance Analyzer. | Create a performance testing profile in Performance Analyzer by configuring specific queries and their corresponding weightage, duration of the test run, and stages. For more information, see the [MongoDB Data Generator & Perf. Analyzer](https://vimeo.com/570068857) video. | MongoDB engineer | | Run performance testing. | Use the performance testing profile you created to start the test against the target collection by configuring the required parallelism. Horizontally scale the performance test tool to run queries against MongoDB Atlas. | MongoDB engineer | | Record test results. | Record P95, P99 latency for the queries. | MongoDB engineer | | Tune your schema and query patterns. | Modify indexes and query patterns to address any performance issues. | MongoDB engineer | ### Close the project | Task | Description | Skills required | | --- | --- | --- | | Shut down temporary AWS resources. | Delete all temporary resources that you used for Test Data Generator and Performance Analyzer. | AWS administrator | | Update performance test results. | Understand MongoDB query performance and compare it against your SLAs. If necessary, fine-tune the MongoDB schema and rerun the process. | MongoDB engineer | | Conclude the project. | Close out the project and provide feedback. | MongoDB engineer | ## Related resources + GitHub repository: [S3toAtlas](https://github.com/mongodb-partners/S3toAtlas) + Schema: [MongoDB Schema design](https://www.mongodb.com/developer/products/mongodb/mongodb-schema-design-best-practices/) + Aggregation pipelines : [MongoDB aggregation pipelines](https://www.mongodb.com/docs/manual/core/aggregation-pipeline/) + MongoDB Atlas sizing : [Sizing tier selection](https://www.mongodb.com/docs/atlas/sizing-tier-selection/) + Video: [MongoDB Data Generator](https://vimeo.com/570068857) & Perf. Analyzer + References: [MongoDB documentation](https://www.mongodb.com/docs/) + Tutorials:** **[MongoDB developer guide, ](https://www.mongodb.com/docs/develop-applications/)[MongoDB Jumpstart](https://www.youtube.com/playlist?list=PL4RCxklHWZ9v2lcat4oEVGQhZg6r4IQGV) + AWS Marketplace:** **[MongoDB Atlas on AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=c9032c7b-70dd-459f-834f-c1e23cf3d092) + AWS Partner Solutions:** **[ MongoDB Atlas on AWS Reference Deployment](https://aws.amazon.com/quickstart/architecture/mongodb-atlas/) Additional resources: + [SQL analysis](https://engineering.peerislands.io/sql2mongo-data-migration-journey-fec91a421d60) + [MongoDB Developer Community forums](https://www.mongodb.com/community/forums/) + [MongoDB Performance Tuning Questions](https://www.mongodb.com/developer/products/mongodb/performance-tuning-tips/) + [Operational Analytics with Atlas and Redshift](https://github.com/mongodb-partners/Atlas_to_Redshift) + [Application modernization with MongoDB Atlas and AWS Elastic Beanstalk](https://github.com/mongodb-partners/MEANStack_with_Atlas_on_AWS_EB) # Automate blue/green deployments of Amazon Aurora global databases by using IaC principles *Ishwar Chauthaiwale, ANKIT JAIN, and Ramu Jagini, Amazon Web Services* ## Summary Managing database updates, migrations, or scaling efforts can be challenging for organizations that run critical workloads on [Amazon Aurora global databases](https://aws.amazon.com/rds/aurora/global-database/). Ensuring that these operations are carried out seamlessly, with zero downtime, is essential to maintaining service availability and avoiding disruptions for your users. A blue/green deployment strategy offers a solution to this challenge by allowing you to run two identical environments concurrently: blue (the current environment) and green (the new environment). A blue/green strategy enables you to implement changes, perform testing, and switch traffic between environments with minimal risk and downtime. This pattern helps you automate the blue/green deployment process for Aurora global databases by using infrastructure as code (IaC) principles. It uses [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html), [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), and [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) to simplify blue/green deployments. To improve reliability, it uses global transaction identifiers (GTIDs) for replication. GTID-based replication provides better data consistency and failover capabilities between environments compared with binary log (binlog) replication. **Note** This pattern assumes that you're using an Aurora MySQL-Compatible Edition global database cluster. If you're using Aurora PostgreSQL-Compatible instead, please use the PostgreSQL equivalents of the MySQL commands. By following the steps in this pattern, you can: + Provision a green Aurora global database: Using CloudFormation templates, you create a green environment that mirrors your existing blue environment. + Set up GTID-based replication: You configure GTID replication to keep the blue and green environments synchronized. + Seamlessly switch traffic: You use Route 53 and Lambda to automatically switch the traffic from the blue to the green environment after full synchronization. + Finalize the deployment: You validate that the green environment is fully operational as the primary database, and then stop replication and clean up any temporary resources. The approach in this pattern provides these benefits: + Reduces downtime during critical database updates or migrations: Automation ensures a smooth transition between environments with minimal service disruption. + Enables rapid rollbacks: If an issue arises after traffic is switched to the green environment, you can quickly revert to the blue environment and maintain service continuity. + Enhances testing and verification: The green environment can be fully tested without affecting the live blue environment, which reduces the likelihood of errors in production. + Ensures data consistency: GTID-based replication keeps your blue and green environments in sync, which prevents data loss or inconsistencies during migration. + Maintains business continuity: Automating your blue/green deployments helps avoid long outages and financial losses by keeping your services available during updates or migrations. ## Prerequisites and limitations **Prerequisites** + An active AWS account. + A source Aurora MySQL-Compatible global database cluster (blue environment). Global databases provide a multi-Region configuration for high availability and disaster recovery. For instructions for setting up a global database cluster, see the [Aurora documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-getting-started.html). + [GTID-based replication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-replication-gtid.html) enabled on the source cluster. **Limitations** + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service. **Product versions** + Aurora MySQL-Compatible 8.0 or later ## Architecture ![\[Using GTID replication to sync blue and green environments in different Regions.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/19922266-c2e5-460b-9a0f-22e6d6736094/images/7a8c3095-7904-4080-906f-0c403c289a4f.png) The diagram illustrates the following: + Global database setup: An Aurora global database cluster is strategically deployed across two AWS Regions. This configuration enables geographic distribution and Regional redundancy for enhanced disaster recovery capabilities. + Primary to secondary Region replication: The logical replication mechanism ensures seamless data synchronization from the primary Region to the secondary Region. This replication maintains data consistency with minimal latency across geographical distances. + GTID-based replication between clusters: GTID-based replication maintains transactional consistency and ordered data flow between the blue primary cluster and the green primary cluster, and ensures reliable data synchronization. + Blue primary to secondary replication: Logical replication establishes a robust data pipeline between the blue primary cluster and its secondary cluster. This replication enables continuous data synchronization and high availability. + Route 53 DNS configuration: Route 53 hosted zone records manage the DNS resolution for all blue and green cluster database endpoints. This configuration provides seamless endpoint mapping and enables efficient traffic routing during failover scenarios. ## Tools **AWS services** + [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL. + [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing on your applications that run on AWS. You create a template that describes all the AWS resources that you want, and CloudFormation takes care of provisioning and configuring those resources for you. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that supports running code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. + [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) is a highly available and scalable DNS web service. ## Best practices We recommend that you thoroughly review AWS documentation to deepen your understanding of the [blue/green deployment strategy](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/blue-green-deployments-overview.html), [GTID-based replication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-replication-gtid.html), and [weighted routing policies](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-weighted.html) in Route 53. This knowledge is crucial for effectively implementing and managing your database migrations, ensuring data consistency, and optimizing traffic routing. By gaining a comprehensive understanding of these AWS features and best practices, you'll be better equipped to handle future updates, minimize downtime, and maintain a resilient and secure database environment. For guidelines for using the AWS services for this pattern, see the following AWS documentation: + [Best practices with Amazon Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.BestPractices.html) + [CloudFormation best practices](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html) + [Best practices for working with AWS Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html) + [Best practices for Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/best-practices.html) ## Epics ### Create the green environment | Task | Description | Skills required | | --- | --- | --- | | Create a snapshot backup from the blue cluster. | In a blue/green deployment, the green environment represents a new, identical version of your current (blue) database environment. You use the green environment to safely test updates, validate changes, and ensure stability before switching production traffic. It acts as a staging ground for implementing database changes with minimal disruption to the live environment.To create a green environment, you first create a snapshot of the primary (blue) cluster in your Aurora MySQL-Compatible global database. This snapshot serves as the foundation for creating the green environment.To create a snapshot:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)Alternatively, you can use the AWS Command Line Interface (AWS CLI) to create the snapshot:
(OWNER VARCHAR2(50 BYTE),
OBJECT_NAME VARCHAR2(50 BYTE),
USER_NAME VARCHAR2(50 BYTE),
PRIVS VARCHAR2(50 BYTE));
aws rds create-db-cluster-snapshot --db-cluster-snapshot-identifier blue-green-demo --db-cluster-identifier ex-global-cluster --region eu-west-1Make sure that the snapshot completes successfully before proceeding to the next step. | DBA | | Generate the CloudFormation template for your global database and its resources. | The CloudFormation IaC generator helps you generate CloudFormation templates from existing AWS resources. Use this feature to create a CloudFormation template for your existing Aurora MySQL-Compatible global database and its associated resources. This template configures subnet groups, security groups, parameter groups, and other settings.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html) | DBA | | Modify the CloudFormation template for the green environment. | Customize the CloudFormation template to reflect the settings for the green environment. This includes updating resource names and identifiers to ensure that the green environment operates independently of the blue cluster.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)If you use the `SnapshotIdentifier` property to restore a DB cluster, avoid specifying properties such as `GlobalClusterIdentifier`, `MasterUsername`, or `MasterUserPassword`. | DBA | | Deploy the CloudFormation stack to create resources for the green environment. | In this step, you deploy the customized CloudFormation template to create the resources for the green environment.To deploy the CloudFormation stack:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)CloudFormation initiates the process of creating the green environment resources. This process might take several minutes to complete. | DBA | | Validate the CloudFormation stack and resources. | When the CloudFormation stack deployment is complete, you’ll need to verify that the green environment has been created successfully:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)After verification, your green environment is ready for further setup, including replication from the blue environment. | DBA | ### Configure GTID-based replication | Task | Description | Skills required | | --- | --- | --- | | Verify GTID settings on the blue cluster. | GTIDs provide a highly reliable method for replicating data between your blue and green environments. [GTID-based replication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-replication-gtid.html) offers a resilient, simplified approach by assigning a unique identifier to every transaction in the blue environment. This method ensures that data synchronization between environments is seamless, consistent, and easier to manage than traditional binlog replication.Before you configure replication, you need to ensure that GTID-based replication is properly enabled on the blue cluster. This step guarantees that each transaction in the blue environment is uniquely tracked and can be replicated in the green environment.To confirm that GTID is enabled:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)These settings enable GTID tracking for all future transactions in the blue environment. After you confirm these settings, you can start setting up replication. | DBA | | Create a replication user. | To replicate data from the blue environment to the green environment, you need to create a dedicated replication user on the blue cluster. This user will be responsible for managing the replication process.To set up the replication user:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)This user now has the necessary permissions to replicate data between the two environments. | DBA | | Configure GTID-based replication on the green cluster. | The next step is to configure the green cluster for GTID-based replication. This setup ensures that the green environment will continuously mirror all transactions that happen in the blue environment.To configure the green cluster:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html) | DBA | | Start replication on the green cluster. | You can now start the replication process. On the green cluster, run the command:
START SLAVE;This enables the green environment to start synchronizing data, and receiving and applying transactions from the blue environment. | DBA | | Verify the replication process. | To verify that the green environment is accurately replicating the data from the blue cluster:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)If all indicators are correct, GTID-based replication is functioning smoothly, and the green environment is fully synchronized with the blue environment. | DBA | ### Switch traffic from blue to green cluster | Task | Description | Skills required | | --- | --- | --- | | Configure Route 53 weighted routing policies. | After you verify data consistency between the blue and green environments, you can switch traffic from the blue cluster to the green cluster. This transition should be smooth and should minimize downtime and ensure the integrity of your application’s database. To address these requirements, you can use Route 53 for DNS routing and Lambda to automate traffic switching. Additionally, a well-defined rollback plan ensures that you can revert to the blue cluster in case of any issues.The first step is to configure weighted routing in Route 53. Weighted routing allows you to control the distribution of traffic between the blue and green clusters, and gradually shift traffic from one environment to the other.To configure weighted routing:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)For more information about weighted routing policies, see the [Route 53 documentation](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-weighted.html). | AWS DevOps | | Deploy a Lambda function to monitor replication lag. | To ensure that the green environment is fully synchronized with the blue environment, deploy a Lambda function that monitors replication lag between the clusters. This function can check the replication status, specifically the **Seconds\$1Behind\$1Master** metric, to determine whether the green cluster is ready to handle all traffic.Here’s a sample Lambda function you can use:
import boto3This function checks the replication lag and returns the value. If the lag is zero, the green cluster is fully in sync with the blue cluster. | AWS DevOps | | Automate DNS weight adjustment by using Lambda. | When the replication lag reaches zero, it's time to switch all traffic to the green cluster. You can automate this transition by using another Lambda function that adjusts the DNS weights in Route 53 to direct 100 percent of traffic to the green cluster.Here’s an example of a Lambda function that automates the traffic switch:
def check_replication_lag(event, context):
client = boto3.client('rds')
response = client.describe_db_instances(DBInstanceIdentifier='green-cluster-instance')
replication_status = response['DBInstances'][0]['ReadReplicaDBInstanceIdentifiers']
if replication_status:
lag = replication_status[0]['ReplicationLag']
return lag
return -1
import boto3This function checks replication lag and updates the Route 53 DNS weights when the lag is zero to fully switch traffic to the green cluster.** **During the cutover process, If the blue cluster experiences heavy write traffic, consider temporarily pausing write operations during the cutover. This ensures that replication catches up, and prevents data inconsistencies between the blue and green clusters. | AWS DevOps | | Verify the traffic switch. | After the Lambda function adjusts the DNS weights, you should verify that all traffic is directed to the green cluster and that the switch was successful.To verify:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)If everything is performing as expected, the traffic switch is complete. | AWS DevOps | | If you encounter any issues, roll back changes. | Having a rollback plan is critical in case any issues arise after the traffic switch. Here's how to quickly revert to the blue cluster if necessary:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)By implementing this rollback plan, you can ensure minimal disruption to your users in the event of any unexpected issues. | AWS DevOps | ### Validate and stop GTID-based replication | Task | Description | Skills required | | --- | --- | --- | | Stop GTID-based replication on the green cluster. | After you switch traffic from the blue environment to the green environment, you should validate the success of the transition and ensure that the green cluster is functioning as expected. Additionally, the GTID-based replication between the blue and green clusters must be stopped, because the green environment now serves as the primary database. Completing these tasks ensures that your environment is secure, streamlined, and optimized for ongoing operations.To stop replication:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)When you stop the replication, the green cluster becomes fully independent and operates as the primary database environment for your workloads. | DBA | | Clean up resources. | Cleaning up any temporary or unused resources that were created during the migration from the blue to the green cluster ensures that your environment remains optimized, secure, and cost-effective. The cleanup includes adjusting security settings, taking final backups, and decommissioning unnecessary resources.To clean up resources:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/p-automate-blue-green-deployments-aurora-global-databases-iac.html)Cleaning up resources helps maintain a secure and streamlined environment, reduces costs, and ensures that only necessary infrastructure remains. | AWS DevOps | ## Related resources CloudFormation: + [CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) + [CloudFormation best practices](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html) + [Generate templates from existing resources with IaC Generator](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/generate-IaC.html) + [Import entire applications into CloudFormation](https://aws.amazon.com/blogs/devops/import-entire-applications-into-aws-cloudformation/) (AWS blog post) Amazon Aurora: + [Amazon Aurora User Guide](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Welcome.html) + [Managing an Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Aurora.html) Blue/green deployment strategy: + [Overview of Amazon Aurora Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/blue-green-deployments-overview.html) GTID-based replication: + [Using GTID-based replication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-replication-gtid.html) (Amazon RDS documentation) AWS Lambda: + [AWS Lambda Developer Guide](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) + [Best practices for working with AWS Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html) Amazon Route 53: + [Amazon Route 53 Developer Guide](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) + [Weighted routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-weighted.html) MySQL client tools: + [PyMYSQL](https://github.com/PyMySQL/PyMySQL) # Automate the replication of Amazon RDS instances across AWS accounts *Parag Nagwekar and Arun Chandapillai, Amazon Web Services* ## Summary This pattern shows you how to automate the process of replicating, tracking, and rolling back your Amazon Relational Database Service (Amazon RDS) DB instances across different AWS accounts by using AWS Step Functions and AWS Lambda. You can use this automation to perform large-scale replication of RDS DB instances without any performance impact or operational overhead—regardless of the size of your organization. You can also use this pattern to help your organization comply with mandatory data governance strategies or compliance requirements that call for your data to be replicated and redundant across different AWS accounts and AWS Regions. Cross-account replication of Amazon RDS data at scale is an inefficient and error-prone manual process that can be costly and time-consuming, but the automation in this pattern can help you achieve cross-account replication safely, effectively, and efficiently. ## Prerequisites and limitations **Prerequisites** + Two AWS accounts + An RDS DB instance, up and running in the source AWS account + A subnet group for the RDS DB instance in the destination AWS account + An AWS Key Management Service (AWS KMS) key created in the source AWS account and shared with the destination account (For more information about policy details, see the [Additional information](#automate-the-replication-of-amazon-rds-instances-across-aws-accounts-additional) section of this pattern.) + An AWS KMS key in the destination AWS account to encrypt the database in the destination account **Limitations** + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service. **Product versions** + Python 3.9 (using AWS Lambda) + PostgreSQL 11.3, 13.x, and 14.x ## Architecture **Technology stack** + Amazon Relational Database Service (Amazon RDS) + Amazon Simple Notification Service (Amazon SNS) + AWS Key Management Service (AWS KMS) + AWS Lambda + AWS Secrets Manager + AWS Step Functions **Target architecture** The following diagram shows an architecture for using Step Functions to orchestrate scheduled, on-demand replication of RDS DB instances from a source account (account A) to a destination account (account B). ![\[Replicating Amazon RDS DB instances across source and destination accounts by using Step Functions.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6310ad9b-1b1a-4a67-b684-ef605fef3e87/images/001550bb-cf6b-493d-9de9-0229a43753a1.png) In the source account (account A in the diagram), the Step Functions state machine performs the following: 1. Creates a snapshot from the RDS DB instance in account A. 1. Copies and encrypts the snapshot with an AWS KMS key from account A. To ensure encryption in transit, the snapshot is encrypted whether or not the DB instance is encrypted. 1. Shares the DB snapshot with account B by giving account B access to the snapshot. 1. Pushes a notification to the SNS topic, and then the SNS topic invokes the Lambda function in account B. In the destination account (account B in the diagram), the Lambda function runs the Step Functions state machine to orchestrate the following: 1. Copies the shared snapshot from account A to account B, while using the AWS KMS key from account A to decrypt the data first and then encrypt the data by using the AWS KMS key in account B. 1. Reads the secret from Secrets Manager to capture the name of the current DB instance. 1. Restores the DB instance from the snapshot with a new name and default AWS KMS key for Amazon RDS. 1. Reads the endpoint of the new database and updates the secret in Secrets Manager with the new database endpoint, and then tags the previous DB instance so that it can be deleted later. 1. Keeps the latest N instances of the databases and deletes all the other instances. ## Tools **AWS services** + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. + [Amazon Simple Notification Service (Amazon SNS)](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses. + [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions. + [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. + [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html) is a software development kit that helps you integrate your Python application, library, or script with AWS services. + [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically. + [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) is a serverless orchestration service that helps you combine Lambda functions and other AWS services to build business-critical applications. **Code repository** The code for this pattern is available in the GitHub [Crossaccount RDS Replication](https://github.com/aws-samples/aws-rds-crossaccount-replication) repository. ## Epics ### Automate the replication of RDS DB instances across AWS accounts with a single click | Task | Description | Skills required | | --- | --- | --- | | Deploy the CloudFormation stack in the source account. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-the-replication-of-amazon-rds-instances-across-aws-accounts.html) | Cloud administrator, Cloud architect | | Deploy the CloudFormation stack in the destination account. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-the-replication-of-amazon-rds-instances-across-aws-accounts.html) | Cloud architect, DevOps engineer, Cloud administrator | | Verify the creation of the RDS DB instance in the destination account. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-the-replication-of-amazon-rds-instances-across-aws-accounts.html) | Cloud administrator, Cloud architect, DevOps engineer | | Subscribe the Lambda function to the SNS topic. | You must run the following AWS Command Line Interface (AWS CLI) commands to subscribe the Lambda function in the destination account (account B) to the SNS topic in the source account (account A).In account A, run following command:
def switch_traffic(event, context):
route53 = boto3.client('route53')
lag = check_replication_lag(event, context)
if lag == 0:
response = route53.change_resource_record_sets(
HostedZoneId='YOUR_HOSTED_ZONE_ID',
ChangeBatch={
'Changes': [
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': 'db.example.com',
'Type': 'CNAME',
'SetIdentifier': 'GreenCluster',
'Weight': 100,
'TTL': 60,
'ResourceRecords': [{'Value': 'green-cluster-endpoint'}]
}
},
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': 'db.example.com',
'Type': 'CNAME',
'SetIdentifier': 'BlueCluster',
'Weight': 0,
'TTL': 60,
'ResourceRecords': [{'Value': 'blue-cluster-endpoint'}]
}
}
]
}
)
return response
aws sns add-permission \In account B, run following command:
--label lambda-access --aws-account-id\
--topic-arn\
--action-name Subscribe ListSubscriptionsByTopic
aws lambda add-permission \In account B, run following command:
--function-name\
--source-arn\
--statement-id function-with-sns \
--action lambda:InvokeFunction \
--principal sns.amazonaws.com
aws sns subscribe \| Cloud administrator, Cloud architect, DBA | | Sync the RDS DB instance from the source account with the destination account. | Initiate the on-demand database replication by starting the Step Functions state machine in the source account.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-the-replication-of-amazon-rds-instances-across-aws-accounts.html)A scheduler is in place to help you run the replication automatically on schedule, but the scheduler is turned off by default. You can find the name of the Amazon CloudWatch rule for the scheduler in the **Resources** tab of the CloudFormation stack in the destination account. For instructions on how to modify the CloudWatch Events rule, see [Deleting or Disabling a CloudWatch Events Rule](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Delete-or-Disable-Rule.html) in the CloudWatch documentation. | Cloud architect, DevOps engineer, Cloud administrator | | Roll back your database to any of the previous copies when needed. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-the-replication-of-amazon-rds-instances-across-aws-accounts.html) | Cloud administrator, DBA, DevOps engineer | ## Related resources + [Cross-Region read replicas](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.CrossRegionReadReplicas.html) (Amazon RDS documentation) + [Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.BlueGreenDeployments.html) (Amazon RDS documentation) ## Additional information You can use the following example policy to share your AWS KMS key across AWS accounts. ``` { "Version": "2012-10-17", "Id": "cross-account-rds-kms-key", "Statement": [ { "Sid": "Enable user permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::
--protocol "lambda" \
--topic-arn\
--notification-endpoint
sqlcmd -S servername -U username -P password -i[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-database-tasks-in-sql-server-express-edition-running-on-amazon-ec2.html)
| AWS DevOps, DBA | | Create tasks in Task Scheduler. | To create a task in Task Scheduler, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-database-tasks-in-sql-server-express-edition-running-on-amazon-ec2.html)To run the task manually, right-click the newly created task and then select **Run**. | DBA | | View task status. | To view the status of a task in Task Scheduler, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-database-tasks-in-sql-server-express-edition-running-on-amazon-ec2.html) | DBA, AWS DevOps | ### Automate tasks with AWS Lambda | Task | Description | Skills required | | --- | --- | --- | | Implement the solution. | To implement this pattern’s solution, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-database-tasks-in-sql-server-express-edition-running-on-amazon-ec2.html) | AWS DevOps, DevOps engineer | ## Troubleshooting | Issue | Solution | | --- | --- | | Lambda issues | For help with errors and issues that you might encounter when using AWS Lambda, see [Troubleshooting issues in Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-troubleshooting.html) in the AWS documentation. | ## Related resources + [Amazon EC2 instance types](https://aws.amazon.com/ec2/instance-types/) + [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-eventbridge-scheduler.html) + [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/) + [Task Scheduler for developers](https://learn.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page) (Microsoft website) # Automate cross-Region failover and failback by using DR Orchestrator Framework *Jitendra Kumar, Pavithra Balasubramanian, and Oliver Francis, Amazon Web Services* ## Summary This pattern describes how to use [DR Orchestrator Framework](https://docs.aws.amazon.com/prescriptive-guidance/latest/automate-dr-solution-relational-database/dr-orchestrator-framework-overview.html) to orchestrate and automate the manual, error-prone steps to perform disaster recovery across Amazon Web Services (AWS) Regions. The pattern covers the following databases: + Amazon Relational Database Service (Amazon RDS) for MySQL, Amazon RDS for PostgreSQL, or Amazon RDS for MariaDB + Amazon Aurora MySQL-Compatible Edition or Amazon Aurora PostgreSQL-Compatible Edition (using a centralized file) + Amazon ElastiCache (Redis OSS) To demonstrate the functionality of DR Orchestrator Framework, you create two DB instances or clusters. The primary is in the AWS Region `us-east-1`, and the secondary is in `us-west-2`. To create these resources, you use the AWS CloudFormation templates in the `App-Stack` folder of the [aws-cross-region-dr-databases](https://github.com/aws-samples/aws-cross-region-dr-databases) GitHub repository. ## Prerequisites and limitations **General prerequisites** + DR Orchestrator Framework deployed in both primary and secondary AWS Regions + Two [Amazon Simple Storage Service](https://aws.amazon.com/s3/) buckets + A [virtual private cloud (VPC)](https://aws.amazon.com/vpc/) with two subnets and an AWS security group **Engine-specific prerequisites** + **Amazon Aurora** – At least one Aurora global database must be available in two AWS Regions. You can use `us-east-1` as the primary Region, and use `us-west-2` as the secondary Region. + **Amazon ElastiCache (Redis OSS)** – An ElastiCache global datastore must be available in two AWS Regions. You can `use us-east-1` as the primary Region, and use `us-west-2` as the secondary Region. **Amazon RDS limitations** + DR Orchestrator Framework doesn't check the replication lag before doing a failover or failback. Replication lag must be checked manually. + This solution has been tested using a primary database instance with one read replica. If you want to use more than one read replica, test the solution thoroughly before implementing it in a production environment. **Aurora limitations** + Feature availability and support vary across specific versions of each database engine and across AWS Regions. For more information on feature and Region availability for cross-Region replication, see [Cross-Region read replicas](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.CrossRegionReadReplicas.html). + Aurora global databases have specific configuration requirements for supported Aurora DB instance classes and the maximum number of AWS Regions. For more information, see [Configuration requirements of an Amazon Aurora global database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-getting-started.html#aurora-global-database.configuration.requirements). + This solution has been tested using a primary database instance with one read replica. If you want to use more than one read replica, test the solution thoroughly before implementing it in a production environment. **ElastiCache limitations** + For information about Region availability for Global Datastore and ElastiCache configuration requirements, see [Prerequisites and limitations](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Getting-Started.html) in the ElastiCache documentation. **Amazon RDS p****roduct versions** Amazon RDS supports the following engine versions: + **MySQL** – Amazon RDS supports DB instances running the following versions of [MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html): MySQL 8.0 and MySQL 5.7 + **PostgreSQL** – For information about supported versions of Amazon RDS for PostgreSQL, see [Available PostgreSQL database versions](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions). + **MariaDB** – Amazon RDS supports DB instances running the following versions of [MariaDB](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MariaDB.html): + MariaDB 10.11 + MariaDB 10.6 + MariaDB 10.5 **Aurora product versions** + Amazon Aurora global database switchover requires Aurora MySQL-Compatible with MySQL 5.7 compatibility, version 2.09.1 and higher For more information, see [Limitations of Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html#aurora-global-database.limitations). **ElastiCache (Redis OSS) product versions** Amazon ElastiCache (Redis OSS) supports the following Redis versions: + Redis 7.1 (enhanced) + Redis 7.0 (enhanced) + Redis 6.2 (enhanced) + Redis 6.0 (enhanced) + Redis 5.0.6 (enhanced) For more information, see [Supported ElastiCache (Redis OSS) versions](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Getting-Started.html). ## Architecture **Amazon RDS**** architecture** The Amazon RDS architecture includes the following resources: + The primary Amazon RDS DB instance created in the primary Region (`us-east-1`) with read/write access for clients + An Amazon RDS read replica created in the secondary Region (`us-west-2`) with read-only access for clients + DR Orchestrator Framework deployed in both the primary and secondary Regions ![\[Diagram of two-Region RDS architecture in a single AWS account.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8d39561f-924e-4b3e-8175-c5c3cab163bd/images/ad217033-600c-40da-929c-b9f9aecb4c2c.png) The diagram shows the following: 1. Asynchronous replication between the primary instance and the secondary instance 1. Read/write access for clients in the primary Region 1. Read-only access for clients in the secondary Region **Aurora architecture** The Amazon Aurora architecture includes the following resources: + The primary Aurora DB cluster created in the primary Region (`us-east-1`) with an active-writer endpoint + An Aurora DB cluster created in the secondary Region (`us-west-2`) with an inactive-writer endpoint + DR Orchestrator Framework deployed in both the primary and secondary Regions ![\[Diagram of two-Region Aurora deployment in a single AWS account.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8d39561f-924e-4b3e-8175-c5c3cab163bd/images/524ec002-5aa7-47b2-8c8d-6d1a3b535e9e.png) The diagram shows the following: 1. Asynchronous replication between the primary cluster and the secondary cluster 1. The primary DB cluster with an active-writer endpoint 1. The secondary DB cluster with an inactive-writer endpoint **ElastiCache (Redis OSS) architecture** The Amazon ElastiCache (Redis OSS) architecture includes the following resources: + An ElastiCache (Redis OSS) global datastore created with two clusters: 1. The primary cluster in the primary Region (`us-east-1`) 1. The secondary cluster in the secondary Region (`us-west-2`) + An Amazon cross-Region link with TLS 1.2 encryption between the two clusters + DR Orchestrator Framework deployed in both primary and secondary Regions ![\[Diagram of a two-Region ElastiCache deployment with Amazon cross-Region link.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8d39561f-924e-4b3e-8175-c5c3cab163bd/images/cf6620a0-dd42-4042-8dc2-012bf514ffc0.png) **Automation and scale** DR Orchestrator Framework is scalable and supports the failover or failback of more than one AWS database in parallel. You can use the following payload code to fail over multiple AWS databases in your account. In this example, three AWS databases (two global databases such as Aurora MySQL-Compatible or Aurora PostgreSQL-Compatible, and one Amazon RDS for MySQL instance) fail over to the DR Region: ``` { "StatePayload": [ { "layer": 1, "resources": [ { "resourceType": "PlannedFailoverAurora", "resourceName": "Switchover (planned failover) of Amazon Aurora global databases (MySQL)", "parameters": { "GlobalClusterIdentifier": "!Import dr-globaldb-cluster-mysql-global-identifier", "DBClusterIdentifier": "!Import dr-globaldb-cluster-mysql-cluster-identifier" } }, { "resourceType": "PlannedFailoverAurora", "resourceName": "Switchover (planned failover) of Amazon Aurora global databases (PostgreSQL)", "parameters": { "GlobalClusterIdentifier": "!Import dr-globaldb-cluster-postgres-global-identifier", "DBClusterIdentifier": "!Import dr-globaldb-cluster-postgres-cluster-identifier" } }, { "resourceType": "PromoteRDSReadReplica", "resourceName": "Promote RDS for MySQL Read Replica", "parameters": { "RDSInstanceIdentifier": "!Import rds-mysql-instance-identifier", "TargetClusterIdentifier": "!Import rds-mysql-instance-global-arn" } } ] } ] } ``` ## Tools **AWS services** + [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL. + [Amazon ElastiCache](https://docs.aws.amazon.com/elasticache/) helps you set up, manage, and scale distributed in-memory cache environments in the AWS Cloud. This pattern uses Amazon ElastiCache (Redis OSS). + [AWS Lambda](https://aws.amazon.com/lambda/) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. In this pattern, Lambda functions are used by AWS Step Functions to perform the steps. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. This pattern supports Amazon RDS for MySQL, Amazon RDS for PostgreSQL, and Amazon RDS for MariaDB. + [AWS SDK for Python (Boto3)](https://aws.amazon.com/sdk-for-python/) helps you integrate your Python application, library, or script with AWS services. In this pattern, Boto3 APIs are used to communicate with the database instances or global databases. + [AWS Step Functions](https://aws.amazon.com/step-functions/) is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications. In this pattern, Step Functions state machines are used to orchestrate and run the cross-Region failover and failback of the database instances or global databases. **Code repository** The code for this pattern is available in the [aws-cross-region-dr-databases](https://github.com/aws-samples/aws-cross-region-dr-databases/tree/main/App-Stack) repository on GitHub. ## Epics ### Install DR Orchestrator Framework | Task | Description | Skills required | | --- | --- | --- | | Clone the GitHub repository. | To clone the repository, run the following command:
@echo off
sqlcmd -S [ServerName] -d [DatabaseName] -U username -P password -i "PathToSQLScript\Script.sql" -o "PathToOutput\Output.txt"
git clone https://github.com/aws-samples/aws-cross-region-dr-databases.git| AWS DevOps, AWS administrator | | Package Lambda functions code in a .zip file archive. | Create the archive files for Lambda functions to include the DR Orchestrator Framework dependencies:
cd| AWS administrator | | Create S3 buckets. | S3 buckets are needed to store DR Orchestrator Framework along with your latest configuration. Create two S3 buckets, one in the primary Region (`us-east-1`), and one in the secondary Region (`us-west-2`):[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-cross-region-failover-and-failback-by-using-dr-orchestrator-framework.html)Replace `xxxxxx` with a random value to make the bucket names unique. | AWS administrator | | Create subnets and security groups. | In both the primary Region (`us-east-1`) and the secondary Region (`us-west-2`), create two subnets and one security group for Lambda function deployment in your VPC:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-cross-region-failover-and-failback-by-using-dr-orchestrator-framework.html) | AWS administrator | | Update the DR Orchestrator parameter files. | In the `/DR-Orchestration-artifacts
bash scripts/deploy-orchestrator-sh.sh
[| AWS administrator | | Upload the DR Orchestrator Framework code to the S3 bucket. | The code will be safer in an S3 bucket than in the local directory. Upload the `DR-Orchestration-artifacts` directory, including all files and subfolders, to the S3 buckets.To upload the code, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-cross-region-failover-and-failback-by-using-dr-orchestrator-framework.html) | AWS administrator | | Deploy DR Orchestrator Framework in the primary Region. | To deploy DR Orchestrator Framework in the primary Region (`us-east-1`), run the following commands:
{
"ParameterKey": "TemplateStoreS3BucketName",
"ParameterValue": "dr-orchestrator-xxxxxx-us-east-1"
},
{
"ParameterKey": "TemplateVPCId",
"ParameterValue": "vpc-xxxxxx"
},
{
"ParameterKey": "TemplateLambdaSubnetID1",
"ParameterValue": "subnet-xxxxxx"
},
{
"ParameterKey": "TemplateLambdaSubnetID2",
"ParameterValue": "subnet-yyyyyy"
},
{
"ParameterKey": "TemplateLambdaSecurityGroupID",
"ParameterValue": "sg-xxxxxxxxxx"
}
]
cd| AWS administrator | | Deploy DR Orchestrator Framework in the secondary Region. | In the secondary Region (`us-west-2`), run the following commands:/DR-Orchestration-artifacts/cloudformation
aws cloudformation deploy \
--region us-east-1 \
--stack-name dr-orchestrator \
--template-file Orchestrator-Deployer.yaml \
--parameter-overrides file://Orchestrator-Deployer-parameters-us-east-1.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
cd| AWS administrator | | Verify the deployment. | If the CloudFormation command runs successfully, it returns the following output:/DR-Orchestration-artifacts/cloudformation
aws cloudformation deploy \
--region us-west-2 \
--stack-name dr-orchestrator \
--template-file Orchestrator-Deployer.yaml \
--parameter-overrides file://Orchestrator-Deployer-parameters-us-west-2.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
Successfully created/updated stack - dr-orchestratorAlternatively, you can navigate to the CloudFormation console and verify the status of the `dr-orchestrator` stack. | AWS administrator | ### Create the database instances or clusters | Task | Description | Skills required | | --- | --- | --- | | Create the database subnets and security groups. | In your VPC, create two subnets and one security group for the DB instance or global database in both the primary (`us-east-1`) and the secondary (`us-west-2`) Regions:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automate-cross-region-failover-and-failback-by-using-dr-orchestrator-framework.html) | AWS administrator | | Update the parameter file for the primary DB instance or cluster. | In the `
{
"Parameters": {
"SubnetIds": "subnet-xxxxxx,subnet-xxxxxx",
"DBSecurityGroup": "sg-xxxxxxxxxx",
"MySqlGlobalIdentifier":"rds-mysql-instance",
"InitialDatabaseName": "mysqldb",
"DBPortNumber": "3789",
"PrimaryRegion": "us-east-1",
"SecondaryRegion": "us-west-2",
"KMSKeyAliasName": "rds/rds-mysql-instance-KmsKeyId"
}
}
**Amazon Aurora** In the `Aurora-MySQL-parameter-us-east-1.json` file, update `SubnetIds` and `DBSecurityGroup` with the names of resources that you created:{
"Parameters": {
"SubnetIds": "subnet1-xxxxxx,subnet2-xxxxxx",
"DBSecurityGroup": "sg-xxxxxxxxxx",
"GlobalClusterIdentifier":"dr-globaldb-cluster-mysql",
"DBClusterName":"dbcluster-01",
"SourceDBClusterName":"dbcluster-02",
"DBPortNumber": "3787",
"DBInstanceClass":"db.r5.large",
"InitialDatabaseName": "sampledb",
"PrimaryRegion": "us-east-1",
"SecondaryRegion": "us-west-2",
"KMSKeyAliasName": "rds/dr-globaldb-cluster-mysql-KmsKeyId"
}
}**Amazon ElastiCache (Redis OSS)**In the `ElastiCache-parameter-us-east-1.json` file, update `SubnetIds` and `DBSecurityGroup` with the names of resources that you created.{
"Parameters": {
"CacheNodeType": "cache.m5.large",
"DBSecurityGroup": "sg-xxxxxxxxxx",
"SubnetIds": "subnet-xxxxxx,subnet-xxxxxx",
"EngineVersion": "5.0.6",
"GlobalReplicationGroupIdSuffix": "demo-redis-global-datastore",
"NumReplicas": "1",
"NumShards": "1",
"ReplicationGroupId": "demo-redis-cluster",
"DBPortNumber": "3788",
"TransitEncryption": "true",
"KMSKeyAliasName": "elasticache/demo-redis-global-datastore-KmsKeyId",
"PrimaryRegion": "us-east-1",
"SecondaryRegion": "us-west-2"
}
} | AWS administrator |
| Deploy your DB instance or cluster in the primary Region. | To deploy your instance or cluster in the primary Region (`us-east-1`), run the following commands based on your database engine.**Amazon RDS**cd**Amazon Aurora**/App-Stack
aws cloudformation deploy \
--region us-east-1 \
--stack-name rds-mysql-app-stack \
--template-file RDS-MySQL-Primary.yaml \
--parameter-overrides file://RDS-MySQL-parameter-us-east-1.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
cd**Amazon ElastiCache (Redis OSS)**/App-Stack
aws cloudformation deploy \
--region us-east-1 \
--stack-name aurora-mysql-app-stack \
--template-file Aurora-MySQL-Primary.yaml \
--parameter-overrides file://Aurora-MySQL-parameter-us-east-1.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
cdVerify that the CloudFormation resources deployed successfully. | AWS administrator | | Update the parameter file for the secondary DB instance or cluster. | In the `/App-Stack
aws cloudformation deploy \
--region us-east-1 --stack-name elasticache-ds-app-stack \
--template-file ElastiCache-Primary.yaml \
--parameter-overrides file://ElastiCache-parameter-us-east-1.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
{
"Parameters": {
"SubnetIds": "subnet-aaaaaaaaa,subnet-bbbbbbbbb",
"DBSecurityGroup": "sg-cccccccccc",
"MySqlGlobalIdentifier":"rds-mysql-instance",
"InitialDatabaseName": "mysqldb",
"DBPortNumber": "3789",
"PrimaryRegion": "us-east-1",
"SecondaryRegion": "us-west-2",
"KMSKeyAliasName": "rds/rds-mysql-instance-KmsKeyId",
"PrimaryRegionKMSKeyArn":"arn:aws:kms:us-east-1:xxxxxxxxx:key/mrk-xxxxxxxxxxxxxxxxxxxxx"
}
} **Amazon Aurora**In the `Aurora-MySQL-parameter-us-west-2.json` file, update `SubnetIDs` and `DBSecurityGroup` with the names of resources you created. Update the `PrimaryRegionKMSKeyArn` with the value of `AuroraKmsKeyId` taken from the **Outputs** section of the CloudFormation stack for the primary DB instance:{
"Parameters": {
"SubnetIds": "subnet1-aaaaaaaaa,subnet2-bbbbbbbbb",
"DBSecurityGroup": "sg-cccccccccc",
"GlobalClusterIdentifier":"dr-globaldb-cluster-mysql",
"DBClusterName":"dbcluster-01",
"SourceDBClusterName":"dbcluster-02",
"DBPortNumber": "3787",
"DBInstanceClass":"db.r5.large",
"InitialDatabaseName": "sampledb",
"PrimaryRegion": "us-east-1",
"SecondaryRegion": "us-west-2",
"KMSKeyAliasName": "rds/dr-globaldb-cluster-mysql-KmsKeyId"
}
}**Amazon ElastiCache (Redis OSS)**In the `ElastiCache-parameter-us-west-2.json` file, update `SubnetIDs` and `DBSecurityGroup` with the names of resources that you created. Update the `PrimaryRegionKMSKeyArn` with the value of `ElastiCacheKmsKeyId` taken from the **Outputs** section of the CloudFormation stack for the primary DB instance:{
"Parameters": {
"CacheNodeType": "cache.m5.large",
"DBSecurityGroup": "sg-cccccccccc",
"SubnetIds": "subnet-aaaaaaaaa,subnet-bbbbbbbbb",
"EngineVersion": "5.0.6",
"GlobalReplicationGroupIdSuffix": "demo-redis-global-datastore",
"NumReplicas": "1",
"NumShards": "1",
"ReplicationGroupId": "demo-redis-cluster",
"DBPortNumber": "3788",
"TransitEncryption": "true",
"KMSKeyAliasName": "elasticache/demo-redis-global-datastore-KmsKeyId",
"PrimaryRegion": "us-east-1",
"SecondaryRegion": "us-west-2"
}
} | AWS administrator |
| Deploy your DB instance or cluster in the secondary Region. | Run the following commands, based on your database engine.**Amazon RDS**cd**Amazon Aurora**/App-Stack
aws cloudformation deploy \
--region us-west-2 \
--stack-name rds-mysql-app-stack \
--template-file RDS-MySQL-DR.yaml \
--parameter-overrides file://RDS-MySQL-parameter-us-west-2.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
cd**Amazon ElastiCache (Redis OSS)**/App-Stack
aws cloudformation deploy \
--region us-west-2 \
--stack-name aurora-mysql-app-stack \
--template-file Aurora-MySQL-DR.yaml \
--parameter-overrides file://Aurora-MySQL-parameter-us-west-2.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
cdVerify that the CloudFormation resources deployed successfully. | AWS administrator | ## Related resources + [Disaster recovery strategy for databases on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-database-disaster-recovery/welcome.html) (AWS Prescriptive Guidance strategy) + [Automate your DR solution for relational databases on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/automate-dr-solution-relational-database/dr-orchestrator-framework-overview.html) (AWS Prescriptive Guidance guide) + [Using Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) + [Replication across AWS Regions using global datastores](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastore.html) + [Automate your DR solution for relational databases on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/automate-dr-solution-relational-database/introduction.html) (AWS Prescriptive Guidance guide) # Automatically back up SAP HANA databases using Systems Manager and EventBridge *Ambarish Satarkar and Gaurav Rath, Amazon Web Services* ## Summary This pattern describes how to automate SAP HANA database backups using AWS Systems Manager, Amazon EventBridge, Amazon Simple Storage Service (Amazon S3), and AWS Backint Agent for SAP HANA. This pattern provides a shell script-based approach using the `BACKUP DATA` command and removes the need to maintain scripts and job configurations for each operating system (OS) instance across numerous systems. | | | Note: As of April 2023, AWS Backup announced support for SAP HANA databases on Amazon Elastic Compute Cloud (Amazon EC2). For more information, see [SAP HANA databases on Amazon EC2 instances backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/backup-saphana.html).Based on your organization’s needs, you can use the AWS Backup service to automatically back up your SAP HANA databases or you can use this pattern. | | --- | ## Prerequisites and limitations **Prerequisites ** + An existing SAP HANA instance with a supported release in running state on a managed Amazon Elastic Compute Cloud (Amazon EC2) instance that is configured for Systems Manager + Systems Manager Agent (SSM Agent) 2.3.274.0 or later installed + An S3 bucket that doesn’t have public access enabled + An `hdbuserstore` key named `SYSTEM` + An AWS Identity and Access Management (IAM) role for the Automation runbook to run on schedule + `AmazonSSMManagedInstanceCore` and `ssm:StartAutomationExecution` policies are attached to Systems Manager Automation service role. **Limitations ** + AWS Backint Agent for SAP HANA doesn't support deduplication. + AWS Backint Agent for SAP HANA doesn't support data compression. **Product versions** AWS Backint Agent is supported on the following operating systems: + SUSE Linux Enterprise Server + SUSE Linux Enterprise Server for SAP + Red Hat Enterprise Linux for SAP AWS Backint Agent supports the following databases: + SAP HANA 1.0 SP12 (single node and multiple nodes) + SAP HANA 2.0 and later (single node and multiple nodes) ## Architecture **Target technology stack** + AWS Backint Agent + Amazon S3 + AWS Systems Manager + Amazon EventBridge + SAP HANA **Target architecture** The following diagram shows the installation scripts that install AWS Backint Agent, the S3 bucket, and Systems Manager and EventBridge, which use a Command document to schedule regular backups. ![\[Workflow for scheduling regular backups.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/0aa22a27-d100-483d-95f9-c3101f40402c/images/201d2b9a-b88e-4432-82cd-240b81da981e.png) **Automation and scale** + Multiple AWS Backint Agents can be installed by using a Systems Manager Automation runbook. + Each run of the Systems Manager runbook can scale to *n* number of SAP HANA instances, based on target selection. + EventBridge can automate SAP HANA backups. ## Tools + [AWS Backint Agent for SAP HANA](https://docs.aws.amazon.com/sap/latest/sap-hana/aws-backint-agent-sap-hana.html) is a standalone application that integrates with your existing workflows to back up your SAP HANA database to an S3 bucket that you specify in the configuration file. AWS Backint Agent supports full, incremental, and differential backups of SAP HANA databases. It runs on an SAP HANA database server, where backups and catalogs are transferred from the SAP HANA database to the AWS Backint Agent. + [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) is a serverless event bus service that you can use to connect your applications with data from a variety of sources. EventBridge delivers a stream of real-time data from your applications, software as a service (SaaS) applications, and AWS services to targets such as AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other accounts. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is an object storage service. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. + [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) helps you to view and control your infrastructure on AWS. Using the Systems Manager console, you can view operational data from multiple AWS services and automate operational tasks across your AWS resources. **Code** The code for this pattern is available in the [aws-backint-automated-backup](https://github.com/aws-samples/aws-backint-automated-backup) GitHub repository. ## Epics ### Create an hdbuserstore key SYSTEM | Task | Description | Skills required | | --- | --- | --- | | Create an hdbuserstore key. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-back-up-sap-hana-databases-using-systems-manager-and-eventbridge.html) | AWS administrator, SAP HANA Administrator | ### Install AWS Backint Agent | Task | Description | Skills required | | --- | --- | --- | | Install AWS Backint Agent. | Follow the instructions in [Install and configure AWS Backint Agent for SAP HANA](https://docs.aws.amazon.com/sap/latest/sap-hana/aws-backint-agent-installing-configuring.html) in the AWS Backint Agent documentation. | AWS administrator, SAP HANA administrator | ### Create the Systems Manager Command document | Task | Description | Skills required | | --- | --- | --- | | Create the Systems Manager Command document. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-back-up-sap-hana-databases-using-systems-manager-and-eventbridge.html) | AWS administrator, SAP HANA administrator | ### Schedule backups on a regular frequency | Task | Description | Skills required | | --- | --- | --- | | Schedule regular backups using Amazon EventBridge. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-back-up-sap-hana-databases-using-systems-manager-and-eventbridge.html)You can verify backup success from the S3 bucket path./App-Stack
aws cloudformation deploy \
--region us-west-2 \
--stack-name elasticache-ds-app-stack \
--template-file ElastiCache-DR.yaml \
--parameter-overrides file://ElastiCache-parameter-us-west-2.json \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM \
--disable-rollback
s3:/You can also verify backups from the SAP HANA backup catalog. | AWS administrator, SAP HANA administrator | ## Related resources + [AWS Backint Agent for SAP HANA](https://docs.aws.amazon.com/sap/latest/sap-hana/aws-backint-agent-sap-hana.html) + [Install and configure AWS Backint Agent for SAP HANA](https://docs.aws.amazon.com/sap/latest/sap-hana/aws-backint-agent-installing-configuring.html) # Automatically generate a PynamoDB model and CRUD functions for Amazon DynamoDB by using a Python application *Vijit Vashishtha, Dheeraj Alimchandani, and Dhananjay Karanjkar, Amazon Web Services* ## Summary It's common to require entities and create, read, update, and delete (CRUD) operations functions to efficiently perform Amazon DynamoDB database operations. PynamoDB is a Python-based interface that supports Python 3. It also provides features such as support for Amazon DynamoDB transactions, automatic attribute value serialization and deserialization, and compatibility with common Python frameworks, such as Flask and Django. This pattern helps developers working with Python and DynamoDB by providing a library that streamlines the automatic creation of PynamoDB models and CRUD operation functions. While it generates essential CRUD functions for database tables, it can also reverse engineer PynamoDB models and CRUD functions from Amazon DynamoDB tables. This pattern is designed to simplify database operations by using a Python-based application. The following are the key features of this solution: + **JSON schema to PynamoDB model** – Automatically generate PynamoDB models in Python by importing a JSON schema file. + **CRUD function generation** – Automatically generate functions to perform CRUD operations on DynamoDB tables. + **Reverse engineering from DynamoDB** – Use PynamoDB object-relational mapping (ORM) to reverse engineer PynamoDB models and CRUD functions for existing Amazon DynamoDB tables. ## Prerequisites and limitations **Prerequisites** + An active AWS account + Python version 3.8 or later, [downloaded](https://www.python.org/downloads/) and installed + Jinja2 version 3.1.2 or later, [downloaded](https://pypi.org/project/Jinja2/#files) and installed + Amazon DynamoDB tables for which you want to generate ORM + AWS Command Line Interface (AWS CLI), [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) + PynamoDB version 5.4.1 or later, [installed](https://pynamodb.readthedocs.io/en/stable/tutorial.html#installation) ## Architecture **Target technology stack** + JSON script + Python application + PynamoDB model + Amazon DynamoDB database instance **Target architecture** ![\[Using a Python app to generate CRUD functions and PynamoDB model from DynamoDB tables.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/74cc4c73-5c8b-448d-98fb-b681cfa5f860/images/c2c367d6-d88a-4f49-8571-89160539eb08.png) 1. You create an input JSON schema file. This JSON schema file represents the attributes of the respective DynamoDB tables that you want to create PynamoDB models from and CRUD functions for. It contains the following three important keys: + `name` –The name of the target DynamoDB table. + `region` – The AWS Region where the table is hosted + `attributes` – The attributes that are part of the target table, such as the [partition key](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.PrimaryKey) (also known as a *hash attribute*), [sort key](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.PrimaryKey), [local secondary indexes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html), [global secondary indexes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html), and any [non-key attributes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.TablesItemsAttributes). This tool expects the input schema to only provide the non-key attributes as the application fetches the key attributes directly from the target table. For an example of how to specify attributes in the JSON schema file, see the [Additional information](#automatically-generate-a-pynamodb-model-and-crud-functions-for-amazon-dynamodb-by-using-a-python-application-additional) section of this pattern. 1. Run the Python application and provide the JSON schema file as an input. 1. The Python application reads the JSON schema file. 1. The Python application connects to the DynamoDB tables to derive the schema and data types. The application runs the [describe\$1table](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/describe_table.html) operation and fetches the key and index attributes for the table. 1. The Python application combines the attributes from the JSON schema file and DynamoDB table. It uses the Jinja template engine to generate a PynamoDB model and corresponding CRUD functions. 1. You access the PynamoDB model to perform CRUD operations on the DynamoDB table. ## Tools **AWS services** + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance. **Other tools** + [Jinja](https://jinja.palletsprojects.com/en/) is an extensible templating engine that compiles templates into optimized Python code. This pattern uses Jinja to to generate dynamic content by embedding placeholders and logic within templates. + [PynamoDB](https://pynamodb.readthedocs.io/en/stable/) is a Python-based interface for Amazon DynamoDB. + [Python](https://www.python.org/) is a general-purpose computer programming language. **Code repository** The code for this pattern is available in the GitHub [Auto-generate PynamoDB models and CRUD functions](https://github.com/aws-samples/amazon-reverse-engineer-dynamodb) repository. The repository is divided into two main parts: the controller package and the templates. *Controller package* The controller Python package contains the main application logic that helps generate the PynamoDB model and the CRUD functions. It contains the following: + `input_json_validator.py` – This Python scripts validates the input JSON schema file and creates the Python objects that contain the list of target DynamoDB tables and the required attributes for each. + `dynamo_connection.py` – This script establishes a connection to the DynamoDB table and uses the `describe_table` operation to extract the attributes that are necessary to create the PynamoDB model. + `generate_model.py` – This script contains a Python class `GenerateModel` that creates the PynamoDB model based on the input JSON schema file and the `describe_table` operation. + `generate_crud.py` – For the DynamoDB tables that are defined in the JSON schema file, this script uses the `GenerateCrud` operation to create the Python classes. *Templates* This Python directory contains the following Jinja templates: + `model.jinja` – This Jinja template contains the template expression for generating the PynamoDB model script. + `crud.jinja` – This Jinja template contains the template expression for generating the CRUD functions script. ## Epics ### Set up the environment | Task | Description | Skills required | | --- | --- | --- | | Clone the repository. | Enter the following command to clone the [Auto-generate PynamoDB models and CRUD functions](https://github.com/aws-samples/amazon-reverse-engineer-dynamodb) repository./ / /usr/sap/ /SYS/global/hdb/backint/DB_ /
git clone https://github.com/aws-samples/amazon-reverse-engineer-dynamodb.git| App developer | | Set up the Python environment. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-generate-a-pynamodb-model-and-crud-functions-for-amazon-dynamodb-by-using-a-python-application.html) | App developer | ### Generate the PynamoDB model and CRUD functions | Task | Description | Skills required | | --- | --- | --- | | Modify the JSON schema file. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-generate-a-pynamodb-model-and-crud-functions-for-amazon-dynamodb-by-using-a-python-application.html) | App developer | | Run the Python application. | Enter the following command to generate the PynamoDB models and CRUD functions, where `
python main.py --file| App developer | ### Verify the PynamoDB model and CRUD functions | Task | Description | Skills required | | --- | --- | --- | | Verify the generated PynamoDB model. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-generate-a-pynamodb-model-and-crud-functions-for-amazon-dynamodb-by-using-a-python-application.html) | App developer | | Verify the generated CRUD functions. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-generate-a-pynamodb-model-and-crud-functions-for-amazon-dynamodb-by-using-a-python-application.html) | App developer | ## Related resources + [Core components of Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html) (DynamoDB documentation) + [Improving data access with secondary indexes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html) (DynamoDB documentation) ## Additional information **Sample attributes for the JSON schema file** ``` [ { "name": "test_table", "region": "ap-south-1", "attributes": [ { "name": "id", "type": "UnicodeAttribute" }, { "name": "name", "type": "UnicodeAttribute" }, { "name": "age", "type": "NumberAttribute" } ] } ] ``` # Block public access to Amazon RDS by using Cloud Custodian *abhay kumar and Dwarika Patra, Amazon Web Services* ## Summary Many organizations run their workloads and services on multiple cloud vendors. In these hybrid cloud environments, the cloud infrastructure needs strict cloud governance, in addition to the security provided by the individual cloud providers. A cloud database such as Amazon Relational Database Service (Amazon RDS) is one important service that must be monitored for any access and permission vulnerabilities. Although you can restrict access to the Amazon RDS database by configuring a security group, you can add a second layer of protection to prohibit actions such as public access. Blocking public access helps you with General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPAA), National Institute of Standards and Technology (NIST), and Payment Card Industry Data Security Standard (PCI DSS) compliance. Cloud Custodian is an open-source rules engine that you can use to enforce access restrictions for Amazon Web Services (AWS) resources such as Amazon RDS. With Cloud Custodian, you can set rules that validate the environment against defined security and compliance standards. You can use Cloud Custodian to manage your cloud environments by helping to ensure compliance with security policies, tag policies, and garbage collection of unused resources and cost management. With Cloud Custodian, you can use a single interface for implementing governance in a hybrid cloud environment. For example, you could use the Cloud Custodian interface to interact with AWS and Microsoft Azure, reducing the effort of working with mechanisms such as AWS Config, AWS security groups, and Azure policies. This pattern provides instructions for using Cloud Custodian on AWS to enforce restriction of public accessibility on Amazon RDS instances. ## Prerequisites and limitations **Prerequisites** + An active AWS account + [A key pair](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds-create) + AWS Lambda installed ## Architecture The following diagram shows Cloud Custodian deploying the policy to AWS Lambda, AWS CloudTrail initiating the `CreateDBInstance` event, and the Lambda function setting `PubliclyAccessible` to false on Amazon RDS. ![\[Using Cloud Custodian on AWS to restrict public access to Amazon RDS instances.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/90f9537e-9365-4da2-8a28-da0ff374743c/images/6d04ca3b-6aa4-4c62-ade9-8b7474928c5e.png) ## Tools **AWS services** + [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html) helps you audit the governance, compliance, and operational risk of your AWS account. + [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command line shell. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. **Other tools** + [Cloud Custodian](https://cloudcustodian.io/) unifies the tools and scripts that many organizations use to manage their public cloud accounts into one open source tool. It uses a stateless rules engine for policy definition and enforcement, with metrics, structured outputs, and detailed reporting for cloud infrastructure. It integrates tightly with a serverless runtime to provide real-time remediation and response with low operational overhead. ## Epics ### Set up the AWS CLI | Task | Description | Skills required | | --- | --- | --- | | Install the AWS CLI. | To install the AWS CLI, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html). | AWS administrator | | Set up AWS credentials. | Configure the settings that the AWS CLI uses to interact with AWS, including the AWS Region and the output format that you want to use.
$>aws configureFor more information, see the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). | AWS administrator | | Create an IAM role. | To create an IAM role with the Lambda execution role, run the following command.
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]:
Default output format [None]:
aws iam create-role --role-name lambda-ex --assume-role-policy-document '{"Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]} | AWS DevOps |
### Set up Cloud Custodian
| Task | Description | Skills required |
| --- | --- | --- |
| Install Cloud Custodian. | To install Cloud Custodian for your operating system and environment, follow the instructions in the [Cloud Custodian documentation](https://cloudcustodian.io/docs/quickstart/index.html#install-cc). | DevOps engineer |
| Check the Cloud Custodian schema. | To see the complete list of Amazon RDS resources against which you can run policies, use the following command.custodian schema aws.rds| DevOps engineer | | Create the Cloud Custodian policy. | Save the code that’s under *Cloud Custodian policy file* in the [Additional information](#block-public-access-to-amazon-rds-by-using-cloud-custodian-additional) section using a YAML extension. | DevOps engineer | | Define Cloud Custodian actions to change the publicly accessible flag. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/block-public-access-to-amazon-rds-by-using-cloud-custodian.html) | DevOps engineer | | Perform a dry run. | (Optional) To check which resources are identified by the policy without running any actions on the resources, use the following command.
custodian run -dryrun| DevOps engineer | ### Deploy the policy | Task | Description | Skills required | | --- | --- | --- | | Deploy the policy by using Lambda. | To create the Lambda function that will run the policy, use the following command..yaml -s
custodian run -s policy.yamlThis policy will then be initiated by the AWS CloudTrail `CreateDBInstance` event.As a result, AWS Lambda will set the publicly accessible flag to `false` for instances that match the criteria. | DevOps engineer | ## Related resources + [AWS Lambda website](https://aws.amazon.com/lambda/) + [Amazon RDS website](https://aws.amazon.com/rds/) + [Cloud Custodian documentation](https://cloudcustodian.io/docs/quickstart/index.html) ## Additional information **Cloud Custodian policy YAML file** ``` policies: - name: "block-public-access" resource: rds description: | This Enforcement blocks public access for RDS instances. mode: type: cloudtrail events: - event: CreateDBInstance # Create RDS instance cloudtrail event source: rds.amazonaws.com ids: requestParameters.dBInstanceIdentifier role: arn:aws:iam::1234567890:role/Custodian-compliance-role filters: - type: event key: 'detail.requestParameters.publiclyAccessible' value: true actions: - type: set-public-access state: false ``` **c7n resources rds.py file** ``` @actions.register('set-public-access') class RDSSetPublicAvailability(BaseAction): schema = type_schema( "set-public-access", state={'type': 'boolean'}) permissions = ('rds:ModifyDBInstance',) def set_accessibility(self, r): client = local_session(self.manager.session_factory).client('rds') waiter = client.get_waiter('db_instance_available') waiter.wait(DBInstanceIdentifier=r['DBInstanceIdentifier']) client.modify_db_instance( DBInstanceIdentifier=r['DBInstanceIdentifier'], PubliclyAccessible=self.data.get('state', False)) def process(self, rds): with self.executor_factory(max_workers=2) as w: futures = {w.submit(self.set_accessibility, r): r for r in rds} for f in as_completed(futures): if f.exception(): self.log.error( "Exception setting public access on %s \n %s", futures[f]['DBInstanceIdentifier'], f.exception()) return rds ``` **Security Hub CSPM integration** Cloud Custodian can be integrated with [AWS Security Hub CSPM](https://aws.amazon.com/security-hub/) to send security findings and attempt remediation actions. For more information, see [Announcing Cloud Custodian Integration with AWS Security Hub CSPM](https://aws.amazon.com/blogs/opensource/announcing-cloud-custodian-integration-aws-security-hub/). # Configure Windows authentication for Amazon RDS for Microsoft SQL Server using AWS Managed Microsoft AD *Ramesh Babu Donti, Amazon Web Services* ## Summary This pattern shows how to configure Windows authentication for an Amazon Relational Database Service (Amazon RDS) for SQL Server instances using AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD). Windows authentication allows users to connect to the RDS instance using their domain credentials instead of database-specific usernames and passwords. You can enable Windows authentication either when creating a new RDS SQL Server database or by adding it to an existing database instance. The database instance integrates with AWS Managed Microsoft AD to provide centralized authentication and authorization for domain users accessing the SQL Server database. This configuration enhances security by leveraging existing Active Directory infrastructure and eliminates the need to manage separate database credentials for domain users. ## Prerequisites and limitations **Prerequisites** + An active AWS account with appropriate permissions + A virtual private cloud (VPC) with the following: + Configured Internet gateways and route tables + NAT gateways in public subnets (if internet access is required for instances) + AWS Identity and Access Management (IAM) roles: + A domain role with the following AWS managed policies: + `AmazonSSMManagedInstanceCore` to enable AWS Systems Manager + `AmazonSSMDirectoryServiceAccess` to provide permissions to join instances to directories + An RDS enhanced monitoring role (if enhanced monitoring is enabled) + Security groups: + Directory service security group to allow Active Directory communication ports + An Amazon Elastic Compute Cloud (Amazon EC2) security group to allow RDP `3389` and domain communications + An RDS security group to allow SQL Server port `1433` from authorized sources + Network connectivity: + Proper DNS resolution and network connectivity between subnets **Limitations** + For information about AWS Regions that support AWS Managed Microsoft AD with RDS for SQL Server, see [Region and version availability](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServerWinAuth.html#USER_SQLServerWinAuth.RegionVersionAvailability). + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service. ## Architecture **Source technology stack** + An on-premises Active Directory or AWS Managed Microsoft AD **Target technology stack** + Amazon EC2 + Amazon RDS for Microsoft SQL Server + AWS Managed Microsoft AD **Target architecture** ![\[alt text not found\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e02f6059-6631-46f6-819c-5961af7ba4ae/images/1aa50e3b-b4f6-4d44-9f9e-6cbb248a159c.png) The architecture includes the following: + An IAM role that joins the Amazon EC2 instance to the AWS Managed Microsoft AD domain. + An Amazon EC2 Windows instance for database administration and testing. + An Amazon VPC with a private subnet to host the Amazon RDS instance and internal resources across Availability Zones. + Security groups for network access control: + An Amazon RDS security group to control inbound access to SQL Server port `1433` from authorized sources. + An Amazon EC2 security group to manage RDP access through port `3389` and domain communication ports. + A Directory Services security group for Active Directory communications through ports `53`, `88`, `389`, and `445`. + AWS Managed Microsoft AD to provide centralized authentication and authorization services for Windows resources. + An Amazon RDS for SQL Server database instance in the private subnet with Windows authentication enabled. ## Tools **AWS services** + [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down. + Amazon Relational Database Service (Amazon RDS) helps you set up, operate, and scale a relational database in the AWS Cloud. + [AWS Directory Service](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/what_is.html) provides multiple ways to use Microsoft Active Directory (AD) with other AWS services such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS) for SQL Server, and Amazon FSx for Windows File Server. + [AWS Directory Service for Microsoft Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) enables your directory-aware workloads and AWS resources to use Microsoft Active Directory in the AWS Cloud. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. **Other services** + [Microsoft SQL Server Management Studio (SSMS)](https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms) is a tool for managing SQL Server, including accessing, configuring, and administering SQL Server components. ## Best practices + For general best practices, see [Best practices for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html). ## Epics ### Configure AWS Managed Microsoft AD | Task | Description | Skills required | | --- | --- | --- | | Set up the directory type. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure directory information. | In the **Directory information** section, enter the required information, and keep the optional values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure the VPC and subnets. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Review and create the directory. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | ### Create and configure an Amazon EC2 instance | Task | Description | Skills required | | --- | --- | --- | | Configure an AMI for Windows. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure network settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure storage. | Configure the Amazon EBS volumes as needed. | DBA, DevOps engineer | | Configure advanced details and launch the instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | ### Create and configure RDS for SQL Server with Windows authentication | Task | Description | Skills required | | --- | --- | --- | | Create a database and configure engine options. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Choose a template. | Choose a sample template that meets your requirements. | DBA, DevOps engineer | | Configure database settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure the instance. | In the **Instance configuration** section, under **DB instance class**, select an instance size that meets your requirements. | DBA, DevOps engineer | | Configure storage. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure connectivity. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure Windows authentication. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure monitoring. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html)Note: Metrics are useful when you want to see how different processes or threads use the CPU. You can also export error logs to Amazon CloudWatch if **Error log** is enabled. | DBA, DevOps engineer | | Configure additional settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Review costs and create database. | Review the **Estimated monthly costs** section, and then choose **Create database**. | DBA, DevOps engineer | ### Configure database access and test connections | Task | Description | Skills required | | --- | --- | --- | | Connect to the Windows machine. | Connect to your Windows machine and launch SQL Server Management Studio.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure the SSMS connection. | Set up the database connection using Windows authentication.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Configure security settings. | Set the required security parameters for SSMS version 20 or later.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | | Create a Windows login. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html)
CREATE LOGIN [| DBA, DevOps engineer | | Test Windows authentication. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-windows-authentication-for-amazon-rds-using-microsoft-ad.html) | DBA, DevOps engineer | ## Related resources + [Creating your AWS Managed Microsoft AD](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_getting_started_create_directory.html) + [Joining an Amazon EC2 Windows instance to your AWS Managed Microsoft AD](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/launching_instance.html) + [Supported Regions and DB engines for Kerberos authentication in Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.KerberosAuthentication.html) + [What is SQL Server Management Studio (SSMS)?](https://learn.microsoft.com/en-us/ssms/sql-server-management-studio-ssms) # Configure cross-account access to Amazon DynamoDB *Shashi Dalmia, Imhoertha Ojior, and Esteban Serna Parra, Amazon Web Services* ## Summary This pattern explains the steps for configuring cross-account access to Amazon DynamoDB by using resource-based policies. For workloads that use DynamoDB, it's becoming more common to use [workload isolation strategies](https://aws.amazon.com/solutions/guidance/workload-isolation-on-aws/?did=sl_card&trk=sl_card) to minimize security threats and to meet compliance requirements. Implementing workload isolation strategies often requires cross-account and cross-Region access to DynamoDB resources by using AWS Identity and Access Management (IAM) identity-based policies. This involves setting IAM permissions and establishing a trust relationship between the AWS accounts. [Resource-based policies for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/access-control-resource-based.html) greatly simplify the security posture for cross-account workloads. This pattern provides steps and sample code to demonstrate how you can configure AWS Lambda functions in one AWS account to write data to a DynamoDB database table in a different account. ## Prerequisites and limitations **Prerequisites** + Two active AWS accounts. This pattern refers to these accounts as *Account A* and *Account B*. + AWS Command Line Interface (AWS CLI) [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to access Account A, to create the DynamoDB table. The other steps in this pattern provide instructions for using the IAM, DynamoDB, and Lambda consoles. If you’re planning to use AWS CLI instead, configure it to access both accounts. **Limitations** + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service. ## Architecture The following diagram shows a single-account architecture. AWS Lambda, Amazon Elastic Compute Cloud (Amazon EC2), and DynamoDB are all in the same account. In this scenario, Lambda functions and Amazon EC2 instances can access DynamoDB. To grant access to the DynamoDB table, you can create an identity-based policy in IAM, or you can create a resource-based policy in DynamoDB. ![\[Using IAM permissions to access a DynamoDB table in the same account.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bfc32fe8-5db0-4cac-a30f-b870a1a82875/images/cbb009eb-422d-4833-a1bc-0c571d83c21f.png) The following diagram shows a multi-account architecture. If resources in one AWS account require access to a DynamoDB table in a different account, you need to set up a resource-based policy in DynamoDB to grant the required access. For example, in the following diagram, access to the DynamoDB table in Account A is granted to a Lambda function in Account B by using a resource-based policy. ![\[Using a resource-based policy to access a DynamoDB table in a different account.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/bfc32fe8-5db0-4cac-a30f-b870a1a82875/images/9f9165a8-b767-4427-a2ae-31b5b8c83326.png) This pattern describes cross-account access between Lambda and DynamoDB. You can use similar steps for other AWS services if the appropriate permissions are configured on both accounts. For example, if you want to provide a Lambda function access to an Amazon Simple Storage Service (Amazon S3) bucket in Account A, you can create a [resource-based policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html) in Amazon S3 and add the permissions to the [Lambda execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html) in Account B. ## Tools **AWS services** + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. **Code** This pattern includes sample code in the [Additional information](#configure-cross-account-access-to-amazon-dynamodb-additional) section to show how you can configure a Lambda function in Account B to write to the DynamoDB table in Account A. The code is provided only for illustration and testing purposes. If you’re implementing this pattern in a production environment, use the code as a reference, and customize it for your own environment. ## Best practices + Follow the [best practices for resource-based policies](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/rbac-best-practices.html) in the DynamoDB documentation. + Follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation. ## Epics ### Create an IAM policy and role for the Lambda function in Account B | Task | Description | Skills required | | --- | --- | --- | | Create a policy in Account B. | This IAM policy allows the [PutItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html) action for a DynamoDB table in Account A.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html) | General AWS | | Create a role in Account B. | The Lambda function in Account B uses this IAM role to access the DynamoDB table in Account A.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html)For more information about creating roles, see the [IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html). | General AWS | | Note the role ARN. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html) | General AWS | ### Create a DynamoDB table in Account A | Task | Description | Skills required | | --- | --- | --- | | Create a DynamoDB table. | Use the following AWS CLI command to create a DynamoDB table.\ ] FROM WINDOWS;
GO
aws dynamodb create-table \Replace the following in this code sample:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html)You specify the resource-based policy configuration in the `create-table` statement by using the `--resource-policy` flag. This policy refers to the ARN for the DynamoDB table in Account A.For more information about creating tables, see the [DynamoDB documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html). | General AWS | ### Create a Lambda function in Account B | Task | Description | Skills required | | --- | --- | --- | | Create a Lambda function to write data to DynamoDB. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html)For more information about creating Lambda functions, see the [Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html). | General AWS | ### Clean up | Task | Description | Skills required | | --- | --- | --- | | Delete resources. | To avoid incurring costs associated with the resources created in this pattern, do the following to delete these resources:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/configure-cross-account-access-to-amazon-dynamodb.html) | General AWS | ## Troubleshooting | Issue | Solution | | --- | --- | | When creating the Lambda function, you receive a `ResourceNotFoundException` error. | Confirm that you have correctly entered the AWS Region and ID of Account A. These are part of the ARN for the DynamoDB table. | ## Related resources + [Getting started with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html) (DynamoDB documentation) + [Getting started with Lambda](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html) (Lambda documentation) + [Using resource-based policies for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/access-control-resource-based.html) (DynamoDB documentation) + [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) (IAM documentation) + [Cross-account policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic-cross-account.html) (IAM documentation) + [IAM JSON policy elements reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html) (IAM documentation) ## Additional information *Sample code* ``` import boto3 from datetime import datetime dynamodb_client = boto3.client('dynamodb') def lambda_handler(event, context): now = datetime.now().isoformat() data = dynamodb_client.put_item(TableName='arn:aws:dynamodb:
--table-name Table-Account-A \
--attribute-definitions \
AttributeName=category,AttributeType=S \
AttributeName=item,AttributeType=S \
--key-schema \
AttributeName=category,KeyType=HASH \
AttributeName=item,KeyType=RANGE \
--provisioned-throughput \
ReadCapacityUnits=5,WriteCapacityUnits=5 \
--resource-policy \
'{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:::role/ "
},
"Action": "dynamodb:PutItem",
"Resource": "arn:aws:dynamodb:: :table/Table-Account-A"
}
]
}'
SELECT SERVERPROPERTY('ComputernamePhysicalNetBios') .The output will display the current secondary replica name (`WSFCNODE1`). | DBA |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Creating the listener fails with the message 'The WSFC cluster could not bring the Network Name resource online'. | For information, see the Microsoft blog post [Create Listener Fails with Message 'The WSFC cluster could not bring the Network Name resource online'](https://techcommunity.microsoft.com/blog/sqlserversupport/create-listener-fails-with-message-the-wsfc-cluster-could-not-bring-the-network-/318235). |
| Potential issues, including other listener issues or network access issues. | See [Troubleshoot Always On Availability Groups Configuration (SQL Server)](https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/troubleshoot-always-on-availability-groups-configuration-sql-server?view=sql-server-ver16) in the Microsoft documentation. |
## Related resources
+ [Configure read-only routing for an Always On availability group](https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/configure-read-only-routing-for-an-availability-group-sql-server?view=sql-server-ver16)
+ [Troubleshoot Always On Availability Groups Configuration (SQL Server)](https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/troubleshoot-always-on-availability-groups-configuration-sql-server?view=sql-server-ver16)
## Additional information
*Step 1. Update the replicas to read-only*
```
ALTER AVAILABILITY GROUP [SQLAG1] MODIFY REPLICA ON N'WSFCNODE1' WITH (SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY))
GO
ALTER AVAILABILITY GROUP [SQLAG1] MODIFY REPLICA ON N'WSFCNODE2' WITH (SECONDARY_ROLE (ALLOW_CONNECTIONS = READ_ONLY))
GO
```
*Step 2. Create the routing URL*
```
ALTER AVAILABILITY GROUP [SQLAG1] MODIFY REPLICA ON N'WSFCNODE1' WITH (SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N'TCP://WSFCNode1.tagechtalk.com:1433'))
GO
ALTER AVAILABILITY GROUP [SQLAG1] MODIFY REPLICA ON N'WSFCNODE2' WITH (SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N'TCP://WSFCNode2.tagechtalk.com:1433'))
GO
```
*Step 3. Create the routing list*
```
ALTER AVAILABILITY GROUP [SQLAG1] MODIFY REPLICA ON N'WSFCNODE1' WITH (PRIMARY_ROLE(READ_ONLY_ROUTING_LIST=('WSFCNODE2','WSFCNODE1')));
GO
ALTER AVAILABILITY GROUP [SQLAG1] MODIFY REPLICA ON N'WSFCNODE2' WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=('WSFCNODE1','WSFCNODE2')));
GO
```
*Step 4. Validate the routing list*
```
SELECT AGSrc.replica_server_name AS PrimaryReplica, AGRepl.replica_server_name AS ReadOnlyReplica, AGRepl.read_only_routing_url AS RoutingURL , AGRL.routing_priority AS RoutingPriority FROM sys.availability_read_only_routing_lists AGRL INNER JOIN sys.availability_replicas AGSrc ON AGRL.replica_id = AGSrc.replica_id INNER JOIN sys.availability_replicas AGRepl ON AGRL.read_only_replica_id = AGRepl.replica_id INNER JOIN sys.availability_groups AV ON AV.group_id = AGSrc.group_id ORDER BY PrimaryReplica
```
*Step 5. SQL Command Utility*
```
sqlcmd -S SQLAG1,1433 -E -d test -K ReadOnly
```
# Connect by using an SSH tunnel in pgAdmin
*Jeevan Shetty and Bhanu Ganesh Gudivada, Amazon Web Services*
## Summary
For security reasons, it’s always good to place databases in a private subnet. Queries against the database can be run by connecting through an Amazon Elastic Compute Cloud (Amazon EC2) bastion host in a public subnet on the Amazon Web Services (AWS) Cloud. This requires installing software, such as pgAdmin or DBeaver, which are commonly used by developers or database administrators, on the Amazon EC2 host.
Running pgAdmin on a Linux server and accessing it through a web browser requires the installation of additional dependencies, permissions setup, and configuration.
As an alternate solution, developers or database administrators can connect to a PostgreSQL database by using pgAdmin to enable an SSH tunnel from their local system. In this approach, pgAdmin uses the Amazon EC2 host in the public subnet as an intermediary host before connecting to the database. The diagram in the *Architecture* section shows the setup.
**Note**
Ensure that the security group attached to the PostgreSQL database allows connection on port 5432 from the Amazon EC2 host.
## Prerequisites and limitations
**Prerequisites **
+ An existing AWS account
+ A virtual private cloud (VPC) with a public subnet and a private subnet
+ An EC2 instance with a security group attached
+ An Amazon Aurora PostgreSQL-Compatible Edition database with a security group attached
+ A Secure Shell (SSH) key pair for setting up the tunnel
**Product versions**
+ pgAdmin version 6.2\$1
+ Amazon Aurora PostgreSQL-Compatible Edition version 12.7\$1
## Architecture
**Target technology stack**
+ Amazon EC2
+ Amazon Aurora PostgreSQL-Compatible
**Target architecture**
The following diagram shows using pgAdmin with an SSH tunnel to connect through an internet gateway to the EC2 instance, which connects to the database.
![\[pgAdmin with SSH tunnel connects through internet gateway to EC2 instance that connects to database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7d25d570-5685-4f1a-bef0-212e257cb589/images/4556d930-f9b3-4b65-be5d-d40dd9437d5a.png)
## Tools
**AWS services**
+ [Amazon Aurora PostgreSQL-Compatible Edition](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html) is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
**Other services**
+ [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects.
## Epics
### Create the connection
| Task | Description | Skills required |
| --- | --- | --- |
| Create a server. | In pgAdmin, choose **Create**, and then choose **Server**. For additional help with setting up pgAdmin to register a server, configure a connection, and connect through SSH tunneling by using the Server Dialog, see the links in the *Related resources* section. | DBA |
| Provide a name for the server. | On the **General** tab, enter a name. | DBA |
| Enter the database details. | On the **Connection** tab, enter values for the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/connect-by-using-an-ssh-tunnel-in-pgadmin.html) | DBA |
| Enter the Amazon EC2 server details. | On the **SSH Tunnel** tab, provide the details of the Amazon EC2 instance that is in the public subnet.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/connect-by-using-an-ssh-tunnel-in-pgadmin.html) | DBA |
| Save and connect. | Choose **Save** to complete the setup and connect to the Aurora PostgreSQL-Compatible database by using the SSH tunnel. | DBA |
## Related resources
+ [Server Dialog](https://www.pgadmin.org/docs/pgadmin4/latest/server_dialog.html)
+ [Connect to Server](https://www.pgadmin.org/docs/pgadmin4/latest/connect_to_server.html)
# Convert JSON Oracle queries into PostgreSQL database SQL
*Pinesh Singal and Lokesh Gurram, Amazon Web Services*
## Summary
This migration process for moving from on-premises to the Amazon Web Services (AWS) Cloud uses the AWS Schema Conversion Tool (AWS SCT) to convert the code from an Oracle database into a PostgreSQL database. Most of the code is automatically converted by AWS SCT. However, JSON-related Oracle queries are not automatically converted.
Starting from Oracle 12.2 version, Oracle Database supports various JSON functions that help in converting JSON-based data into ROW-based data. However, AWS SCT doesn’t automatically convert JSON-based data into language that is supported by PostgreSQL.
This migration pattern primarily focuses on manually converting the JSON-related Oracle queries with functions such as `JSON_OBJECT`, `JSON_ARRAYAGG`, and `JSON_TABLE` from an Oracle database to a PostgreSQL database.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ An on-premises Oracle database instance (up and running)
+ An Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition database instance (up and running)
**Limitations **
+ JSON-related queries require a fixed `KEY` and `VALUE` format. Not using that format returns the wrong result.
+ If any change in JSON structure adds new `KEY` and `VALUE` pairs in the result section, the corresponding procedure or function must be changed in the SQL query.
+ Some JSON-related functions are supported in earlier versions of Oracle and PostgreSQL but with fewer capabilities.
**Product versions**
+ Oracle Database version 12.2 and later
+ Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible version 9.5 and later
+ AWS SCT latest version (tested using version 1.0.664)
## Architecture
**Source technology stack **
+ An Oracle database instance with version 19c
**Target technology stack **
+ An Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database instance with version 13
**Target architecture **
![\[Description follows the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5e2c3b07-9ef5-417f-b049-bcea58f2c3ec/images/2ff8b00b-8849-4ef1-9be1-579f7b51be10.png)
1. Use AWS SCT with the JSON function code to convert the source code from Oracle to PostgreSQL.
1. The conversion produces PostgreSQL-supported migrated .sql files.
1. Manually convert the non-converted Oracle JSON function codes to PostgreSQL JSON function codes.
1. Run the .sql files on the target Aurora PostgreSQL-Compatible DB instance.
## Tools
**AWS services**
+ [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL.
+ [Amazon Relational Database Service (Amazon RDS) for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud.
+ [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) supports heterogeneous database migrations by automatically converting the source database schema and a majority of the custom code to a format that’s compatible with the target database.
**Other services**
+ [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) is an integrated development environment that simplifies the development and management of Oracle databases in both traditional and cloud-based deployments.
+ pgAdmin or DBeaver. [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects. [DBeaver](https://dbeaver.io/) is a universal database tool.
## Best practices
Oracle query has type `CAST` as the default when using the `JSON_TABLE` function. A best practice is to use `CAST` in PostgreSQL too, using double greater-than characters (`>>`).
For more information, see *Postgres\$1SQL\$1Read\$1JSON* in the *Additional information* section.
## Epics
### Generate the JSON data in the Oracle and PostgreSQL databases
| Task | Description | Skills required |
| --- | --- | --- |
| Store the JSON data in the Oracle database. | Create a table in the Oracle database, and store the JSON data in the `CLOB` column. Use the *Oracle\$1Table\$1Creation\$1Insert\$1Script* that’s in the *Additional information* section. | Migration engineer |
| Store the JSON data in the PostgreSQL database. | Create a table in the PostgreSQL database, and store the JSON data in the `TEXT` column. Use the *Postgres\$1Table\$1Creation\$1Insert\$1Script* that’s in the *Additional information* section. | Migration engineer |
### Convert the JSON into ROW format
| Task | Description | Skills required |
| --- | --- | --- |
| Convert the JSON data on the Oracle database. | Write an Oracle SQL query to read the JSON data into ROW format. For more details and example syntax, see *Oracle\$1SQL\$1Read\$1JSON* in the *Additional information* section. | Migration engineer |
| Convert the JSON data on the PostgreSQL database. | Write a PostgreSQL query to read the JSON data into ROW format. For more details and example syntax, see *Postgres\$1SQL\$1Read\$1JSON* in the *Additional information* section. | Migration engineer |
### Manually convert the JSON data using the SQL query and report the output in JSON format
| Task | Description | Skills required |
| --- | --- | --- |
| Perform aggregations and validation on the Oracle SQL query. | To manually convert the JSON data, perform a join, aggregation, and validation on the Oracle SQL query, and report the output in JSON format. Use the code under *Oracle\$1SQL\$1JSON\$1Aggregation\$1Join* in the *Additional information* section.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-json-oracle-queries-into-postgresql-database-sql.html) | Migration engineer |
| Perform aggregations and validation on the Postgres SQL query. | To manually convert the JSON data, perform a join, aggregation, and validation on the PostgreSQL query, and report the output in JSON format. Use the code under *Postgres\$1SQL\$1JSON\$1Aggregation\$1Join* in the *Additional information* section.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/convert-json-oracle-queries-into-postgresql-database-sql.html) | Migration engineer |
### Convert the Oracle procedure into a PostgreSQL function that contains JSON queries
| Task | Description | Skills required |
| --- | --- | --- |
| Convert the JSON queries in the Oracle procedure into rows. | For the example Oracle procedure, use the previous Oracle query and the code under *Oracle\$1procedure\$1with\$1JSON\$1Query* in the *Additional information* section. | Migration engineer |
| Convert the PostgreSQL functions that have JSON queries into row-based data. | For the example PostgreSQL functions, use the previous PostgreSQL query and the code that’s under *Postgres\$1function\$1with\$1JSON\$1Query* in the *Additional information* section. | Migration engineer |
## Related resources
+ [Oracle JSON functions](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/adjsn/generation.html)
+ [PostgreSQL JSON functions](https://www.postgresql.org/docs/13/functions-json.html)
+ [Oracle JSON Functions Examples](https://oracle-base.com/articles/12c/sql-json-functions-12cr2)
+ [PostgreSQL JSON function examples](https://dba.stackexchange.com/questions/69655/select-columns-inside-json-agg)
+ [AWS Schema Conversion Tool](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)
## Additional information
To convert JSON code from the Oracle database to PostgreSQL database, use the following scripts, in order.
**1. Oracle\$1Table\$1Creation\$1Insert\$1Script**
```
create table aws_test_table(id number,created_on date default sysdate,modified_on date,json_doc clob);
REM INSERTING into EXPORT_TABLE
SET DEFINE OFF;
Insert into aws_test_table (ID,CREATED_ON,MODIFIED_ON,json_doc)
values (1,to_date('02-AUG-2022 12:30:14','DD-MON-YYYY HH24:MI:SS'),to_date('02-AUG-2022 12:30:14','DD-MON-YYYY HH24:MI:SS'),TO_CLOB(q'[{
"metadata" : {
"upperLastNameFirstName" : "ABC XYZ",
"upperEmailAddress" : "abc@gmail.com",
"profileType" : "P"
},
"data" : {
"onlineContactId" : "032323323",
"displayName" : "Abc, Xyz",
"firstName" : "Xyz",
"lastName" : "Abc",
"emailAddress" : "abc@gmail.com",
"productRegistrationStatus" : "Not registered",
"positionId" : "0100",
"arrayPattern" : " -'",
"a]')
|| TO_CLOB(q'[ccount" : {
"companyId" : "SMGE",
"businessUnitId" : 7,
"accountNumber" : 42000,
"parentAccountNumber" : 32000,
"firstName" : "john",
"lastName" : "doe",
"street1" : "retOdertcaShr ",
"city" : "new york",
"postalcode" : "XY ABC",
"country" : "United States"
},
"products" : [
{
"appUserGuid" : "i0acc4450000001823fbad478e2eab8a0",
"id" : "0000000046",
]')
|| TO_CLOB(q'[ "name" : "ProView",
"domain" : "EREADER",
"registrationStatus" : false,
"status" : "11"
}
]
}
}]'));
Insert into aws_test_table (ID,CREATED_ON,MODIFIED_ON,json_doc) values (2,to_date('02-AUG-2022 12:30:14','DD-MON-YYYY HH24:MI:SS'),to_date('02-AUG-2022 12:30:14','DD-MON-YYYY HH24:MI:SS'),TO_CLOB(q'[{
"metadata" : {
"upperLastNameFirstName" : "PQR XYZ",
"upperEmailAddress" : "pqr@gmail.com",
"profileType" : "P"
},
"data" : {
"onlineContactId" : "54534343",
"displayName" : "Xyz, pqr",
"firstName" : "pqr",
"lastName" : "Xyz",
"emailAddress" : "pqr@gmail.com",
"productRegistrationStatus" : "Not registered",
"positionId" : "0090",
"arrayPattern" : " -'",
"account" : {
"companyId" : "CARS",
"busin]')
|| TO_CLOB(q'[essUnitId" : 6,
"accountNumber" : 42001,
"parentAccountNumber" : 32001,
"firstName" : "terry",
"lastName" : "whitlock",
"street1" : "UO 123",
"city" : "TOTORON",
"region" : "NO",
"postalcode" : "LKM 111",
"country" : "Canada"
},
"products" : [
{
"appUserGuid" : "ia744d7790000016899f8cf3f417d6df6",
"id" : "0000000014",
"name" : "ProView eLooseleaf",
]')
|| TO_CLOB(q'[ "domain" : "EREADER",
"registrationStatus" : false,
"status" : "11"
}
]
}
}]'));
commit;
```
**2. Postgres\$1Table\$1Creation\$1Insert\$1Script**
```
create table aws_test_pg_table(id int,created_on date ,modified_on date,json_doc text);
insert into aws_test_pg_table(id,created_on,modified_on,json_doc)
values(1,now(),now(),'{
"metadata" : {
"upperLastNameFirstName" : "ABC XYZ",
"upperEmailAddress" : "abc@gmail.com",
"profileType" : "P"
},
"data" : {
"onlineContactId" : "032323323",
"displayName" : "Abc, Xyz",
"firstName" : "Xyz",
"lastName" : "Abc",
"emailAddress" : "abc@gmail.com",
"productRegistrationStatus" : "Not registered",
"positionId" : "0100",
"arrayPattern" : " -",
"account" : {
"companyId" : "SMGE",
"businessUnitId" : 7,
"accountNumber" : 42000,
"parentAccountNumber" : 32000,
"firstName" : "john",
"lastName" : "doe",
"street1" : "retOdertcaShr ",
"city" : "new york",
"postalcode" : "XY ABC",
"country" : "United States"
},
"products" : [
{
"appUserGuid" : "i0acc4450000001823fbad478e2eab8a0",
"id" : "0000000046",
"name" : "ProView",
"domain" : "EREADER",
"registrationStatus" : false,
"status" : "11"
}
]
}
}');
insert into aws_test_pg_table(id,created_on,modified_on,json_doc)
values(2,now(),now(),'{
"metadata" : {
"upperLastNameFirstName" : "PQR XYZ",
"upperEmailAddress" : "pqr@gmail.com",
"profileType" : "P"
},
"data" : {
"onlineContactId" : "54534343",
"displayName" : "Xyz, pqr",
"firstName" : "pqr",
"lastName" : "Xyz",
"emailAddress" : "a*b**@h**.k**",
"productRegistrationStatus" : "Not registered",
"positionId" : "0090",
"arrayPattern" : " -",
"account" : {
"companyId" : "CARS",
"businessUnitId" : 6,
"accountNumber" : 42001,
"parentAccountNumber" : 32001,
"firstName" : "terry",
"lastName" : "whitlock",
"street1" : "UO 123",
"city" : "TOTORON",
"region" : "NO",
"postalcode" : "LKM 111",
"country" : "Canada"
},
"products" : [
{
"appUserGuid" : "ia744d7790000016899f8cf3f417d6df6",
"id" : "0000000014",
"name" : "ProView eLooseleaf",
"domain" : "EREADER",
"registrationStatus" : false,
"status" : "11"
}
]
}
}');
```
**3. Oracle\$1SQL\$1Read\$1JSON**
The following code blocks show how to convert Oracle JSON data into row format.
*Example query and syntax*
```
SELECT JSON_OBJECT(
'accountCounts' VALUE JSON_ARRAYAGG(
JSON_OBJECT(
'businessUnitId' VALUE business_unit_id,
'parentAccountNumber' VALUE parent_account_number,
'accountNumber' VALUE account_number,
'totalOnlineContactsCount' VALUE online_contacts_count,
'countByPosition' VALUE
JSON_OBJECT(
'taxProfessionalCount' VALUE tax_count,
'attorneyCount' VALUE attorney_count,
'nonAttorneyCount' VALUE non_attorney_count,
'clerkCount' VALUE clerk_count
) ) ) ) FROM
(SELECT tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number,
SUM(1) online_contacts_count,
SUM(CASE WHEN tab_data.position_id = '0095' THEN 1 ELSE 0 END) tax_count,
SUM(CASE WHEN tab_data.position_id = '0100' THEN 1 ELSE 0 END) attorney_count,
SUM(CASE WHEN tab_data.position_id = '0090' THEN 1 ELSE 0 END) non_attorney_count,
SUM(CASE WHEN tab_data.position_id = '0050' THEN 1 ELSE 0 END) clerk_count
FROM aws_test_table scco,JSON_TABLE ( json_doc, '$' ERROR ON ERROR COLUMNS (
parent_account_number NUMBER PATH
'$.data.account.parentAccountNumber',
account_number NUMBER PATH '$.data.account.accountNumber',
business_unit_id NUMBER PATH '$.data.account.businessUnitId',
position_id VARCHAR2 ( 4 ) PATH '$.data.positionId' )
) AS tab_data
INNER JOIN JSON_TABLE ( '{
"accounts": [{
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}, {
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}]
}', '$.accounts[*]' ERROR ON ERROR
COLUMNS (
parent_account_number PATH '$.parentAccountNumber',
account_number PATH '$.accountNumber',
business_unit_id PATH '$.businessUnitId')
) static_data
ON ( static_data.parent_account_number = tab_data.parent_account_number
AND static_data.account_number = tab_data.account_number
AND static_data.business_unit_id = tab_data.business_unit_id )
GROUP BY
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number );
```
The JSON document stores the data as collections. Each collection can have `KEY` and `VALUE` pairs. Every `VALUE` can have nested `KEY` and `VALUE` pairs. The following table provides information about reading the specific `VALUE` from the JSON document.
|
|
| KEY | HIERARCHY or PATH to be used to get the VALUE | VALUE |
| --- |--- |--- |
| `profileType` | `metadata` -> `profileType` | "P" |
| `positionId` | `data` -> `positionId` | "0100" |
| `accountNumber` | `data` -> account -> `accountNumber` | 42000 |
In the previous table, the `KEY` `profileType` is a `VALUE` of the `metadata` `KEY`. The `KEY` `positionId` is a `VALUE` of the `data` `KEY`. The `KEY` `accountNumber` is a `VALUE` of the `account` `KEY`, and the `account` `KEY` is a `VALUE` of the `data` `KEY`.
*Example JSON document*
```
{
"metadata" : {
"upperLastNameFirstName" : "ABC XYZ",
"upperEmailAddress" : "abc@gmail.com",
"profileType" : "P"
},
"data" : {
"onlineContactId" : "032323323",
"displayName" : "Abc, Xyz",
"firstName" : "Xyz",
"lastName" : "Abc",
"emailAddress" : "abc@gmail.com",
"productRegistrationStatus" : "Not registered",
"positionId" : "0100",
"arrayPattern" : " -",
"account" : {
"companyId" : "SMGE",
"businessUnitId" : 7,
"accountNumber" : 42000,
"parentAccountNumber" : 32000,
"firstName" : "john",
"lastName" : "doe",
"street1" : "retOdertcaShr ",
"city" : "new york",
"postalcode" : "XY ABC",
"country" : "United States"
},
"products" : [
{
"appUserGuid" : "i0acc4450000001823fbad478e2eab8a0",
"id" : "0000000046",
"name" : "ProView",
"domain" : "EREADER",
"registrationStatus" : false,
"status" : "11"
}
]
}
}
```
*SQL query that is used to get the selected fields from the JSON document*
```
select parent_account_number,account_number,business_unit_id,position_id from aws_test_table aws,JSON_TABLE ( json_doc, '$' ERROR ON ERROR
COLUMNS (
parent_account_number NUMBER PATH '$.data.account.parentAccountNumber',
account_number NUMBER PATH '$.data.account.accountNumber',
business_unit_id NUMBER PATH '$.data.account.businessUnitId',
position_id VARCHAR2 ( 4 ) PATH '$.data.positionId'
)) as sc
```
In the previous query, `JSON_TABLE` is a built-in function in Oracle that converts the JSON data into row format. The JSON\$1TABLE function expects parameters in JSON format.
Every item in `COLUMNS` has a predefined `PATH`, and there an appropriate `VALUE` for a given `KEY` is returned in row format.
*Result from the previous query*
|
|
| PARENT\$1ACCOUNT\$1NUMBER | ACCOUNT\$1NUMBER | BUSINESS\$1UNIT\$1ID | POSITION\$1ID |
| --- |--- |--- |--- |
| 32000 | 42000 | 7 | 0100 |
| 32001 | 42001 | 6 | 0090 |
**4. Postgres\$1SQL\$1Read\$1JSON**
** ***Example query and syntax*
```
select *
from (
select (json_doc::json->'data'->'account'->>'parentAccountNumber')::INTEGER as parentAccountNumber,
(json_doc::json->'data'->'account'->>'accountNumber')::INTEGER as accountNumber,
(json_doc::json->'data'->'account'->>'businessUnitId')::INTEGER as businessUnitId,
(json_doc::json->'data'->>'positionId')::VARCHAR as positionId
from aws_test_pg_table) d ;
```
In Oracle, `PATH` is used to identify the specific `KEY` and `VALUE`. However, PostgreSQL uses a `HIERARCHY` model for reading `KEY` and `VALUE` from JSON. The same JSON data mentioned under `Oracle_SQL_Read_JSON` is used in the following examples.
*SQL query with type CAST not allowed*
(If you force type `CAST`, the query fails with a syntax error.)
```
select *
from (
select (json_doc::json->'data'->'account'->'parentAccountNumber') as parentAccountNumber,
(json_doc::json->'data'->'account'->'accountNumber')as accountNumber,
(json_doc::json->'data'->'account'->'businessUnitId') as businessUnitId,
(json_doc::json->'data'->'positionId')as positionId
from aws_test_pg_table) d ;
```
Using a single greater-than operator (`>`) will return the `VALUE` defined for that `KEY`. For example, `KEY`: `positionId`, and `VALUE`: `"0100"`.
Type `CAST` is not allowed when you use the single greater-than operator (`>`).
*SQL query with type CAST allowed*
```
select *
from (
select (json_doc::json->'data'->'account'->>'parentAccountNumber')::INTEGER as parentAccountNumber,
(json_doc::json->'data'->'account'->>'accountNumber')::INTEGER as accountNumber,
(json_doc::json->'data'->'account'->>'businessUnitId')::INTEGER as businessUnitId,
(json_doc::json->'data'->>'positionId')::varchar as positionId
from aws_test_pg_table) d ;
```
To use type `CAST`, you must use the double greater-than operator. If you use the single greater-than operator, the query returns the `VALUE` defined (for example, `KEY`: `positionId`, and `VALUE`: `"0100"`). Using the double greater-than operator (`>>`) will return the actual value defined for that `KEY` (for example, `KEY`: `positionId`, and `VALUE`: `0100`, without double quotation marks).
In the preceding case, `parentAccountNumber` is type `CAST` to `INT`, `accountNumber` is type `CAST` to `INT`, `businessUnitId` is type `CAST` to `INT`, and `positionId` is type `CAST` to `VARCHAR`.
The following tables show query results that explain role of the single greater-than operator (`>`) and the double greater-than operator (`>>`).
In the first table table, the query uses the single greater-than operator (`>`). Each column is in JSON type and can’t be converted into another data type.
|
|
| parentAccountNumber | accountNumber | businessUnitId | positionId |
| --- |--- |--- |--- |
| 2003565430 | 2003564830 | 7 | "0100" |
| 2005284042 | 2005284042 | 6 | "0090" |
| 2000272719 | 2000272719 | 1 | "0100" |
In the second table, the query uses the double greater-than operator (`>>`). Each column supports type `CAST` based on the column value. For example, `INTEGER` in this context.
|
|
| parentAccountNumber | accountNumber | businessUnitId | positionId |
| --- |--- |--- |--- |
| 2003565430 | 2003564830 | 7 | 0100 |
| 2005284042 | 2005284042 | 6 | 0090 |
| 2000272719 | 2000272719 | 1 | 0100 |
**5. Oracle\$1SQL\$1JSON\$1Aggregation\$1Join**
*Example query*
```
SELECT
JSON_OBJECT(
'accountCounts' VALUE JSON_ARRAYAGG(
JSON_OBJECT(
'businessUnitId' VALUE business_unit_id,
'parentAccountNumber' VALUE parent_account_number,
'accountNumber' VALUE account_number,
'totalOnlineContactsCount' VALUE online_contacts_count,
'countByPosition' VALUE
JSON_OBJECT(
'taxProfessionalCount' VALUE tax_count,
'attorneyCount' VALUE attorney_count,
'nonAttorneyCount' VALUE non_attorney_count,
'clerkCount' VALUE clerk_count
) ) ) )
FROM
(SELECT
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number,
SUM(1) online_contacts_count,
SUM(CASE WHEN tab_data.position_id = '0095' THEN 1 ELSE 0 END) tax_count,
SUM(CASE WHEN tab_data.position_id = '0100' THEN 1 ELSE 0 END) attorney_count,
SUM(CASE WHEN tab_data.position_id = '0090' THEN 1 ELSE 0 END) non_attorney_count,
SUM(CASE WHEN tab_data.position_id = '0050' THEN 1 ELSE 0 END) clerk_count
FROM aws_test_table scco,JSON_TABLE ( json_doc, '$' ERROR ON ERROR COLUMNS (
parent_account_number NUMBER PATH
'$.data.account.parentAccountNumber',
account_number NUMBER PATH '$.data.account.accountNumber',
business_unit_id NUMBER PATH '$.data.account.businessUnitId',
position_id VARCHAR2 ( 4 ) PATH '$.data.positionId' )
) AS tab_data
INNER JOIN JSON_TABLE ( '{
"accounts": [{
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}, {
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}]
}', '$.accounts[*]' ERROR ON ERROR
COLUMNS (
parent_account_number PATH '$.parentAccountNumber',
account_number PATH '$.accountNumber',
business_unit_id PATH '$.businessUnitId')
) static_data
ON ( static_data.parent_account_number = tab_data.parent_account_number
AND static_data.account_number = tab_data.account_number
AND static_data.business_unit_id = tab_data.business_unit_id )
GROUP BY
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number
);
```
To convert the row-level data into JSON format, Oracle has built-in functions such as `JSON_OBJECT`, `JSON_ARRAY`, `JSON_OBJECTAGG`, and `JSON_ARRAYAGG`.
+ `JSON_OBJECT` accepts two parameters: `KEY` and `VALUE`. The `KEY` parameter should be hardcoded or static in nature. The `VALUE` parameter is derived from table output.
+ `JSON_ARRAYAGG` accepts `JSON_OBJECT` as a parameter. This helps in grouping the set of `JSON_OBJECT` elements as a list. For example, if you have a `JSON_OBJECT` element that has multiple records (multiple `KEY` and `VALUE` pairs in the dataset), `JSON_ARRAYAGG` appends the dataset and creates a list. According to the Data Structure language, `LIST` is group of elements. In this context, `LIST` is a group of `JSON_OBJECT` elements.
The following example shows one `JSON_OBJECT` element.
```
{
"taxProfessionalCount": 0,
"attorneyCount": 0,
"nonAttorneyCount": 1,
"clerkCount": 0
}
```
This next example shows two `JSON_OBJECT` elements, with `LIST` indicated by square braces (`[ ]`).
```
[
{
"taxProfessionalCount": 0,
"attorneyCount": 0,
"nonAttorneyCount": 1,
"clerkCount": 0
}
,
{
"taxProfessionalCount": 2,
"attorneyCount": 1,
"nonAttorneyCount": 3,
"clerkCount":4
}
]
```
*Example SQL query*
```
SELECT
JSON_OBJECT(
'accountCounts' VALUE JSON_ARRAYAGG(
JSON_OBJECT(
'businessUnitId' VALUE business_unit_id,
'parentAccountNumber' VALUE parent_account_number,
'accountNumber' VALUE account_number,
'totalOnlineContactsCount' VALUE online_contacts_count,
'countByPosition' VALUE
JSON_OBJECT(
'taxProfessionalCount' VALUE tax_count,
'attorneyCount' VALUE attorney_count,
'nonAttorneyCount' VALUE non_attorney_count,
'clerkCount' VALUE clerk_count
)
)
)
)
FROM
(SELECT
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number,
SUM(1) online_contacts_count,
SUM(CASE WHEN tab_data.position_id = '0095' THEN 1 ELSE 0 END
) tax_count,
SUM(CASE WHEN tab_data.position_id = '0100' THEN 1 ELSE 0 END
) attorney_count,
SUM(CASE WHEN tab_data.position_id = '0090' THEN 1 ELSE 0 END
) non_attorney_count,
SUM(CASE WHEN tab_data.position_id = '0050' THEN 1 ELSE 0 END
) clerk_count
FROM
aws_test_table scco, JSON_TABLE ( json_doc, '$' ERROR ON ERROR
COLUMNS (
parent_account_number NUMBER PATH '$.data.account.parentAccountNumber',
account_number NUMBER PATH '$.data.account.accountNumber',
business_unit_id NUMBER PATH '$.data.account.businessUnitId',
position_id VARCHAR2 ( 4 ) PATH '$.data.positionId' )
) AS tab_data
INNER JOIN JSON_TABLE ( '{
"accounts": [{
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}, {
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}]
}', '$.accounts[*]' ERROR ON ERROR
COLUMNS (
parent_account_number PATH '$.parentAccountNumber',
account_number PATH '$.accountNumber',
business_unit_id PATH '$.businessUnitId')
) static_data ON ( static_data.parent_account_number = tab_data.parent_account_number
AND static_data.account_number = tab_data.account_number
AND static_data.business_unit_id = tab_data.business_unit_id )
GROUP BY
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number
);
```
*Example output from the previous SQL query*
```
{
"accountCounts": [
{
"businessUnitId": 6,
"parentAccountNumber": 32001,
"accountNumber": 42001,
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 0,
"nonAttorneyCount": 1,
"clerkCount": 0
}
},
{
"businessUnitId": 7,
"parentAccountNumber": 32000,
"accountNumber": 42000,
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 1,
"nonAttorneyCount": 0,
"clerkCount": 0
}
}
]
}
```
**6. Postgres\$1SQL\$1JSON\$1Aggregation\$1Join**
The PostgreSQL built-in functions `JSON_BUILD_OBJECT` and `JSON_AGG` convert the ROW-level data into JSON format. PostgreSQL `JSON_BUILD_OBJECT` and `JSON_AGG` are equivalent to Oracle `JSON_OBJECT` and `JSON_ARRAYAGG`.
*Example query*
```
select
JSON_BUILD_OBJECT ('accountCounts',
JSON_AGG(
JSON_BUILD_OBJECT ('businessUnitId',businessUnitId
,'parentAccountNumber',parentAccountNumber
,'accountNumber',accountNumber
,'totalOnlineContactsCount',online_contacts_count,
'countByPosition',
JSON_BUILD_OBJECT (
'taxProfessionalCount',tax_professional_count
,'attorneyCount',attorney_count
,'nonAttorneyCount',non_attorney_count
,'clerkCount',clerk_count
)
)
)
)
from (
with tab as (select * from (
select (json_doc::json->'data'->'account'->>'parentAccountNumber')::INTEGER as parentAccountNumber,
(json_doc::json->'data'->'account'->>'accountNumber')::INTEGER as accountNumber,
(json_doc::json->'data'->'account'->>'businessUnitId')::INTEGER as businessUnitId,
(json_doc::json->'data'->>'positionId')::varchar as positionId
from aws_test_pg_table) a ) ,
tab1 as ( select
(json_array_elements(b.jc -> 'accounts') ->> 'accountNumber')::integer accountNumber,
(json_array_elements(b.jc -> 'accounts') ->> 'businessUnitId')::integer businessUnitId,
(json_array_elements(b.jc -> 'accounts') ->> 'parentAccountNumber')::integer parentAccountNumber
from (
select '{
"accounts": [{
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}, {
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}]
}'::json as jc) b)
select
tab.businessUnitId::text,
tab.parentAccountNumber::text,
tab.accountNumber::text,
SUM(1) online_contacts_count,
SUM(CASE WHEN tab.positionId::text = '0095' THEN 1 ELSE 0 END) tax_professional_count,
SUM(CASE WHEN tab.positionId::text = '0100' THEN 1 ELSE 0 END) attorney_count,
SUM(CASE WHEN tab.positionId::text = '0090' THEN 1 ELSE 0 END) non_attorney_count,
SUM(CASE WHEN tab.positionId::text = '0050' THEN 1 ELSE 0 END) clerk_count
from tab1,tab
where tab.parentAccountNumber::INTEGER=tab1.parentAccountNumber::INTEGER
and tab.accountNumber::INTEGER=tab1.accountNumber::INTEGER
and tab.businessUnitId::INTEGER=tab1.businessUnitId::INTEGER
GROUP BY tab.businessUnitId::text,
tab.parentAccountNumber::text,
tab.accountNumber::text) a;
```
*Example output from the preceding query*
Output from Oracle and PostgreSQL is exactly the same.
```
{
"accountCounts": [
{
"businessUnitId": 6,
"parentAccountNumber": 32001,
"accountNumber": 42001,
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 0,
"nonAttorneyCount": 1,
"clerkCount": 0
}
},
{
"businessUnitId": 7,
"parentAccountNumber": 32000,
"accountNumber": 42000,
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 1,
"nonAttorneyCount": 0,
"clerkCount": 0
}
}
]
}
```
**7.Oracle\$1procedure\$1with\$1JSON\$1Query**
This code converts the Oracle procedure into a PostgreSQL function that has JSON SQL queries. It shows how the query transposes JSON into rows and the reverse.
```
CREATE OR REPLACE PROCEDURE p_json_test(p_in_accounts_json IN varchar2, p_out_accunts_json OUT varchar2)
IS
BEGIN
/*
p_in_accounts_json paramter should have following format:
{
"accounts": [{
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}, {
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}]
}
*/
SELECT
JSON_OBJECT(
'accountCounts' VALUE JSON_ARRAYAGG(
JSON_OBJECT(
'businessUnitId' VALUE business_unit_id,
'parentAccountNumber' VALUE parent_account_number,
'accountNumber' VALUE account_number,
'totalOnlineContactsCount' VALUE online_contacts_count,
'countByPosition' VALUE
JSON_OBJECT(
'taxProfessionalCount' VALUE tax_count,
'attorneyCount' VALUE attorney_count,
'nonAttorneyCount' VALUE non_attorney_count,
'clerkCount' VALUE clerk_count
) ) ) )
into p_out_accunts_json
FROM
(SELECT
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number,
SUM(1) online_contacts_count,
SUM(CASE WHEN tab_data.position_id = '0095' THEN 1 ELSE 0 END) tax_count,
SUM(CASE WHEN tab_data.position_id = '0100' THEN 1 ELSE 0 END) attorney_count,
SUM(CASE WHEN tab_data.position_id = '0090' THEN 1 ELSE 0 END) non_attorney_count,
SUM(CASE WHEN tab_data.position_id = '0050' THEN 1 ELSE 0 END) clerk_count
FROM aws_test_table scco,JSON_TABLE ( json_doc, '$' ERROR ON ERROR
COLUMNS (
parent_account_number NUMBER PATH '$.data.account.parentAccountNumber',
account_number NUMBER PATH '$.data.account.accountNumber',
business_unit_id NUMBER PATH '$.data.account.businessUnitId',
position_id VARCHAR2 ( 4 ) PATH '$.data.positionId' )
) AS tab_data
INNER JOIN JSON_TABLE ( p_in_accounts_json, '$.accounts[*]' ERROR ON ERROR
COLUMNS (
parent_account_number PATH '$.parentAccountNumber',
account_number PATH '$.accountNumber',
business_unit_id PATH '$.businessUnitId')
) static_data
ON ( static_data.parent_account_number = tab_data.parent_account_number
AND static_data.account_number = tab_data.account_number
AND static_data.business_unit_id = tab_data.business_unit_id )
GROUP BY
tab_data.business_unit_id,
tab_data.parent_account_number,
tab_data.account_number
);
EXCEPTION
WHEN OTHERS THEN
raise_application_error(-20001,'Error while running the JSON query');
END;
/
```
*Running the procedure*
The following code block explains how you can run the previously created Oracle procedure with example JSON input to the procedure. It also gives you the result or output from this procedure.
```
set serveroutput on;
declare
v_out varchar2(30000);
v_in varchar2(30000):= '{
"accounts": [{
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}, {
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}]
}';
begin
p_json_test(v_in,v_out);
dbms_output.put_line(v_out);
end;
/
```
*Procedure output*
```
{
"accountCounts": [
{
"businessUnitId": 6,
"parentAccountNumber": 32001,
"accountNumber": 42001,
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 0,
"nonAttorneyCount": 1,
"clerkCount": 0
}
},
{
"businessUnitId": 7,
"parentAccountNumber": 32000,
"accountNumber": 42000,
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 1,
"nonAttorneyCount": 0,
"clerkCount": 0
}
}
]
}
```
**8.Postgres\$1function\$1with\$1JSON\$1Query**
*Example function*
```
CREATE OR REPLACE FUNCTION f_pg_json_test(p_in_accounts_json text)
RETURNS text
LANGUAGE plpgsql
AS
$$
DECLARE
v_out_accunts_json text;
BEGIN
SELECT
JSON_BUILD_OBJECT ('accountCounts',
JSON_AGG(
JSON_BUILD_OBJECT ('businessUnitId',businessUnitId
,'parentAccountNumber',parentAccountNumber
,'accountNumber',accountNumber
,'totalOnlineContactsCount',online_contacts_count,
'countByPosition',
JSON_BUILD_OBJECT (
'taxProfessionalCount',tax_professional_count
,'attorneyCount',attorney_count
,'nonAttorneyCount',non_attorney_count
,'clerkCount',clerk_count
))))
INTO v_out_accunts_json
FROM (
WITH tab AS (SELECT * FROM (
SELECT (json_doc::json->'data'->'account'->>'parentAccountNumber')::INTEGER AS parentAccountNumber,
(json_doc::json->'data'->'account'->>'accountNumber')::INTEGER AS accountNumber,
(json_doc::json->'data'->'account'->>'businessUnitId')::INTEGER AS businessUnitId,
(json_doc::json->'data'->>'positionId')::varchar AS positionId
FROM aws_test_pg_table) a ) ,
tab1 AS ( SELECT
(json_array_elements(b.jc -> 'accounts') ->> 'accountNumber')::integer accountNumber,
(json_array_elements(b.jc -> 'accounts') ->> 'businessUnitId')::integer businessUnitId,
(json_array_elements(b.jc -> 'accounts') ->> 'parentAccountNumber')::integer parentAccountNumber
FROM (
SELECT p_in_accounts_json::json AS jc) b)
SELECT
tab.businessUnitId::text,
tab.parentAccountNumber::text,
tab.accountNumber::text,
SUM(1) online_contacts_count,
SUM(CASE WHEN tab.positionId::text = '0095' THEN 1 ELSE 0 END) tax_professional_count,
SUM(CASE WHEN tab.positionId::text = '0100' THEN 1 ELSE 0 END) attorney_count,
SUM(CASE WHEN tab.positionId::text = '0090' THEN 1 ELSE 0 END) non_attorney_count,
SUM(CASE WHEN tab.positionId::text = '0050' THEN 1 ELSE 0 END) clerk_count
FROM tab1,tab
WHERE tab.parentAccountNumber::INTEGER=tab1.parentAccountNumber::INTEGER
AND tab.accountNumber::INTEGER=tab1.accountNumber::INTEGER
AND tab.businessUnitId::INTEGER=tab1.businessUnitId::INTEGER
GROUP BY tab.businessUnitId::text,
tab.parentAccountNumber::text,
tab.accountNumber::text) a;
RETURN v_out_accunts_json;
END;
$$;
```
*Running the function*
```
select f_pg_json_test('{
"accounts": [{
"accountNumber": 42001,
"parentAccountNumber": 32001,
"businessUnitId": 6
}, {
"accountNumber": 42000,
"parentAccountNumber": 32000,
"businessUnitId": 7
}]
}') ;
```
*Function output *
The following output is similar to the Oracle procedure output. The difference is that this output is in Text format.
```
{
"accountCounts": [
{
"businessUnitId": "6",
"parentAccountNumber": "32001",
"accountNumber": "42001",
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 0,
"nonAttorneyCount": 1,
"clerkCount": 0
}
},
{
"businessUnitId": "7",
"parentAccountNumber": "32000",
"accountNumber": "42000",
"totalOnlineContactsCount": 1,
"countByPosition": {
"taxProfessionalCount": 0,
"attorneyCount": 1,
"nonAttorneyCount": 0,
"clerkCount": 0
}
}
]
}
```
# Copy Amazon DynamoDB tables across accounts using AWS Backup
*Ramkumar Ramanujam, Amazon Web Services*
## Summary
When working with Amazon DynamoDB on AWS, a common use case is to copy or sync DynamoDB tables in development, testing, or staging environments with the table data that is in the production environment. As a standard practice, each environment uses a different AWS account.
AWS Backup supports cross-Region and cross-account backup and restoration of data for DynamoDB, Amazon Simple Storage Service (Amazon S3), and other AWS services. This pattern provides the steps for using AWS Backup cross-account backup and restore to copy DynamoDB tables between AWS accounts.
## Prerequisites and limitations
**Prerequisites **
+ Two active AWS accounts that belong to the same organization in AWS Organizations
+ Permissions to create DynamoDB tables in both accounts
+ AWS Identity and Access Management (IAM) permissions to create and use AWS Backup vaults
**Limitations **
+ Source and target AWS accounts should be part of the same organization in AWS Organizations.
## Architecture
**Target technology stack **
+ AWS Backup
+ Amazon DynamoDB
**Target architecture **
![\[Description of copying tables between backup vaults follows the diagram.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/ef6e7393-edb6-4744-be26-43f1cbff9de9/images/fa9f3f2f-7a01-4093-9bd5-fc355e57ba67.png)
1. Create the DynamoDB table backup in the AWS Backup backup vault in the source account.
1. Copy the backup to the backup vault in the target account.
1. Restore the DynamoDB table in the target account by using the backup from the backup vault in the target account.
**Automation and scale**
You can use AWS Backup to schedule backups to run at specific intervals.
## Tools
+ [AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html) is a fully-managed service for centralizing and automating data protection across AWS services, in the cloud, and on premises. Using this service, you can configure backup policies and monitor activity for your AWS resources in one place. It allows you to automate and consolidate backup tasks that were previously performed service by service, and removes the need to create custom scripts and manual processes.
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
## Epics
### Turn on AWS Backup features in the source and target accounts
| Task | Description | Skills required |
| --- | --- | --- |
| Turn on advanced features for DynamoDB and cross-account backup. | In both the source and the target AWS accounts, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-aws-backup.html) | AWS DevOps, Migration engineer |
### Create backup vaults in the source and target accounts
| Task | Description | Skills required |
| --- | --- | --- |
| Create backup vaults. | In both the source and the target AWS accounts, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-aws-backup.html)The ARNs of both the source and the target backup vaults will be required when you copy the DynamoDB table backup between the source and target accounts. | AWS DevOps, Migration engineer |
### Perform backup and restore using backup vaults
| Task | Description | Skills required |
| --- | --- | --- |
| In the source account, create a DynamoDB table backup. | To create a backup for the DynamoDB table in the source account, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-aws-backup.html)A new backup job is created. To monitor the status of the backup job, on the AWS Backup **Jobs** page, choose the **Backup Jobs** tab. All active, in-progress, and completed backup jobs are listed on this tab. | AWS DevOps, DBA, Migration engineer |
| Copy the backup from the source account to the target account. | After the backup job is complete, copy the DynamoDB table backup from the backup vault in the source account to the backup vault in the target account.To copy the backup vault, in the source account, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-aws-backup.html) | AWS DevOps, Migration engineer, DBA |
| Restore the backup in the target account. | In the target AWS account, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-aws-backup.html) | AWS DevOps, DBA, Migration engineer |
## Related resources
+ [Using AWS Backup with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/backuprestore_HowItWorksAWS.html)
+ [Creating backup copies across AWS accounts](https://docs.aws.amazon.com/aws-backup/latest/devguide/create-cross-account-backup.html)
+ [AWS Backup pricing](https://aws.amazon.com/backup/pricing/)
# Copy Amazon DynamoDB tables across accounts using a custom implementation
*Ramkumar Ramanujam, Amazon Web Services*
## Summary
When working with Amazon DynamoDB on Amazon Web Services (AWS), a common use case is to copy or sync DynamoDB tables in development, testing, or staging environments with the table data that are in the production environment. As a standard practice, each environment uses a different AWS account.
DynamoDB now supports cross-account backup using AWS Backup. For information about associated storage costs when using AWS Backup, see [AWS Backup pricing](https://aws.amazon.com/backup/pricing/). When you use AWS Backup to copy across accounts, the source and target accounts must be part of an AWS Organizations organization. There are other solutions for cross-account backup and restore using AWS services such as AWS Glue. Using those solutions, however, increases the application footprint, because there are more AWS services to deploy and maintain.
You can also use Amazon DynamoDB Streams to capture table changes in the source account. Then you can initiate an AWS Lambda function, and make the corresponding changes in the target table in the target account. But that solution applies to use cases in which source and target tables must always be kept in sync. It might not apply to development, testing, and staging environments where data are updated frequently.
This pattern provides steps to implement a custom solution to copy a Amazon DynamoDB table from one account to another. This pattern can be implemented using common programming languages such as C\$1, Java, and Python. We recommend using a language that is supported by an [AWS SDK](https://aws.amazon.com/tools/).
## Prerequisites and limitations
**Prerequisites **
+ Two active AWS accounts
+ DynamoDB tables in both the accounts
+ Knowledge of AWS Identity and Access Management (IAM) roles and policies
+ Knowledge of how to access Amazon DynamoDB tables using any common programming language, such as C\$1, Java, or Python
**Limitations **
This pattern applies to DynamoDB tables that are around 2 GB or smaller. With additional logic to handle connection or session interruptions, throttling, and failures and retries, it can be used for larger tables.
The DynamoDB scan operation, which reads items from the source table, can fetch only up to 1 MB of data in a single call. For larger tables, greater than 2 GB, this limitation can increase the total time to perform a full table copy.
## Architecture
The following diagram shows the custom implementation between the source and target AWS accounts. IAM policies and security tokens are used with the custom implementation. Data is read from Amazon DynamoDB in the source account and written to DynamoDB in the target account.
![\[Source and target account architecture for copying by using the custom implementation.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/ba8175be-9809-4c2e-b2d1-6b9180ed056c/images/d9d4c2c8-ff04-443f-9137-e37b8e23ccb5.png)
**Automation and scale**
This pattern applies to DynamoDB tables that are smaller in size, around 2 GB.
To apply this pattern for larger tables, address the following issues:
+ During the table copy operation, two active sessions are maintained, using different security tokens. If the table copy operation takes longer than the token expiration times, you must put in place logic to refresh the security tokens.
+ If enough read capacity units (RCUs) and write capacity units (WCUs) are not provisioned, reads or writes on the source or target table might get throttled. Be sure to catch and handle these exceptions.
+ Handle any other failures or exceptions and put a retry mechanism in place to retry or continue from where the copy operation failed.
## Tools
**Tools**
+ [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) – Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
+ The additional tools required will differ based on the programming language that you choose for the implementation. For example, if you use C\$1, you will need Microsoft Visual Studio and the following NuGet packages:
+ `AWSSDK`
+ `AWSSDK.DynamoDBv2`
**Code **
The following Python code snippet deletes and recreates a DynamoDB table using the Boto3 library.
Do not use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` of an IAM user because these are long-term credentials, which should be avoided for programmatic access to AWS services. For more information about temporary credentials, see the *Best practices* section.
The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `TEMPORARY_SESSION_TOKEN` used in the following code snippet are temporary credentials fetched from AWS Security Token Service (AWS STS).
```
import boto3
import sys
import json
#args = input-parameters = GLOBAL_SEC_INDEXES_JSON_COLLECTION, ATTRIBUTES_JSON_COLLECTION, TARGET_DYNAMODB_NAME, TARGET_REGION, ...
#Input param: GLOBAL_SEC_INDEXES_JSON_COLLECTION
#[{"IndexName":"Test-index","KeySchema":[{"AttributeName":"AppId","KeyType":"HASH"},{"AttributeName":"AppType","KeyType":"RANGE"}],"Projection":{"ProjectionType":"INCLUDE","NonKeyAttributes":["PK","SK","OwnerName","AppVersion"]}}]
#Input param: ATTRIBUTES_JSON_COLLECTION
#[{"AttributeName":"PK","AttributeType":"S"},{"AttributeName":"SK","AttributeType":"S"},{"AttributeName":"AppId","AttributeType":"S"},{"AttributeName":"AppType","AttributeType":"N"}]
region = args['TARGET_REGION']
target_ddb_name = args['TARGET_DYNAMODB_NAME']
global_secondary_indexes = json.loads(args['GLOBAL_SEC_INDEXES_JSON_COLLECTION'])
attribute_definitions = json.loads(args['ATTRIBUTES_JSON_COLLECTION'])
# Drop and create target DynamoDB table
dynamodb_client = boto3.Session(
aws_access_key_id=args['AWS_ACCESS_KEY_ID'],
aws_secret_access_key=args['AWS_SECRET_ACCESS_KEY'],
aws_session_token=args['TEMPORARY_SESSION_TOKEN'],
).client('dynamodb')
# Delete table
print('Deleting table: ' + target_ddb_name + ' ...')
try:
dynamodb_client.delete_table(TableName=target_ddb_name)
#Wait for table deletion to complete
waiter = dynamodb_client.get_waiter('table_not_exists')
waiter.wait(TableName=target_ddb_name)
print('Table deleted.')
except dynamodb_client.exceptions.ResourceNotFoundException:
print('Table already deleted / does not exist.')
pass
print('Creating table: ' + target_ddb_name + ' ...')
table = dynamodb_client.create_table(
TableName=target_ddb_name,
KeySchema=[
{
'AttributeName': 'PK',
'KeyType': 'HASH' # Partition key
},
{
'AttributeName': 'SK',
'KeyType': 'RANGE' # Sort key
}
],
AttributeDefinitions=attribute_definitions,
GlobalSecondaryIndexes=global_secondary_indexes,
BillingMode='PAY_PER_REQUEST'
)
waiter = dynamodb_client.get_waiter('table_exists')
waiter.wait(TableName=target_ddb_name)
print('Table created.')
```
## Best practices
**Temporary credentials**
As a security best practice, while accessing AWS services programmatically, avoid using the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` of an IAM user because these are long-term credentials. Always try to use temporary credentials to access AWS services programmatically.
As an example, a developer hardcodes the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` of an IAM user in the application during development but fails to remove the hardcoded values before pushing changes to code repository. These exposed credentials can be used by unintended or malicious users, which can have serious implications (especially if the exposed credentials have admin privileges). These exposed credentials should be deactivated or deleted immediately by using the IAM console or AWS Command Line Interface (AWS CLI).
To get temporary credentials for programmatic access to AWS services, use AWS STS. Temporary credentials are valid only for the specified time (from 15 minutes up to 36 hours). The maximum allowed duration of temporary credentials varies depending on such factors as role settings and role chaining. For more information about AWS STS, see the [documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html).
## Epics
### Set up DynamoDB tables
| Task | Description | Skills required |
| --- | --- | --- |
| Create DynamoDB tables. | Create DynamoDB tables, with indexes, in both source and target AWS accounts.Set the capacity provisioning as on-demand mode, which allows DynamoDB to scale read/write capacities dynamically based on the workload. Alternatively, you can use provisioned capacity with 4000 RCUs and 4000 WCUs. | App developer, DBA, Migration engineer |
| Populate the source table. | Populate the DynamoDB table in the source account with test data. Having at least 50 MB or more of test data helps you to see the peak and average RCUs consumed during table copy. You can then change the capacity provisioning as needed. | App developer, DBA, Migration engineer |
### Set up credentials to access the DynamoDB tables
| Task | Description | Skills required |
| --- | --- | --- |
| Create IAM roles to access the source and target DynamoDB tables. | Create an IAM role in the source account with permissions to access (read) the DynamoDB table in the source account.Add the source account as a trusted entity for this role.Create an IAM role in the target account with permissions to access (create, read, update, delete) the DynamoDB table in the target account. Add the target account as a trusted entity for this role. | App developer, AWS DevOps |
### Copy table data from one account to another
| Task | Description | Skills required |
| --- | --- | --- |
| Get temporary credentials for the IAM roles. | Get temporary credentials for IAM role created in source account.Get temporary credentials for IAM role created in target account.One way to get the temporary credentials for the IAM role is to use AWS STS from the AWS CLI.aws sts assume-role --role-arn arn:aws:iam::Use the appropriate AWS profile (corresponding to the source or target account).For more information about different ways to get temporary credentials, see the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-a-custom-implementation.html) | App developer, Migration engineer | | Initialize the DynamoDB clients for source and target DynamoDB access. | Initialize the DynamoDB clients, which are provided by the AWS SDK, for the source and target DynamoDB tables.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-a-custom-implementation.html)For more information about making requests by using IAM temporary credentials, see the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/AuthUsingTempSessionToken.html). | App developer | | Drop and recreate the target table. | Delete and recreate the target DynamoDB table (along with indexes) in the target account, using the target account DynamoDB client.Deleting all records from a DynamoDB table is a costly operation because it consumes provisioned WCUs. Deleting and recreating the table avoids those extra costs.You can add indexes to a table after you create it, but this takes 2–5 minutes longer. Creating indexes during table creation, by passing the indexes collection to the `createTable` call, is more efficient. | App developer | | Perform the table copy. | Repeat the following steps until all data are copied:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-a-custom-implementation.html)For more information, see the reference implementation in C\$1 (for dropping, creating, and populating tables) in the *Attachments* section. An example table config JavaScript Object Notation (JSON) file is also attached. | App developer | ## Related resources + [Amazon DynamoDB documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) + [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) + [AWS SDKs](https://aws.amazon.com/tools/) + [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html) ## Additional information This pattern was implemented using C\$1 to copy a DynamoDB table with 200,000 items (average item size of 5 KB and table size of 250 MB). The target DynamoDB table was set up with provisioned capacity of 4000 RCUs and 4000 WCUs. The complete table copy operation (from source account to target account), including dropping and recreating the table, took 5 minutes. Total capacity units consumed: 30,000 RCUs and approximately 400,000 WCUs. For more information on DynamoDB capacity modes, see [Read/Write capacity mode](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html) in the AWS documentation. ## Attachments To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/ba8175be-9809-4c2e-b2d1-6b9180ed056c/attachments/attachment.zip) # Create detailed cost and usage reports for Amazon RDS and Amazon Aurora *Lakshmanan Lakshmanan and Sudarshan Narasimhan, Amazon Web Services* ## Summary This pattern shows how to track usage costs for Amazon Relational Database Service (Amazon RDS) or Amazon Aurora clusters by configuring [user-defined cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/custom-tags.html). You can use these tags to create detailed cost and usage reports in AWS Cost Explorer for clusters across multiple dimensions. For example, you can track usage costs at the team, project, or cost center level, and then analyze the data in Amazon Athena. ## Prerequisites and limitations **Prerequisites** + An active AWS account + One or more [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html) or [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html) instances **Limitations** For tagging restrictions, see the [AWS Billing User Guide](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html). ## Architecture **Target technology stack** + Amazon RDS or Amazon Aurora + AWS Cost and Usage Report + AWS Cost Explorer + Amazon Athena **Workflow and architecture** The tagging and analysis workflow consists of these steps: 1. A data engineer, database administrator, or AWS administrator creates user-defined cost allocation tags for the Amazon RDS or Aurora clusters. 1. An AWS administrator activates the tags. 1. The tags report metadata to AWS Cost Explorer. 1. A data engineer, database administrator, or AWS administrator creates a [monthly cost allocation report](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/configurecostallocreport.html#allocation-viewing). 1. A data engineer, database administrator, or AWS administrator analyzes the monthly cost allocation report by using Amazon Athena. The following diagram shows how to apply tags to track usage costs for Amazon RDS or Aurora instances. ![\[Applying tags to track usage costs for database instances and clusters\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/eab5001d-5115-4aa6-bdd2-23063b08b262/images/63292b18-01d6-4523-b8ac-2c3b12b11b84.png) The following architecture diagram shows how the cost allocation report is integrated with Amazon Athena for analysis. ![\[Querying cost allocation reports in Athena\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/eab5001d-5115-4aa6-bdd2-23063b08b262/images/9c028405-1e93-4f6a-a0e5-36154e2b8eab.png) The monthly cost allocation report is stored in an Amazon S3 bucket that you specify. When you set up Athena with the AWS CloudFormation template, as described in the *Epics* section, the template provisions several additional resources, including an AWS Glue crawler, an AWS Glue database, an Amazon Simple Notification System (Amazon SNS) event, AWS Lambda functions, and AWS Identity and Access Management (IAM) roles for the Lambda functions. As new cost data files arrive in the S3 bucket, event notifications are used to forward these files to a Lambda function for processing. The Lambda function initiates an AWS Glue crawler job to create or update the table in the AWS Glue Data Catalog. This table is then used to query data in Athena. ## Tools + [Amazon Athena](https://aws.amazon.com/athena/) is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. + [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. + [AWS CloudFormation](https://aws.amazon.com/cloudformation/) is an infrastructure as code (IaC) service that allows you to easily model, provision, and manage AWS and third-party resources. + [AWS Cost Explorer](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-what-is.html) helps you view and analyze your AWS costs and usage. ## Epics ### Create and activate tags for your Amazon RDS or Aurora cluster | Task | Description | Skills required | | --- | --- | --- | | Create user-defined cost allocation tags for your Amazon RDS or Aurora cluster. | To add tags to a new or existing Amazon RDS or Aurora cluster, follow the instructions in [Adding, listing, and removing tags](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_Tagging.html#Tagging.HowTo) in the *Amazon Aurora User Guide*.For information about how to set up an Amazon Aurora cluster, see the instructions for [MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) and [PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) in the *Amazon Aurora User Guide*. | AWS administrator, Data engineer, DBA | | Activate the user-defined cost allocation tags. | Follow the instructions in [Activating user-defined cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/activating-tags.html) in the *AWS Billing User Guide*. | AWS administrator | ### Create cost and usage reports | Task | Description | Skills required | | --- | --- | --- | | Create and configure cost and usage reports for your clusters. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-detailed-cost-and-usage-reports-for-amazon-rds-and-amazon-aurora.html)The data will be available in 24 hours. | App owner, AWS administrator, DBA, General AWS, Data engineer | ### Analyze cost and usage report data | Task | Description | Skills required | | --- | --- | --- | | Analyze the cost and usage report data. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-detailed-cost-and-usage-reports-for-amazon-rds-and-amazon-aurora.html):role/ --role-session-name --profile
select status from cost_and_usage_data_statusFor more information, see [Running Amazon Athena queries](https://docs.aws.amazon.com/cur/latest/userguide/cur-ate-run.html) in the *AWS Cost and Usage Reports User Guide*.When you run your SQL query, make sure that the correct database is selected from the dropdown list. | App owner, AWS administrator, DBA, General AWS, Data engineer | ## Related resources **References** + [Setting up Athena using AWS CloudFormation templates](https://docs.aws.amazon.com/cur/latest/userguide/use-athena-cf.html) (recommended) + [Setting up Athena manually](https://docs.aws.amazon.com/cur/latest/userguide/cur-ate-manual.html) + [Running Amazon Athena queries](https://docs.aws.amazon.com/cur/latest/userguide/cur-ate-run.html) + [Loading report data to other resources](https://docs.aws.amazon.com/cur/latest/userguide/cur-query-other.html) **Tutorials and videos** + [Analyze Cost and Usage Reports using Amazon Athena ](https://youtu.be/KEeJEZTYE8E)(YouTube video) # Deploy SQL Server failover cluster instances on Amazon EC2 and Amazon FSx by using Terraform *Mark Hudson and Matt Burgess, Amazon Web Services* ## Summary This pattern uses Terraform to deploy SQL Server failover cluster instances (FCIs) across Windows Server Failover Cluster (WSFC) nodes on Amazon Elastic Compute Cloud (Amazon EC2). In addition, the pattern uses Amazon FSx shared storage for data and log files. When SQL Server databases are migrated to AWS, the first choice is Amazon RDS for SQL Server. However, sometimes Amazon RDS for SQL Server isn’t suitable and SQL Server must be deployed on Amazon EC2 in a highly available architecture. In this solution, SQL Server FCIs are installed across WSFC nodes. The Terraform module included with this pattern provisions up to two Amazon EC2 SQL Server instances. An Amazon FSx for Windows File Server file system acts as the quorum witness and stores shared data and log files. Regardless of the number of instances configured, the SQL Server instance nodes will always create and join an FCI cluster to ensure environmental parity. (Typically, one instance is configured for development and two instances for production environments.) For configurations that use two nodes for high availability, an internal Network Load Balancer is provisioned. The Network Load Balancer uses a health probe configured on the FCI cluster to identify which node is the primary. ## Prerequisites and limitations **Prerequisites** + An active AWS account. + Amazon Virtual Private Cloud (Amazon VPC) with two subnets in separate Availability Zones. + Amazon VPC [DHCP option set](https://docs.aws.amazon.com/vpc/latest/userguide/DHCPOptionSet.html). Configure the domain name to resolve to your Active Directory domain name and the domain and NetBIOS name servers to point to your Active Directory domain controllers. For more information, see *VPC configuration* in [Additional information](#deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx-additional). + AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD). + Custom Amazon Machine Image (AMI). For more details, see *AMI configuration* in [Additional information](#deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx-additional). + Amazon Simple Storage Service (Amazon S3) bucket containing the SQL Server ISO image. This prerequisite is only required if using [EC2 Image Builder](https://docs.aws.amazon.com/imagebuilder/latest/userguide/what-is-image-builder.html) with the provided `component.yaml` file to build the custom AMI. + AWS Key Management Service (AWS KMS) encryption key. + By default, SQL Server is installed using a developer edition product key. Production systems are expected to use a valid product key passed to the module by the relevant variable. **Limitations** + This solution requires AWS Managed Microsoft AD. However, if you prefer, you can use a self-managed Active Directory implementation instead. To do so, modify the included Amazon FSx Terraform module to remove the `active_directory_id` attribute. Then, add the four attributes that are required for self-managed Active Directory as shown in the [Terraform documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/fsx_windows_file_system). + SQL Server is set up to use mixed mode authentication. If you prefer, you can use Windows-only authentication. To do so, in the provided user data script, remove the `/SECURITYMODE` and `/SAPWD` parameters that are supplied to the `setup.exe` command. You can remove the `sql_accounts.tf` file, and you can modify the `instances.tf` file to remove the `sql_sa_password` entry. + When deleting a deployed cluster, you must remove the corresponding virtual computer objects and individual computer objects in Active Directory. To remove the objects, use Active Directory administrative tools. + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service. **Product versions** This solution was tested with the following versions: + Windows Server 2019 + SQL Server 2019 + [Terraform v0.13.0](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/0-13) ## Architecture **Source technology stack** + SQL Server **Target technology stack** + SQL Server FCI on WSFC nodes using Amazon EC2 + Amazon FSx for Windows File Server + Amazon S3 bucket + AWS Secrets Manager + AWS Managed Microsoft AD + AWS KMS + AWS Identity and Access Management (IAM) **Target architecture** The following diagram shows the architecture for this solution. ![\[Architecture to deploy SQL Server failover cluster instances across Windows Server Failover Cluster nodes on Amazon EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/45f3ab19-d240-4353-ab6e-f6e565f537a4/images/0bff16f2-94e7-4e86-91ea-7ab5f3725620.png) The diagram shows the following: + An IAM role providing the EC2 instances access to AWS KMS and Secrets Manager + Two SQL Server nodes deployed on Amazon EC2 instances in private subnets across two Availability Zones + A Network Load Balancer for facilitating connections to the active SQL Server instance (not deployed when setting up a single node cluster) + Amazon FSx for Windows File Server file system deployed in both private subnets for shared storage by the SQL Server nodes + Secrets Manager for storing Active Directory and SQL Server credentials and configuration + Amazon S3 bucket for storing the SQL Server installation image + AWS Managed Microsoft AD for Windows authentication + AWS KMS for creating the encryption key **Automation and scale** You can automate the deployment of the target architecture by using the Terraform modules that are in the [GitHub repository](https://github.com/aws-samples/cluster-amazon-elastic-compute-cloud-amazon-fsx-microsoft-sql-server). You must modify the `terraform.tfvars` file to include variable values that are specific to your environment. The Amazon S3 bucket, AWS Managed Microsoft AD components, AWS KMS encryption key, and some secrets are prerequisites for this deployment and are not included in the Terraform code. ## Tools **AWS services** + [AWS Directory Service for Microsoft Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) enables your directory-aware workloads and AWS resources to use Microsoft Active Directory in the AWS Cloud. In this pattern, AWS Managed Microsoft AD is used for Windows Server and SQL Server authentication and for DNS. + [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down. In this pattern, SQL Server failover cluster instances are installed on Amazon EC2 instances. + [EC2 Image Builder](https://docs.aws.amazon.com/imagebuilder/latest/userguide/what-is-image-builder.html) helps you automate the creation, management, and deployment of customized server images. + [Amazon FSx for Windows File Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/what-is.html) provides fully managed shared storage on Windows Server. In this pattern, FSx for Windows File Server provides shared storage for SQL Server data and log files and the quorum witness. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. + [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data. In this pattern, it’s used to encrypt Secrets Manager secrets, the SQL Server storage on Amazon Elastic Block Store (Amazon EBS) volumes, and the FSx for Windows File Server file system. + [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically. In this pattern, Active Directory credentials for installing and running SQL Server, the `sa` user credentials, and the database connection information are stored in Secrets Manager. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. This pattern uses an Amazon S3 bucket to store the SQL Server installation image. + [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. ** ****Other tools** + [Microsoft SQL Server FCIs](https://learn.microsoft.com/en-us/sql/sql-server/failover-clusters/windows/always-on-failover-cluster-instances-sql-server?view=sql-server-ver15) are installed across Windows Server cluster nodes. In addition, they can be installed across multiple subnets. In this pattern, SQL Server FCI instances are installed across WSFC nodes. + [Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that helps you use code to provision and manage cloud infrastructure and resources. In this pattern, Terraform is used to create the resources and configure the SQL Server FCI instances. + [Windows Server Failover Clustering](https://learn.microsoft.com/en-us/sql/sql-server/failover-clusters/windows/windows-server-failover-clustering-wsfc-with-sql-server?view=sql-server-ver15) provides infrastructure features that support the high-availability of hosted server applications such as SQL Server. In this pattern, the FCI nodes make use of WSFC functionality to provide local high availability through redundancy at the instance level. **Code repository** The code for this pattern is available in the GitHub [cluster-amazon-elastic-compute-cloud-amazon-fsx-microsoft-sql-server](https://github.com/aws-samples/cluster-amazon-elastic-compute-cloud-amazon-fsx-microsoft-sql-server) repository. The following resources are available in the repository: + A `README.md` file that provides an overview of the solution and additional installation and usage information + A base set of Terraform configuration files and an Amazon FSx specific module to provision the components for this pattern + An instance setup script that is run as an Amazon EC2 user data script + A `component.yam`l file that Image Builder can use to create a custom AMI ## Best practices **Security and patching** + The AMI prerequisite installations and configurations are the minimum requirements to deploy SQL Server FCI clusters. Additional software and configurations might be required to comply with your organization’s standards and security requirements. + After deployment, patch Windows on an ongoing basis. Either directly patch the running instances, or create a new AMI with the latest Windows patches and replace the instances (one at a time) using the new AMI. AWS releases new Windows AMIs monthly that contain the latest operating system patches, drivers, and launch agents. We recommend that you check for the latest AMI when you launch new instances, or when you build your own custom images. + The Amazon EC2 instances are configured to allow all outgoing traffic. When deployed in a production environment, outbound rules in the security group should be put in place to restrict this traffic to the required destinations. + The FSx for Windows File Server file system can automatically record audit logs for file share and file and folder access and ship them to your desired destination if this is a requirement in your environment. + Rotate Secrets Manager secrets automatically on a regular basis. For the Amazon EC2 instance key pair, consider an automated rotation solution as described in [How to use AWS Secrets Manager to securely store and rotate SSH key pairs](https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-manager-securely-store-rotate-ssh-key-pairs/). For the Active Directory credentials and SQL Server `sa` credential secrets, set up automated rotation according to your policies for password management. **Active Directory management** + As part of the FCI cluster, Windows generates a Computer Name Object (CNO) in Active Directory. The CNO responds to DNS requests and forwards traffic to the active SQL node. We do *not* recommend using this Active Directory-provided DNS. The TTL is too high to provide a reasonable failover time and often takes upwards of 5 minutes to reflect the new primary IP address. In contrast, for highly available installations, the internal Network Load Balancer is configured to failover within 30 seconds. + An Active Directory domain administrator is required to create the cluster. This requirement is because of the elevated permissions required to create the cluster objects and modify permissions in Active Directory. However, the SQL Server services don’t need to run as a domain administrator. Therefore, we recommend that you create a second Active Directory user for this purpose. However, you can eliminate this user if the services will run as the domain administrator user. In that case, the domain administrator user must be added to the Active Directory administrators group that is created as part of this pattern. ## Epics ### Set up cluster credentials | Task | Description | Skills required | | --- | --- | --- | | Create Active Directory groups. | In AWS Managed Microsoft AD, create the following groups:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.html)For more details, see [Creating an AWS Managed Microsoft AD group](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_create_group.html) in the AWS documentation. | AD administrator | | Create Active Directory users. | In AWS Managed Microsoft AD, create the following users[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.html)For more details, see [Creating an AWS Managed Microsoft AD user](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_manage_users_groups_create_user.html) in the AWS documentation. | AD administrator | | Add Active Directory credentials to secrets. | Use Secrets Manager to create four secrets to store the following information:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.html)For more details, see [Create an AWS Secrets Manager secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) in the AWS documentation. | AWS administrator | ### Prepare the environment | Task | Description | Skills required | | --- | --- | --- | | Create the Windows AMI. | Create a custom Windows AMI that includes the prerequisite software and configurations. For more details, see [Additional Information](#deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx-additional). | AWS administrator, AWS DevOps | | Install Terraform. | To install Terraform, follow the instructions on the [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) website. | AWS DevOps | | Clone the repository. | Clone this pattern’s [repository](https://github.com/aws-samples/cluster-amazon-elastic-compute-cloud-amazon-fsx-microsoft-sql-server). For more details, see [Cloning a repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) on the GitHub website. | AWS DevOps | ### Install the cluster | Task | Description | Skills required | | --- | --- | --- | | Modify the Terraform variables. | Update the supplied `terraform.tfvars` file to set all variables to values appropriate for your environment.For example, update the `domain_group_administrators` and `domain_group_rdp_users` variables to use your Active Directory domain name and the name of the previously created Active Directory groups. | AWS DevOps | | Initialize Terraform. | To see the proposed deployment, navigate to the root of the repository. Use the Terraform command line interface (CLI) to run `terraform init` and then run `terraform plan`. | AWS DevOps | | Deploy resources. | To deploy the SQL cluster and associated resources, use the Terraform CLI to run `terraform apply`. | AWS DevOps, AWS administrator | | Validate the deployment. | To validate the deployment, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.html) | DBA, AWS systems administrator | ## Troubleshooting | Issue | Solution | | --- | --- | | Terraform provisioning completed but Windows Failover Cluster Manager does not show a cluster has been created or the cluster is in a non-operable state. | The entire installation of the resources and configuration of the clusters can can take 45-60 minutes. After Terraform has completed, the user data script must run to completion, which requires multiple reboots. To monitor progress, you can use the `Checkpoints` directory in the `C:\` drive and the SQL Server installation logs in `C:\Program Data\Microsoft SQL Server\150\Log`. When finished, the **Installation Complete** message is available in the `C:\ProgramData\Amazon\EC2-Windows\Launch\Log\UserdataExecution.log` file. | | After provisioning a functioning cluster, using Terraform to delete and re-create the cluster does not succeed. Terraform completes, but the cluster is not set up properly. | Part of the provisioning process involves registering machines and virtual objects to Active Directory and Active Directory DNS. When computer names exist for the Amazon EC2 cluster nodes and the cluster node, FCI cannot initialize correctly and will fail provisioning.To fix this issue, perform the following steps: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.html) | ## Related resources **AWS documentation** + [Create custom images with Image Builder](https://docs.aws.amazon.com/imagebuilder/latest/userguide/create-images.html) + [Create a KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) + [Creating a general purpose bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) + [Creating a key policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-overview.html) + [Creating your AWS Managed Microsoft AD](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_getting_started_create_directory.html) ## Additional information **Terraform module information** This module uses a mixture of AMI configuration and user data configuration to obtain a good mix of provisioning time and stability. During provisioning, Windows requires multiple restarts and waits. A checkpoints method has been implemented to protect against infinite loops during restarts of the persistent user data. User data is configured to be persistent. Therefore, the user data configuration scripts have, and must continue, to be developed to be idempotent. Idempotency streamlines the updating process, allowing for instances to be swapped out during an update cycle without manual configuration to rejoin or re-create FCI clusters. **SQL Server connection strings and failover clustering** The module will publish a secret containing the endpoint address that should be used in connection strings for this database. The secret name follows the format `{environment_name}/sqlserver/{cluster_name}/endpoint`. For installations where only one node is used, you can expect this to be the IP address of the Amazon EC2 instance SQL Server interface. For high availability installations (two instances), you can expect this to be the DNS name of the internal Network Load Balancer. Failover clustering virtual IPs aren’t supported in this module. A virtual IP must remain in the same subnet in order to work. In AWS, a single subnet can’t span multiple Availability Zones. Therefore, the use of virtual IPs would remove the ability for this module to be considered highly available. Each Amazon EC2 instance is given three private IP addresses. Their usais as follows: + **Primary IP for network traffic** – The source IP for egress traffic. + **FCI communications** – Used to maintain the state and sync of the failover cluster. + **SQL Server (TCP port 1433)** – Listener and also listens for heartbeat traffic to determine which instance is primary. **VPC configuration** The [prerequisites ](#deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx-prereqs)list a DHCP options set that’s configured to use Active Directory for DNS resolution. However, this prerequisite isn’t a hard requirement. The hard requirement is that the EC2 instances must be able to resolve your Active Directory domain name. Meeting this requirement can be accomplished in other ways, such as by using Amazon Route 53 Resolver endpoints. For more information, see [Integrating your Directory Service’s DNS resolution with Amazon Route 53 Resolvers](https://aws.amazon.com/blogs/networking-and-content-delivery/integrating-your-directory-services-dns-resolution-with-amazon-route-53-resolvers/) (AWS Blog post). **AMI configuration** The AMI used in this pattern must contain the following prerequisite software and configurations: 1. Download and expand the SQL Server 2019 installation files into `C:\SQL_Install_media`. 1. Install the following Windows features: + `Install-WindowsFeature Failover-Clustering` + `Install-WindowsFeature RSAT-AD-PowerShell` + `Install-WindowsFeature RSAT-AD-Tools` + `Install-WindowsFeature RSAT-Clustering-Mgmt` + `Install-WindowsFeature RSAT-Clustering-PowerShell` + `Install-WindowsFeature RSAT-Clustering-CmdInterface` 1. Disable the Windows firewall as follows: + `Get-NetFirewallProfile | Set-NetFirewallProfile -Enabled False` 1. Enable CredSSP authentication method (replace `
DECLARE| DBA | | Run the PL/SQL block. | Run the source PL/SQL block in Oracle. If there are gaps between the index values of an associative array, no data is stored in those gaps. This allows the Oracle loop to iterate only through the index positions. | DBA | | Review the output. | Five elements were inserted into the array (`cc`) at nonconsecutive intervals. The array count is shown in the following output:
TYPE country_codes IS TABLE OF VARCHAR2(100) INDEX BY pls_integer;
cc country_codes;
cc_idx NUMBER := NULL;
BEGIN
cc(7) := 'India';
cc(3) := 'UK';
cc(5) := 'USA';
cc(0) := 'China';
cc(-2) := 'Invalid';
dbms_output.put_line('cc_length:' || cc.COUNT);
IF (cc.COUNT > 0) THEN
cc_idx := cc.FIRST;
FOR i IN 1..cc.COUNT LOOP
dbms_output.put_line('cc_idx:' || cc_idx || ' country:' || cc(cc_idx));
cc_idx := cc.next(cc_idx);
END LOOP;
END IF;
END;
cc_length:5| DBA | ### PostgreSQL associative array behavior (target) | Task | Description | Skills required | | --- | --- | --- | | Create a target PL/pgSQL block in PostgreSQL. | Create a target PL/pgSQL block in PostgreSQL that uses the following associative array:
cc_idx:-2 country:Invalid
cc_idx:0 country:China
cc_idx:3 country:UK
cc_idx:5 country:USA
cc_idx:7 country:India
DO $$| DBA | | Run the PL/pgSQL block. | Run the target PL/pgSQL block in PostgreSQL. If there are gaps between the index values of an associative array, no data is stored in those gaps. This allows the Oracle loop to iterate only through the index positions. | DBA | | Review the output. | The array length is greater than 5 because `NULL` is stored in the gaps between index positions. As shown in the following output, the loop completes 10 iterations to retrieve 5 values in the array.
DECLARE
cc character varying(100)[];
cc_idx integer := NULL;
BEGIN
cc[7] := 'India';
cc[3] := 'UK';
cc[5] := 'USA';
cc[0] := 'China';
cc[-2] := 'Invalid';
RAISE NOTICE 'cc_length: %', ARRAY_LENGTH(cc, 1);
IF (ARRAY_LENGTH(cc, 1) > 0) THEN
FOR i IN ARRAY_LOWER(cc, 1)..ARRAY_UPPER(cc, 1)
LOOP
RAISE NOTICE 'cc_idx:% country:%', i, cc[i];
END LOOP;
END IF;
END;
$$;
cc_length:10| DBA | ### Emulate Oracle associative array behavior | Task | Description | Skills required | | --- | --- | --- | | Create a target PL/pgSQL block with an array and user-defined type. | To optimize performance and match Oracle's functionality, we can create a user-defined type that stores both index positions and their corresponding data. This approach reduces unnecessary iterations by maintaining direct associations between indices and values.
cc_idx:-2 country:Invalid
cc_idx:-1 country:
cc_idx:0 country:China
cc_idx:1 country:
cc_idx:2 country:
cc_idx:3 country:UK
cc_idx:4 country:
cc_idx:5 country:USA
cc_idx:6 country:
cc_idx:7 country:India
DO $$| DBA | | Run the PL/pgSQL block. | Run the target PL/pgSQL block. If there are gaps between the index values of an associative array, no data is stored in those gaps. This allows the Oracle loop to iterate only through the index positions. | DBA | | Review the output. | As shown in the following output, the user-defined type stores only populated data elements, which means that the array length matches the number of values. As a result, `LOOP` iterations are optimized to process only existing data, eliminating the need to track empty positions.
DECLARE
cc country_codes[];
cc_append country_codes := NULL;
i record;
BEGIN
cc_append.idx = 7;
cc_append.val = 'India';
cc := array_append(cc, cc_append);
cc_append.idx = 3;
cc_append.val = 'UK';
cc := array_append(cc, cc_append);
cc_append.idx = 5;
cc_append.val = 'USA';
cc := array_append(cc, cc_append);
cc_append.idx = 0;
cc_append.val = 'China';
cc := array_append(cc, cc_append);
cc_append.idx = - 2;
cc_append.val = 'Invalid';
cc := array_append(cc, cc_append);
RAISE NOTICE 'cc_length: %', ARRAY_LENGTH(cc, 1);
IF (ARRAY_LENGTH(cc, 1) > 0) THEN
FOR i IN (
SELECT
*
FROM
unnest(cc)
ORDER BY
idx)
LOOP
RAISE NOTICE 'cc_idx:% country:%', i.idx, i.val;
END LOOP;
END IF;
END;
$$;
cc_length:5| DBA | ## Related resources **AWS documentation** + [AWS database blog](https://aws.amazon.com/blogs/database/) + [Oracle to Aurora PostgreSQL migration playbook](https://docs.aws.amazon.com/dms/latest/oracle-to-aurora-postgresql-migration-playbook/chap-oracle-aurora-pg.html) **Other documentation** + [Oracle associative arrays](https://docs.oracle.com/en/database/oracle/oracle-database/23/lnpls/associative-arrays.html#GUID-8060F01F-B53B-48D4-9239-7EA8461C2170) + [PostgreSQL array functions and operators](https://www.postgresql.org/docs/current/functions-array.html) + [PostgreSQL user-defined types](https://www.postgresql.org/docs/current/sql-createtype.html) # Enable encrypted connections for PostgreSQL DB instances in Amazon RDS *Rohit Kapoor, Amazon Web Services* ## Summary Amazon Relational Database Service (Amazon RDS) supports SSL encryption for PostgreSQL DB instances. Using SSL, you can encrypt a PostgreSQL connection between your applications and your Amazon RDS for PostgreSQL DB instances. By default, Amazon RDS for PostgreSQL uses SSL/TLS and expects all clients to connect by using SSL/TLS encryption. Amazon RDS for PostgreSQL supports TLS versions 1.1 and 1.2. This pattern describes how you can enable encrypted connections for an Amazon RDS for PostgreSQL DB instance. You can use the same process to enable encrypted connections for Amazon Aurora PostgreSQL-Compatible Edition. ## Prerequisites and limitations + An active AWS account + An [Amazon RDS for PostgreSQL DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_RDS_Configuring.html) + An [SSL bundle](https://www.postgresql.org/docs/current/ssl-tcp.html) ## Architecture ![\[Enabling encrypted connections for PostgreSQL DB instances in Amazon RDS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4f87c6a3-b4ff-4248-96d3-a4a498659735/images/ccc5c880-1191-4c12-a255-6908b96b96a5.png) ## Tools + [pgAdmin](https://www.pgadmin.org/) is an open-source administration and development platform for PostgreSQL. You can use pgAdmin on Linux, Unix, macOS, and Windows to manage your database objects in PostgreSQL 10 and later. + [PostgreSQL editors](https://wiki.postgresql.org/wiki/PostgreSQL_Clients) provide a more user-friendly interface to help you create, develop, and run queries, and to edit code according to your requirements. ## Best practices + Monitor unsecure database connections. + Audit database access rights. + Make sure that backups and snapshots are encrypted at rest. + Monitor database access. + Avoid unrestricted access groups. + Enhance your notifications with [Amazon GuardDuty](https://docs.aws.amazon.com/guardduty/latest/ug/what-is-guardduty.html). + Monitor policy adherence regularly. ## Epics ### Download a trusted certificate and import it into your trust store | Task | Description | Skills required | | --- | --- | --- | | Load a trusted certificate to your computer. | To add certificates to the Trusted Root Certification Authorities store for your computer, follow these steps. (These instructions use Window Server as a example.)[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html) | DevOps engineer, Migration engineer, DBA | ### Force SSL connections | Task | Description | Skills required | | --- | --- | --- | | Create a parameter group and set the rds.force\$1ssl parameter. | If the PostgreSQL DB instance has a custom parameter group, edit the parameter group and change `rds.force_ssl` to 1.If the DB instance uses the default parameter group that doesn’t have `rds.force_ssl` enabled, create a new parameter group. You can modify the new parameter group by using the Amazon RDS API or manually as in the following instructions.To create a new parameter group:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html)To associate the parameter group with your PostgreSQL DB instance:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html)For more information, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html). | DevOps engineer, Migration engineer, DBA | | Force SSL connections. | Connect to the Amazon RDS for PostgreSQL DB instance. Connection attempts that don’t use SSL are rejected with an error message. For more information, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.SSL.html#PostgreSQL.Concepts.General.SSL.Requiring). | DevOps engineer, Migration engineer, DBA | ### Install SSL extension | Task | Description | Skills required | | --- | --- | --- | | Install the SSL extension. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html)For more information, see the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.SSL.html). | DevOps engineer, Migration engineer, DBA | ### Configure your PostgreSQL client for SSL | Task | Description | Skills required | | --- | --- | --- | | Configure a client for SSL. | By using SSL, you can start the PostgreSQL server with support for encrypted connections that use TLS protocols. The server listens for both standard and SSL connections on the same TCP port, and negotiates with any connecting client on whether to use SSL. By default, this is a client option.If you’re using the psql client:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html)For other PostgreSQL clients:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html)Review the following pages for these clients:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.html) | DevOps engineer, Migration engineer, DBA | ## Troubleshooting | Issue | Solution | | --- | --- | | Cannot download the SSL certificate. | Check your connection to the website, and retry downloading the certificate to your local computer. | ## Related resources + [Amazon RDS for PostgreSQL documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html) + [Using SSL with a PostgreSQL DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.SSL.html) (Amazon RDS documentation) + [Secure TCP/IP Connections with SSL](https://www.postgresql.org/docs/9.1/ssl-tcp.html) (PostgreSQL documentation) + [Using SSL](https://jdbc.postgresql.org/documentation/ssl/) (JDBC documentation) # Encrypt an existing Amazon RDS for PostgreSQL DB instance *Piyush Goyal, Shobana Raghu, and Yaser Raja, Amazon Web Services* ## Summary This pattern explains how to encrypt an existing Amazon Relational Database Service (Amazon RDS) for PostgreSQL DB instance in the AWS Cloud with minimal downtime. This process works for Amazon RDS for MySQL DB instances as well. You can enable encryption for an Amazon RDS DB instance when you create it, but not after it's created. However, you can add encryption to an unencrypted DB instance by creating a snapshot of your DB instance, and then creating an encrypted copy of that snapshot. You can then restore a DB instance from the encrypted snapshot to get an encrypted copy of your original DB instance. If your project allows for downtime (at least for write transactions) during this activity, this is all you need to do. When the new, encrypted copy of the DB instance becomes available, you can point your applications to the new database. However, if your project doesn’t allow for significant downtime for this activity, you need an alternate approach that helps minimize the downtime. This pattern uses the AWS Database Migration Service (AWS DMS) to migrate and continuously replicate the data so that the cutover to the new, encrypted database can be done with minimal downtime. Amazon RDS encrypted DB instances use the industry standard AES-256 encryption algorithm to encrypt your data on the server that hosts your Amazon RDS DB instances. After your data is encrypted, Amazon RDS handles authentication of access and decryption of your data transparently, with minimal impact on performance. You don't need to modify your database client applications to use encryption. ## Prerequisites and limitations **Prerequisites** + An active AWS account + An unencrypted Amazon RDS for PostgreSQL DB instance + Experience working with (creating, modifying, or stopping) AWS DMS tasks (see [Working with AWS DMS tasks](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.html) in the AWS DMS documentation) + Familiarity with AWS Key Management Service (AWS KMS) for encrypting databases (see the [AWS KMS documentation](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html)) **Limitations** + You can enable encryption for an Amazon RDS DB instance only when you create it, not after the DB instance is created. + Data in [unlogged tables](https://www.postgresql.org/docs/current/sql-createtable.html) will not be restored using snapshots. For more information, review [Best practices for working with PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html#CHAP_BestPractices.PostgreSQL). + You can't have an encrypted read replica of an unencrypted DB instance or an unencrypted read replica of an encrypted DB instance. + You can't restore an unencrypted backup or snapshot to an encrypted DB instance. + AWS DMS does not automatically transfers the Sequences therefore additional steps are required to handle this. For more information, see [Limitations of Amazon RDS encrypted DB instances](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html#Overview.Encryption.Limitations) in the Amazon RDS documentation. ## Architecture **Source architecture** + Unencrypted RDS DB instance **Target architecture ** + Encrypted RDS DB instance + The destination RDS DB instance is created by restoring the DB snapshot copy of the source RDS DB instance. + An AWS KMS key is used for encryption while restoring the snapshot. + An AWS DMS replication task is used to migrate the data. ![\[Process uses AWS DMS to encrypt an existing Amazon RDS for PostgreSQL DB instance to a new DB.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/820d17c0-0eed-4ed9-9f43-cbada081d924/images/44dd8420-d89d-466e-b7fb-1bdafab8f7f9.png) ## Tools **Tools used to enable encryption:** + AWS KMS key for encryption – When you create an encrypted DB instance, you can choose a customer managed key or the AWS managed key for Amazon RDS to encrypt your DB instance. If you don't specify the key identifier for a customer managed key, Amazon RDS uses the AWS managed key for your new DB instance. Amazon RDS creates an AWS managed key for Amazon RDS for your AWS account. Your AWS account has a different AWS managed key for Amazon RDS for each AWS Region. For more information about using KMS keys for Amazon RDS encryption, see [Encrypting Amazon RDS Resources](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html). **Tools used for ongoing replication**: + AWS DMS – You can use AWS Database Migration Service (AWS DMS) to replicate changes from the source DB to the target DB. It is important to keep the source and target DB in sync to keep downtime to a minimum. For information about setting up AWS DMS and creating tasks, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html). ## Epics ### Create a snapshot of the source DB instance and encrypt it | Task | Description | Skills required | | --- | --- | --- | | Check the details for the source PostgreSQL DB instance. | On the Amazon RDS console, choose the source PostgreSQL DB instance. On the **Configuration **tab, make sure that encryption isn't enabled for the instance. For a screen illustration, see the [Additional information](#encrypt-an-existing-amazon-rds-for-postgresql-db-instance-additional) section. | DBA | | Create the DB snapshot. | Create a DB snapshot of the instance you want to encrypt. The amount of time it takes to create a snapshot depends on the size of your database. For instructions, see [Creating a DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html) in the Amazon RDS documentation. | DBA | | Encrypt the snapshot. | In the Amazon RDS console navigation pane, choose **Snapshots**, and select the DB snapshot you created. For **Actions**, choose **Copy Snapshot**. Provide the destination AWS Region and the name of the DB snapshot copy in the corresponding fields. Select the **Enable Encryption** checkbox. For **Master Key**, specify the KMS key identifier to use to encrypt the DB snapshot copy. Choose **Copy Snapshot**. For more information, see [Copying a snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CopySnapshot.html) in the Amazon RDS documentation. | DBA | ### Prepare the target DB instance | Task | Description | Skills required | | --- | --- | --- | | Restore the DB snapshot. | On the Amazon RDS console, choose **Snapshots**. Choose the encrypted snapshot that you created. For **Actions**, choose **Restore Snapshot**. For **DB Instance Identifier**, provide a unique name for the new DB instance. Review the instance details, and then choose **Restore DB Instance**. A new, encrypted DB Instance will be created from your snapshot. For more information, see [Restoring from a DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html) in the Amazon RDS documentation. | DBA | | Migrate data by using AWS DMS. | On the AWS DMS console, create an AWS DMS task. For **Migration type**, choose **Migrate existing data and replicate ongoing changes**. In **Task Settings**, for **Target table preparation mode**, choose **Truncate**. For more information, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html) in the AWS DMS documentation. | DBA | | Enable data validation. | In **Task Settings**, choose **Enable validation**. This enables you to compare the source data to the target data to verify that the data was migrated accurately. | DBA | | Disable constraints on the target DB instance. | [Disable any triggers and foreign key constraints](https://www.postgresql.org/docs/current/sql-altertable.html) on the target DB instance, and then start the AWS DMS task. For more information about disabling triggers and foreign key constraints, see the [AWS DMS documentation.](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html) | DBA | | Verify data. | After the full load is complete, verify the data on the target DB instance to see if it matches the source data. For more information, see [AWS DMS data validation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html) in the AWS DMS documentation. | DBA | ### Cut over to the target DB instance | Task | Description | Skills required | | --- | --- | --- | | Stop write operations on the source DB instance. | Stop the write operations on the source DB instance so that application downtime can begin. Verify that AWS DMS has completed the replication for the data in the pipeline. Enable triggers and foreign keys on the target DB instance. | DBA | | Update database sequences | If the source database contains any sequence numbers, verify and update the sequences in the target database. | DBA | | Configure the application endpoint. | Configure your application connections to use the new Amazon RDS DB instance endpoints. The DB instance is now encrypted. | DBA, Application owner | ## Related resources + [Creating an AWS DMS task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html) + [Monitoring replication tasks using Amazon CloudWatch](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html#CHAP_Monitoring.CloudWatch) + [Monitoring AWS DMS tasks](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html) + [Updating the Amazon RDS encryption key](https://aws.amazon.com/premiumsupport/knowledge-center/update-encryption-key-rds/) ## Additional information Checking the encryption for the source PostgreSQL DB instance: ![\[The Summary page of source PostgreSQL DB instance shows encryption not enabled for storage.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/820d17c0-0eed-4ed9-9f43-cbada081d924/images/d53d1dab-b5c2-452d-b823-ba3d6508ad15.png) Additional notes for this pattern: + Enable replication on PostgreSQL by setting the `rds.logical_replication` parameter to 1. **Important note:** Replication slots retain the write ahead log (WAL) files until the files are externally consumed—for example, by `pg_recvlogical`; by extract, transform, and load (ETL) jobs; or by AWS DMS. When you set the `rds.logical_replication` parameter value to 1, AWS DMS sets the `wal_level`, `max_wal_senders`, `max_replication_slots`, and `max_connections` parameters. If logical replication slots are present but there is no consumer for the WAL files retained by the replication slot, you might see an increase in the transaction log disk usage and a constant decrease in free storage space. For more information and steps to resolve this issue, see the article [How can I identify what is causing the "No space left on device" or "DiskFull" error on Amazon RDS for PostgreSQL?](https://aws.amazon.com/premiumsupport/knowledge-center/diskfull-error-rds-postgresql/) in the AWS Support Knowledge Center. + Any schema changes that you make to the source DB instance after you create the DB snapshot will not be present on the target DB instance. + After you create an encrypted DB instance, you can't change the KMS key used by that DB instance. Be sure to determine your KMS key requirements before you create your encrypted DB instance. + You must disable triggers and foreign keys on the target DB instance before you run the AWS DMS task. You can re-enable these when the task is complete. # Enforce automatic tagging of Amazon RDS databases at launch *Susanne Kangnoh and Archit Mathur, Amazon Web Services* ## Summary Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the Amazon Web Services (AWS) Cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks. You can use tagging to categorize your AWS resources in different ways. Relational database tagging is useful when you have many resources in your account and you want to quickly identify a specific resource based on the tags. You can use Amazon RDS tags to add custom metadata to your RDS DB instances. A tag consists of a user-defined key and value. We recommend that you create a consistent set of tags to meet your organization's requirements. This pattern provides an AWS CloudFormation template to help you monitor and tag RDS DB instances. The template creates an Amazon CloudWatch Events event that watches for the AWS CloudTrail **CreateDBInstance **event. (CloudTrail captures API calls for Amazon RDS as events.) When it detects this event, it calls an AWS Lambda function that automatically applies tag keys and values that you define. The template also sends out a notification that the instance has been tagged, by using Amazon Simple Notification Service (Amazon SNS). ## Prerequisites and limitations **Prerequisites ** + An active AWS account. + An Amazon Simple Storage Service (Amazon S3) bucket to upload the Lambda code. + An email address where you would like to receive tagging notifications. **Limitations ** + The solution supports CloudTrail **CreateDBInstance** events. It does not create notifications for any other events. ## Architecture **Workflow architecture** ![\[Workflow diagram showing AWS services interaction for RDS instance creation and notification.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5541bc1e-e00f-4b5a-94b7-bb1808b5591a/images/ec0bcf92-f986-4af3-bfe7-d2c5e04051c5.png) **Automation and scale** + You can use the AWS CloudFormation template multiple times for different AWS Regions and accounts. You need to run the template only once in each Region or account. ## Tools **AWS services** + [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html) – AWS CloudTrail is an AWS service that helps you with governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. + [Amazon CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html) – Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. CloudWatch Events becomes aware of operational changes as they occur and takes corrective action as necessary, by sending messages to respond to the environment, activating functions, making changes, and capturing state information. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) – AWS Lambda is a compute service that supports running code without needing to provision or manage servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume—there is no charge when your code is not running. + [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) – Amazon Simple Storage Service (Amazon S3) is a highly scalable object storage service that can be used for a wide range of storage solutions, including websites, mobile applications, backups, and data lakes. + [Amazon SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) – Amazon Simple Notification Service (Amazon SNS) is a web service that enables applications, end-users, and devices to instantly send and receive notifications from the cloud. **Code ** This pattern includes an attachment with two files: + `index.zip` is a compressed file that includes the Lambda code for this pattern. + `rds.yaml` is a CloudFormation template that deploys the Lambda code. See the *Epics *section for information about how to use these files. ## Epics ### Deploy the Lambda code | Task | Description | Skills required | | --- | --- | --- | | Upload the code to an S3 bucket. | Create a new S3 bucket or use an existing S3 bucket to upload the attached `index.zip` file (Lambda code). This bucket must be in the same AWS Region as the resources (RDS DB instances) that you want to monitor. | Cloud architect | | Deploy the CloudFormation template. | Open the Cloudformation console in the same AWS Region as the S3 bucket, and deploy the `rds.yaml` file that's provided in the attachment. In the next epic, provide values for the template parameters. | Cloud architect | ### Complete the parameters in the CloudFormation template | Task | Description | Skills required | | --- | --- | --- | | Provide the S3 bucket name. | Enter the name of the S3 bucket that you created or selected in the first epic. This S3 bucket contains the .zip file for the Lambda code and must be in the same AWS Region as the CloudFormation template and the RDS DB instances that you want to monitor. | Cloud architect | | Provide the S3 key. | Provide the location of the Lambda code .zip file in your S3 bucket, without leading slashes (for example, `index.zip` or `controls/index.zip`). | Cloud architect | | Provide an email address. | Provide an active email address where you want to receive violation notifications. | Cloud architect | | Specify a logging level. | Specify the logging level and verbosity. `Info` designates detailed informational messages on the application’s progress and should be used only for debugging. `Error` designates error events that could still allow the application to continue running. `Warning` designates potentially harmful situations. | Cloud architect | | Enter the tag keys and values for your RDS DB instances. | Enter the required tag keys and values that you want to automatically apply to the RDS instance. For more information, see [Tagging Amazon RDS resources](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_Tagging.html) in the AWS documentation. | Cloud architect | ### Confirm the subscription | Task | Description | Skills required | | --- | --- | --- | | Confirm the email subscription. | When the CloudFormation template deploys successfully, it sends a subscription email message to the email address you provided. To receive notifications when your instances are tagged, you must confirm this email subscription. | Cloud architect | ## Related resources + [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) (Amazon S3 documentation) + [Tagging Amazon RDS resources](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_Tagging.html) (Amazon Aurora documentation) + [Uploading objects](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/upload-objects.html) (Amazon S3 documentation) + [Creating a CloudWatch Events rule that triggers on an AWS API call using AWS CloudTrail](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-CloudTrail-Rule.html) (Amazon CloudWatch documentation) ## Attachments To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/5541bc1e-e00f-4b5a-94b7-bb1808b5591a/attachments/attachment.zip) # Estimate the cost of a DynamoDB table for on-demand capacity *Moinul Al-Mamun, Amazon Web Services* ## Summary [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) is a NoSQL transactional database that provides single-digit millisecond latency even at petabytes scale. This Amazon Web Services (AWS) serverless offering is getting popular because of its consistent performance and scalability. You do not need to provision underlying infrastructure. Your single table can grow up to petabytes. With on-demand capacity mode, you pay per request for the data reads and writes that your application performs on the tables. AWS charges are based on the accumulated read request units (RRUs) and write request units (WRUs) in a month. DynamoDB monitors the size of your table continuously throughout the month to determine your storage charges. It supports continuous backup with point-in-time-recovery (PITR). DynamoDB monitors the size of your PITR-enabled tables continuously throughout the month to determine your backup charges. To estimate the DynamoDB cost for a project, it’s important to calculate how much RRU, WRU, and storage will be consumed at different stages of your product lifecycle. For a rough cost estimation, you can use [AWS Pricing Calculator](https://calculator.aws/#/createCalculator/DynamoDB), but you must provide an approximate number of RRUs, WRUs, and storage requirements for your table. These can be difficult to estimate at the beginning of the project. AWS Pricing Calculator doesn’t consider data growth rate or item size, and it doesn’t consider the number of reads and writes for the base table and global secondary indexes (GSIs) separately. To use AWS Pricing Calculator, you must estimate all those aspects to assume ballpark figures for WRU, RRU, and storage size to obtain your cost estimation. This pattern provides a mechanism and a re-usable Microsoft Excel template to estimate basic DynamoDB cost factors, such as write, read, storage, backup and recovery cost, for on-demand capacity mode. It is more granular than AWS Pricing Calculator, and it considers base table and GSIs requirements independently. It also considers the monthly item data growth rate and forecasts costs for three years. ## Prerequisites and limitations **Prerequisites ** + Basic knowledge of DynamoDB and DynamoDB data model design + Basic knowledge of DynamoDB pricing, WRU, RRU, storage, and backup and recovery (for more information, see [Pricing for On-Demand Capacity](https://aws.amazon.com/dynamodb/pricing/on-demand/)) + Knowledge of your data, data model, and item size in DynamoDB + Knowledge of DynamoDB GSIs **Limitations** + The template provides you with an approximate calculation, but it isn’t appropriate for all configurations. To get a more accurate estimate, you must measure the individual item size for each item in the base table and GSIs. + For a more accurate estimate, you must consider the expected number of writes (insert, update, and delete) and reads for each item in an average month. + This pattern supports estimating only write, read, storage, and backup and recovery costs for next few years based on fixed data growth assumptions. ## Tools **AWS services** + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance. **Other tools** + [AWS Pricing Calculator](https://calculator.aws/#/createCalculator/DynamoDB) is a web-based planning tool that you can use to create estimates for your AWS use cases. ## Best practices To help keep costs low, consider the following DynamoDB design best practices. + [Partition key design](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-uniform-load.html) – Use a high-cardinality partition key to distribute load evenly. + [Adjacency list design pattern](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html) – Use this design pattern for managing one-to-many and many-to-many relationships. + [Sparse index](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-general-sparse-indexes.html) – Use sparse index for your GSIs. When you create a GSI, you specify a partition key and optionally a sort key. Only items in the base table that contain a corresponding GSI partition key appear in the sparse index. This helps to keep GSIs smaller. + [Index overloading](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-gsi-overloading.html) – Use the same GSI for indexing various types of items. + [GSI write sharding](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-gsi-sharding.html) – Shard wisely to distribute data across the partitions for efficient and faster queries. + [Large items](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-use-s3-too.html) – Store only metadata inside the table, save the blob in Amazon S3, and keep the reference in DynamoDB. Break large items into multiple items, and efficiently index by using sort keys. For more design best practices, see the [Amazon DynamoDB Developer Guide](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html). ## Epics ### Extract item information from your DynamoDB data model | Task | Description | Skills required | | --- | --- | --- | | Get item size. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.html) | Data engineer | | Estimate the write cost. | To estimate write cost in on-demand capacity mode, first you have to measure how many WRUs will be consumed in a month. For that, you need to consider the following factors:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.html)For more information, see the *Additional information* section. | Data engineer | | Estimate the read cost. | To estimate read cost in on-demand mode, first you have to measure how many RRUs will be consumed in a month. For that, you need to consider the following factors: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.html) | Data engineer, App developer | | Estimate the storage size and cost. | First, estimate the average monthly storage requirement based on your item size in the table. Then calculate storage cost by multiplying storage size by the per GB storage price for your AWS Region. If you already entered data for estimating the write cost, you don't need to enter it again for calculating storage size. Otherwise, to estimate storage size, you need to consider the following factors: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.html) | Data engineer | ### Enter the item and object information in the Excel template | Task | Description | Skills required | | --- | --- | --- | | Download the Excel template from the Attachments section, and adjust it for your use case table. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.html) | Data engineer | | Enter information in the Excel template. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-cost-of-a-dynamodb-table-for-on-demand-capacity.html)In the template, there are three items, or entities: information, metadata, and relationship. There are two GSIs. For your use case, if you need more items, create new rows. If you need more GSIs, copy an existing GSI block, and paste to create as many GSI blocks as you need. Then adjust the SUM and TOTAL column calculations. | Data engineer | ## Related resources **References** + [Amazon DynamoDB pricing for on-demand capacity](https://aws.amazon.com/dynamodb/pricing/on-demand/) + [AWS Pricing Calculator for DynamoDB](https://calculator.aws/#/createCalculator/DynamoDB) + [Best practices for designing and architecting with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html) + [Getting started with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html) **Guides and patterns** + [Modeling data with Amazon DynamoDB](https://docs.aws.amazon.com/prescriptive-guidance/latest/dynamodb-data-modeling/) + [Estimate storage costs for an Amazon DynamoDB table](https://apg-library.amazonaws.com/content/9b74399d-9655-47ee-b9b3-de46b65bc4e3) ## Additional information **Write cost calculation example** The DynamoDB data model design shows three items for a product, and an average item size of 4 KB. When you add a new product into the DynamoDB base table, it consumes the number of items \$1 (item size/1 KB write unit) = 3 \$1 (4/1) = 12 WRU. In this example, for writing 1 KB, the product consumes 1 WRU. **Read cost calculation example** To get the RRU estimation, consider the average of how many times each item will be read in a month. For example, the Information item will be read, on average, 10 times in a month, and the metadata item will be read two times, and the relationship item will be read five times. In the example template, total RRU for all components = number of new components created each month \$1 RRU per component per month = 10 million \$1 17 RRU = 170 million RRU each month. Every month, new things (components or products will be added, and the total number of products will grow over time. So, RRU requirements will also grow over time. + For the first month RRU, consumption will be 170 million. + For the second month, RRU consumption will be 2 \$1 170 million = 340 million. + For the third month RRU consumption will be 3 \$1 170 million = 510 million. The following graph shows monthly RRU consumption and cost forecasting. ![\[The RRU consumption rising more steeply than the cost.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/1797b48f-a183-4f25-811f-44921c3a48ee/images/3505bfc8-694d-4acc-8585-cd71258fa315.png) Note that prices within the graph are for illustration only. To create accurate forecasts for your use case, check the AWS pricing page, and use those prices in the Excel sheet. **Storage, backup, and recovery cost calculation examples** DynamoDB storage, backup and restore all are connected with each other. Backup is directly connected with storage, and recovery is directly connected with backup size. As the table size increases, corresponding storage, backup, and restore costs will increase proportionally. *Storage size and cost* Storage cost will increase over time based on your data growth rate. For example, assume that the average size of a component or product in the base table and GSIs is 11 KB, and 10 million new products will be added every month into your database table. In that case, your DynamoDB table size will grow (11 KB \$1 10 million)/1024/1024 = 105 GB per month. At the first month, your table storage size will be 105 GB, at second month it will be 105 \$1 105 = 210 GBs, and so on. + For the first month, storage cost will be 105 GB \$1 storage price per GB for your AWS Region. + For the second month, storage cost will be 210 GB \$1 storage price per GB for your Region. + For the third month, storage cost will be 315 GB \$1 storage price per GB for your Region. For storage size and cost for next three years, see the *Storage size and forecasting* section. *Backup cost* Backup cost will increase over time based on your data growth rate. When you turn on continuous backup with point-in-time-recovery (PITR), continuous backup charges are based on average storage GB-Month. In a calendar month, the average backup size would be the same as your table storage size, although the actual size could be a bit different. As new products will be added every month, the total storage size and the backup size will grow over time. For example, for the first month, the average backup size of 105 GB could grow to 210 GB for the second month. + For the first month, backup cost will be 105 GB-month \$1 continuous backup price per GB for your AWS Region. + For the second month, backup cost will be 210 GB-month \$1 continuous backup price per GB for your Region. + For the third month, backup cost will be 315 GB-month \$1 continuous backup price per GB for your Region. + and, so on Backup cost is included in the graph in the *Storage size and cost forecasting* section. *Recovery cost* When you are taking continuous backup with PITR enabled, recovery operation charges are based on the size of the restore. Each time that you restore, you pay based on gigabytes of restored data. If your table size is large and you perform restore multiple times in a month, it will be costly. To estimate restore cost, this example assumes that you perform a PITR recovery one time each month at the end of the month. The example uses the monthly average backup size as the restore data size for that month. For the first month, the average backup size is 105 GB, and for the recovery at end of the month, the restore data size would be 105 GB. For the second month, it would be 210 GBs, and so on. Recovery cost will increase over time based on your data growth rate. + For the first month, recovery cost will be 105 GB \$1 restore price per GB for your AWS Region. + For the second month, recovery cost will be 210 GB \$1 restore price per GB for your Region. + For the third month, recovery cost will be 315 GB \$1 restore price per GB for your Region. For more information, see the Storage, backup and recovery tab in the Excel template and the graph in the following section. *Storage size and cost forecasting* In the template, actual billable storage size is calculated by subtracting the free tier 25 GB per month for the Standard table class. In the sheet, you will get a forecasting graph broken out into monthly values. The following example chart forecasts monthly storage size in GB, billable storage cost, on-demand backup cost, and recovery cost for next 36 calendar months. All costs are in USD. From the graph, it’s clear that storage, backup, and recovery costs increase proportionally to increases in storage size. ![\[The storage size rising above three thousand while the costs are less than one thousand.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/1797b48f-a183-4f25-811f-44921c3a48ee/images/fd9f06d0-bc9c-4b4e-8cbd-3e527fe09e88.png) Note that prices used in the graph are for illustration purposes only. To create accurate prices for your use case, check the AWS pricing page, and use those prices in the Excel template. ## Attachments To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/1797b48f-a183-4f25-811f-44921c3a48ee/attachments/attachment.zip) # Estimate storage costs for an Amazon DynamoDB table *Moinul Al-Mamun, Amazon Web Services* ## Summary Amazon DynamoDB is a NoSQL transactional database that provides single-digit millisecond latency even at petabytes scale. This popular serverless offering from AWS provides consistent performance and scalability. You do not need to provision storage, and your single table can grow up to petabytes. DynamoDB monitors the size of your table continuously throughout the month to determine your storage charges. AWS then charges you for the average size of storage in gigabytes. The more that your table grows over time, the more that your storage cost will grow. To calculate storage cost, you can use [AWS Pricing Calculator](https://calculator.aws/#/createCalculator/DynamoDB), but you need to provide the approximate size of your table, including global secondary indexes (GSIs), which is really difficult to estimate at the beginning of the project. Also, AWS Pricing Calculator does not consider the data growth rate. This pattern provides a mechanism and a reusable Microsoft Excel template to calculate DynamoDB storage size and cost. It considers the storage requirements for the base table and the GSIs independently. It calculates storage size by considering the size of your individual items and data growth rate over time. To get an estimate, insert two pieces of information into the template: + The individual item size in kilobytes for the base table and GSIs + How many new objects or products could be added to the table, on average, in a month, (for example, 10 million) The template generates a storage and cost forecasting graph for the next three years, which is shown in the following example. ![\[Lower line for cost moves up slowly and the higher line for storage moves up more quickly.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9b74399d-9655-47ee-b9b3-de46b65bc4e3/images/c0436252-cc42-4ea3-ac50-c0455aece39d.png) ## Prerequisites and limitations **Prerequisites ** + Basic knowledge of DynamoDB, including DynamoDB storage and pricing + Knowledge of your data, data model, and item size in DynamoDB + Knowledge of DynamoDB global secondary indexes (GSIs) **Limitations ** + The template provides you with an approximate calculation, but it isn’t appropriate for all configurations. To get a more accurate estimate, you must measure the individual item size for each item in the base table and GSIs. + This pattern supports estimating only storage size and cost for the next few years based on fixed data growth assumptions. ## Tools **AWS services** + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance. **Other tools** + [AWS Pricing Calculator](https://docs.aws.amazon.com/pricing-calculator/latest/userguide/what-is-pricing-calculator.html) is a web-based planning tool that you can use to create estimates for your AWS use cases. ## Epics ### Extract item information from your DynamoDB data model | Task | Description | Skills required | | --- | --- | --- | | Get item size. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-storage-costs-for-an-amazon-dynamodb-table.html) | Data engineer | | Get the number of objects added in a month. | Estimate how many components, or objects, will be added into the DynamoDB table, on average, in one month. | Data engineer | ### Enter the item and object information in the Excel template | Task | Description | Skills required | | --- | --- | --- | | Download and adjust the Excel spreadsheet. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-storage-costs-for-an-amazon-dynamodb-table.html) | Data engineer | | Enter information in the Excel template. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-storage-costs-for-an-amazon-dynamodb-table.html) | Data engineer | ## Related resources + [Pricing for On-Demand Capacity](https://aws.amazon.com/dynamodb/pricing/on-demand/) for Amazon DynamoDB + [AWS Pricing Calculator for DynamoDB](https://calculator.aws/#/createCalculator/DynamoDB) ## Additional information Note that the attached template forecasts only storage size and cost for Standard storage table class. Based on the forecast for storage costs, and considering individual item size and product or object growth rate, you can estimate the following: + Data export cost + Backup and recovery cost + Data storage requirements. **Amazon DynamoDB data storage cost** DynamoDB monitors the size of your tables continuously to determine your storage charges. DynamoDB measures the size of your billable data by adding the raw byte size of your data plus a per-item storage overhead that depends on the features you have enabled. For more information, see the [DynamoDB Developer Guide](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CapacityUnitCalculations.html). The price for data storage depends on your table class. The first 25 GB stored each month is free if you’re using the DynamoDB Standard table class. For more information about storage costs for the Standard and Standard-Infrequent Access table classes in different AWS Regions, see [Pricing for On-Demand Capacity](https://aws.amazon.com/dynamodb/pricing/on-demand/). ## Attachments To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/9b74399d-9655-47ee-b9b3-de46b65bc4e3/attachments/attachment.zip) # Estimate the Amazon RDS engine size for an Oracle database by using AWR reports *Abhishek Verma and Eduardo Valentim, Amazon Web Services* ## Summary When you migrate an Oracle database to Amazon Relational Database Service (Amazon RDS) or Amazon Aurora, computing the CPU, memory, and disk I/O for the target database is a key requirement. You can estimate the required capacity of the target database by analyzing the Oracle Automatic Workload Repository (AWR) reports. This pattern explains how to use AWR reports to estimate these values. The source Oracle database could be on premises or hosted on an Amazon Elastic Compute Cloud (Amazon EC2) instance, or it could be an Amazon RDS for Oracle DB instance. The target database could be any Amazon RDS or Aurora database. **Note** Capacity estimates will be more precise if your target database engine is Oracle. For other Amazon RDS databases, the engine size can vary due to differences in database architecture. We recommend that you run the performance test before you migrate your Oracle database. ## Prerequisites and limitations **Prerequisites** + An Oracle Database Enterprise Edition license and Oracle Diagnostics Pack license in order to download AWR reports. **Product versions** + All Oracle Database editions for versions 11g (versions 11.2.0.3.v1 and later) and up to 12.2, and 18c,19c. + This pattern doesn’t cover Oracle Engineered Systems or Oracle Cloud Infrastructure (OCI). ## Architecture **Source technology stack ** One of the following: + An on-premises Oracle database + An Oracle database on an EC2 instance + An Amazon RDS for Oracle DB instance **Target technology stack** + Any Amazon RDS or Amazon Aurora database **Target architecture** For information about the full migration process, see the pattern [Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-database-to-aurora-postgresql-using-aws-dms-and-aws-sct.html). **Automation and scale** If you have multiple Oracle databases to migrate and you want to use additional performance metrics, you can automate the process by following the steps described in the blog post [Right-size Amazon RDS instances at scale based on Oracle performance metrics](https://aws.amazon.com/blogs/database/right-sizing-amazon-rds-instances-at-scale-based-on-oracle-performance-metrics/). ## Tools + [Oracle Automatic Workload Repository (AWR)](https://docs.oracle.com/en-us/iaas/performance-hub/doc/awr-report-ui.html) is a repository that’s built into Oracle databases. It periodically gathers and stores system activity and workload data, which is then analyzed by Automatic Database Diagnostic Monitor (ADDM). AWR takes snapshots of system performance data periodically (by default, every 60 minutes) and stores the information (by default, up to 8 days). You can use AWR views and reports to analyze this data. ## Best practices + To calculate resource requirements for your target database, you can use a single AWR report, multiple AWR reports, or dynamic AWR views. We recommend that you use multiple AWR reports during the peak load period to estimate the resources required to handle those peak loads. In addition, dynamic views provide more data points that help you calculate resource requirements more precisely. + You should estimate IOPS only for the database that you plan to migrate, not for other databases and processes that use the disk. + To calculate how much I/O is being used by the database, don’t use the information in the Load Profile section of the AWR report. Use the I/O Profile section instead, if it’s available, or skip to the Instance Activity Stats section and look at the total values for physical read and write operations. + When you estimate CPU utilization, we recommend that you use the database metrics method instead of operating system (OS) statistics, because it’s based on the CPU used only by databases. (OS statistics also include CPU usage by other processes.) You should also check CPU-related recommendations in the ADDM report to improve performance after migration. + Consider I/O throughput limits―Amazon Elastic Block Store (Amazon EBS) throughput and network throughput―for the specific instance size when you’re determining the right instance type. + Run the performance test before migration to validate the engine size. ## Epics ### Create an AWR report | Task | Description | Skills required | | --- | --- | --- | | Enable the AWR report. | To enable the report, follow the instructions in the [Oracle documentation](https://docs.oracle.com/en/database/oracle/oracle-database/18/tgdba/gathering-database-statistics.html#GUID-26D359FA-F809-4444-907C-B5AFECD9AE29). | DBA | | Check the retention period. | To check the retention period of the AWR report, use the following query.
cc_idx:-2 country:Invalid
cc_idx:0 country:China
cc_idx:3 country:UK
cc_idx:5 country:USA
cc_idx:7 country:India
SQL> SELECT snap_interval,retention FROM dba_hist_wr_control;| DBA | | Generate the snapshot. | If the AWR snapshot interval isn’t granular enough to capture the spike of the peak workload, you can generate the AWR report manually. To generate the manual AWR snapshot, use the following query.
SQL> EXEC dbms_workload_repository.create_snapshot;| DBA | | Check recent snapshots. | To check recent AWR snapshots, use the following query.
SQL> SELECT snap_id, to_char(begin_interval_time,'dd/MON/yy hh24:mi') Begin_Interval,| DBA | ### Estimate disk I/O requirements | Task | Description | Skills required | | --- | --- | --- | | Choose a method. | IOPS is the standard measure of input and output operations per second on a storage device, and includes both read and write operations. If you are migrating an on-premises database to AWS, you need to determine the peak disk I/O used by the database. You can use the following methods to estimate disk I/O for your target database:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)The following steps describe these four methods. | DBA | | Option 1: Use the load profile. | The following table shows an example of the Load Profile section of the AWR report.For more accurate information, we recommend that you use option 2 (I/O profiles) or option 3 (instance activity statistics) instead of the load profile.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)Based on this information, you can calculate IOPs and throughput as follows:* IOPS = Read I/O requests: \$1 Write I/O requests = 3,586.8 \$1 574.7 = 4134.5 ** Throughput = Physical read (blocks) \$1 Physical write (blocks) = 13,575.1 \$1 3,467.3 = 17,042.4*Because the block size in Oracle is 8 KB, you can calculate total throughput as follows:* Total throughput in MB is 17042.4 \$1 8 \$1 1024 / 1024 / 1024 = 133.2 MB*Don’t use the load profile to estimate the instance size. It isn’t as precise as instance activity statistics or I/O profiles. | DBA | | Option 2: Use instance activity statistics. | If you’re using an Oracle Database version before 12c, you can use the Instance Activity Stats section of the AWR report to estimate IOPS and throughput. The following table shows an example of this section.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)Based on this information, you can calculate total IOPS and throughput as follows:* Total IOPS = 3,610.28 \$1 757.11 = 4367 ** Total Mbps = 114,482,426.26 \$1 36,165,631.84 = 150648058.1 / 1024 / 1024 = 143 Mbps* | DBA | | Option 3: Use I/O profiles. | In Oracle Database 12c, the AWR report includes an I/O Profiles section that presents all the information in a single table and provides more accurate data about database performance. The following table shows an example of this section.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)This table provides the following values for throughput and total IOPS:* Throughput = 143 MBPS (from the fifth row, labeled Total, second column)** IOPS = 4,367.4 (from the first row, labeled Total Requests, second column)* | DBA | | Option 4: Use AWR views. | You can see the same IOPS and throughput information by using AWR views. To get this information, use the following query:
to_char(end_interval_time,'dd/MON/yy hh24:mi') End_Interval
FROM dba_hist_snapshot
ORDER BY 1;
break on report| DBA | | | Per Second | Per Transaction | Per Exec | Per Call | | --- |--- |--- |--- |--- | | **DB Time(s):** | 26.6 | 0.2 | 0.00 | 0.02 | | **DB CPU(s):** | 18.0 | 0.1 | 0.00 | 0.01 | | **Background CPU(s):** | 0.2 | 0.0 | 0.00 | 0.00 | | **Redo size (bytes):** | 2,458,539.9 | 17,097.5 | | | | **Logical read (blocks):** | 3,371,931.5 | 23,449.6 | | | | **Block changes:** | 21,643.5 | 150.5 | | | | **Physical read (blocks):** | 13,575.1 | 94.4 | | | | **Physical write (blocks):** | 3,467.3 | 24.1 | | | | **Read IO requests:** | 3,586.8 | 24.9 | | | | **Write IO requests:** | 574.7 | 4.0 | | | | **Read IO (MB):** | 106.1 | 0.7 | | | | **Write IO (MB):** | 27.1 | 0.2 | | | | **IM scan rows:** | 0.0 | 0.0 | | | | **Session Logical Read IM:** | | | | | | **User calls:** | 1,245.7 | 8.7 | | | | **Parses (SQL):** | 4,626.2 | 32.2 | | | | **Hard parses (SQL):** | 8.9 | 0.1 | | | | **SQL Work Area (MB):** | 824.9 | 5.7 | | | | **Logons:** | 1.7 | 0.0 | | | | **Executes (SQL):** | 136,656.5 | 950.4 | | | | **Rollbacks:** | 22.9 | 0.2 | | | | **Transactions:** | 143.8 | | | | | Statistic | Total | per Second | per Trans | | --- |--- |--- |--- | | **physical read total IO requests** | 2,547,333,217 | 3,610.28 | 25.11 | | **physical read total bytes** | 80,776,296,124,928 | 114,482,426.26 | 796,149.98 | | **physical write total IO requests** | 534,198,208 | 757.11 | 5.27 | | **physical write total bytes** | 25,517,678,849,024 | 36,165,631.84 | 251,508.18 | | | Read\$1Write Per Second | Read per Second | Write Per Second | | --- |--- |--- |--- | | **Total Requests:** | 4,367.4 | 3,610.3 | 757.1 | | **Database Requests:** | 4,161.5 | 3,586.8 | 574.7 | | **Optimized Requests:** | 0.0 | 0.0 | 0.0 | | **Redo Requests:** | 179.3 | 2.8 | 176.6 | | **Total (MB):** | 143.7 | 109.2 | 34.5 | | **Database (MB):** | 133.1 | 106.1 | 27.1 | | **Optimized Total (MB):** | 0.0 | 0.0 | 0.0 | | **Redo (MB):** | 7.6 | 2.7 | 4.9 | | **Database (blocks):** | 17,042.4 | 13,575.1 | 3,467.3 | | **Via Buffer Cache (blocks):** | 5,898.5 | 5,360.9 | 537.6 | | **Direct (blocks):** | 11,143.9 | 8,214.2 | 2,929.7 | ### Estimate CPU requirements | Task | Description | Skills required | | --- | --- | --- | | Choose a method. | You can estimate the CPU required for the target database in three ways:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)If you’re looking at utilized cores, we recommend that you use the database metrics method instead of OS statistics, because it’s based on the CPU used only by the databases that you’re planning to migrate. (OS statistics also include CPU usage by other processes.) You should also check CPU-related recommendations in the ADDM report to improve performance after migration.You can also estimate requirements based on CPU generation. If you are using different CPU generations, you can estimate the required CPU of the target database by following the instructions in the whitepaper [Demystifying the Number of vCPUs for Optimal Workload Performance](https://d1.awsstatic.com/whitepapers/Demystifying_vCPUs.df200b766578b75009ad8d15c72e493d6408c68a.pdf). | DBA | | Option 1: Estimate requirements based on available cores. | In AWR reports:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)You can estimate available cores in two ways:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)**To estimate available cores by using OS commands**Use the following command to count the cores in the processor.
compute sum of Value on report
select METRIC_NAME,avg(AVERAGE) as "Value"
from dba_hist_sysmetric_summary
where METRIC_NAME in ('Physical Read Total IO Requests Per Sec','Physical Write Total IO Requests Per Sec')
group by metric_name;
$ cat /proc/cpuinfo |grep "cpu cores"|uniqUse the following command to count the sockets in the processor.
cpu cores : 4
cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l
grep "physical id" /proc/cpuinfo | sort -uWe don’t recommend using OS commands such as **nmon** and **sar** to extract CPU utilization. This is because those calculations include CPU utilization by other processes and might not reflect the actual CPU that is used by the database.**To estimate available cores by using the AWR report**You can also derive CPU utilization from the first section of the AWR report. Here’s an excerpt from the report.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)In this example, the CPUs count is 80, which indicates that these are logical (virtual) CPUs. You can also see that this configuration has two sockets, one physical processor on each socket (for a total of two physical processors), and 40 cores for each physical processor or socket. | DBA | | Option 2: Estimate CPU utilization by using OS statistics. | You can check the OS CPU usage statistics either directly in the OS (using **sar** or another host OS utility) or by reviewing the IDLE/(IDLE\$1BUSY) values from the Operating System Statistics section of the AWR report. You can see the seconds of CPU consumed directly from **v\$1osstat**. The AWR and Statspack reports also show this data in the Operating System Statistics section.If there are multiple databases on the same box, they all have the same **v\$1osstat** values for BUSY\$1TIME.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)If there are no other major CPU consumers in the system, use the following formula to calculate the percentage of CPU utilization:* Utilization = Busy time / Total time** Busy time = requirements = v\$1osstat.BUSY\$1TIME** C = Total time (Busy \$1 Idle)** C = capacity = v\$1ostat.BUSY\$1TIME \$1 v\$1ostat.IDLE\$1TIME** Utilization = BUSY\$1TIME / (BUSY\$1TIME \$1 IDLE\$1TIME)** = -1,305,569,937 / (1,305,569,937 \$1 4,312,718,839 )** = 23% utilized* | DBA | | Option 3: Estimate CPU utilization by using database metrics. | If multiple databases are running in the system, you can use the database metrics that appears at the beginning of the report.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.html)To get CPU utilization metrics, use this formula:* Database CPU usage (% of CPU power available) = CPU time / NUM\$1CPUS / elapsed time*where CPU usage is described by *CPU time* and represents the time spent on CPU, not the time waiting for CPU. This calculation results in:* = 312,625.40 / 11,759.64/80 = 33% of CPU is being used** Number of cores (33%) \$1 80 = 26.4 cores** Total cores = 26.4 \$1 (120%) = 31.68 cores*You can use the greater of these two values to calculate the CPU utilization of the Amazon RDS or Aurora DB instance.On IBM AIX, the calculated utilization doesn’t match the values from the operating system or the database. These values do match on other operating systems. | DBA | | DB Name | DB Id | Instance | Inst num | Startup Time | Release | RAC | | --- |--- |--- |--- |--- |--- |--- | | XXXX |
physical id : 0
physical id : 1
call mysql.rds_set_configuration('binlog retention hours', XX)Confirm the setting by running the following command.call mysql.rds_show_configuration;MySQL-compatible databases managed by AWS purge the binary logs as soon as possible. Therefore, the retention period must be long enough to ensure that the logs are not purged before the AWS DMS task runs. A value of 24 hours is usually sufficient, but the value should be based on the time required to set up the AWS DMS task in the DR Region. | DBA | ### Update the existing AWS DMS task in the primary Region | Task | Description | Skills required | | --- | --- | --- | | Record the AWS DMS task ARN. | Use the Amazon Resource Name (ARN) to obtain the AWS DMS task name for later use. To retrieve the AWS DMS task ARN, view the task in the console or run the following command.
aws dms describe-replication-tasksAn ARN looks like the following.
arn:aws:dms:us-east-1:The characters after the last colon correspond to the task name used in a later step. | AWS administrator | | Modify the existing AWS DMS task to record the checkpoint. | AWS DMS creates checkpoints that contain information so that the replication engine knows the recovery point for the change stream. To record checkpoint information, perform the following steps in the console:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-cross-region-disaster-recovery-with-aws-dms-and-amazon-aurora.html) | AWS administrator | | Validate checkpoint information. | Using a MySQL client connected to the writer endpoint for the cluster, query the new metadata table in the reporter database cluster to verify that it exists and contains the replication state information. Run the following command.:task:AN6HFFMPM246XOZVEUHCNSOVF7MQCLTOZUIRAMY
select * from awsdms_control.awsdms_txn_state;The task name from the ARN should be found in this table in the `Task_Name` column. | DBA | ### Expand both Amazon Aurora clusters to a DR Region | Task | Description | Skills required | | --- | --- | --- | | Create base infrastructure in the DR Region. | Create the base components required for the creation of and access to the Amazon Aurora clusters:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-cross-region-disaster-recovery-with-aws-dms-and-amazon-aurora.html)Ensure that the configuration of both parameter groups matches the configuration in the primary Region. | AWS administrator | | Add the DR Region to both Amazon Aurora clusters. | Add a secondary Region (the DR Region) to the main and reporter Amazon Aurora clusters. For more information, see [Adding an AWS Region to an Amazon Aurora global database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-getting-started.html#aurora-global-database-attaching). | AWS administrator | ### Perform failover | Task | Description | Skills required | | --- | --- | --- | | Stop the AWS DMS task. | The AWS DMS task in the primary Region will not function properly after failover occurs and should be stopped to avoid errors. | AWS administrator | | Perform a managed failover. | Perform a managed failover of the main database cluster to the DR Region. For instructions, see [Performing managed planned failovers for Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html#aurora-global-database-disaster-recovery.managed-failover). After failover on the main database cluster is complete, perform the same activity on the reporter database cluster. | AWS administrator, DBA | | Load data into the main database. | Insert test data into writer node of the main database in the DR database cluster. This data will be used to validate that replication is functioning properly. | DBA | | Create the AWS DMS replication instance. | To create the AWS DMS replication instance in the DR Region, see [Creating a replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Creating.html). | AWS administrator, DBA | | Create the AWS DMS source and target endpoints. | To create the AWS DMS source and target endpoints in the DR Region, see [Creating source and target endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.Creating.html). The source should point to the writer instance of the main database cluster. The target should point to the writer instance of the reporter database cluster. | AWS administrator, DBA | | Obtain the replication checkpoint. | To obtain the replication checkpoint, use a MySQL client to query the metadata table by running the following against the writer node in the reporter database cluster in the DR Region.
select * from awsdms_control.awsdms_txn_state;In the table, find the task\$1name value that corresponds to the AWS DMS task’s ARN that exists in the primary Region that you obtained in the second epic. | DBA | | Create an AWS DMS task. | Using the console, create an AWS DMS task in the DR Region. In the task, specify a migration method of **Replicate data changes only**. For more information, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html). [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-cross-region-disaster-recovery-with-aws-dms-and-amazon-aurora.html)Set the AWS DMS task **Start migration task** setting to **Automatically on create**. | AWS administrator, DBA | | Record the AWS DMS task ARN. | Use the ARN to obtain the AWS DMS task name for later use. To retrieve the AWS DMS task ARN, run the following command.
aws dms describe-replication-tasks| AWS administrator, DBA | | Validate the replicated data. | Query the reporter database cluster in the DR Region to confirm that the test data that you loaded into the main database cluster has been replicated. | DBA | ### Perform failback | Task | Description | Skills required | | --- | --- | --- | | Stop the AWS DMS task. | The AWS DMS task in the DR Region will not function properly after failback occurs and should be stopped to avoid errors. | AWS administrator | | Perform a managed failback. | Fail back the main database cluster to the primary Region. For instructions, see [Performing managed planned failovers for Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html#aurora-global-database-disaster-recovery.managed-failover). After the failback on the main database cluster is complete, perform the same activity on the reporter database cluster. | AWS administrator, DBA | | Obtain the replication checkpoint. | To obtain the replication checkpoint, use a MySQL client to query the metadata table by running the following against the writer node in the reporter database cluster in the DR Region.
select * from awsdms_control.awsdms_txn_state;In the table, find the `task_name` value that corresponds to the AWS DMS task's ARN that exists in the DR Region that you obtained in the fourth epic. | DBA | | Update the AWS DMS source and target endpoints. | After the database clusters have failed back, check the clusters in the primary Region to determine which nodes are the writer instances. Then verify the existing AWS DMS source and target endpoints in the primary Region are pointing to the writer instances. If not, update the endpoints with the writer instance Domain Name System (DNS) names. | AWS administrator | | Create an AWS DMS task. | Using the console, create an AWS DMS task in the primary Region. In the task, specify a migration method of **Replicate data changes only**. For more information, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html). [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/implement-cross-region-disaster-recovery-with-aws-dms-and-amazon-aurora.html) | AWS administrator, DBA | | Record the AWS DMS task Amazon Resource Name (ARN). | Use the ARN to obtain the AWS DMS task name for later use. To retrieve the AWS DMS task ARN, run the following command:
aws dms describe-replication-tasksThe task name will be needed when performing another managed failover or during a DR scenario. | AWS administrator, DBA | | Delete AWS DMS tasks. | Delete the original (currently stopped) AWS DMS task in the primary Region and the existing AWS DMS task (currently stopped) in the secondary Region. | AWS administrator | ## Related resources + [Configuring your Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraSettingUp.html) + [Using Amazon Aurora global databases](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) + [Working with Amazon Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraMySQL.html) + [Working with an AWS DMS replication instance](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.html) + [Working with AWS DMS endpoints](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Endpoints.html) + [Working with AWS DMS tasks](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.html) + [What is AWS CloudFormation?](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) ## Additional information Amazon Aurora global databases are used in this example for DR because they provide an effective recovery time objective (RTO) of 1 second and a recovery point objective (RPO) of less than 1 minute, both lower than traditional replicated solutions and ideal for DR scenarios. Amazon Aurora global databases offer many other advantages, including the following: + Global reads with local latency – Global consumers can access information in a local Region, with local latency. + Scalable secondary Amazon Aurora DB clusters – Secondary clusters can be scaled independently, adding up to 16 read-only replicas. + Fast replication from primary to secondary Amazon Aurora DB clusters – Replication has little performance impact on the primary cluster. It occurs at the storage layer, with typical cross-Region replication latencies of less than 1 second. This pattern also uses AWS DMS for replication. Amazon Aurora databases provide the ability to create read replicas, which can simplify the replication process and the DR setup. However, AWS DMS is often used to replicate when data transformations are required or when the target database requires additional indexes that the source database does not have. # Migrate Oracle functions and procedures that have more than 100 arguments to PostgreSQL *Srinivas Potlachervoo, Amazon Web Services* ## Summary This pattern shows how to migrate Oracle Database functions and procedures that have more than 100 arguments to PostgreSQL. For example, you can use this pattern to migrate Oracle functions and procedures to one of the following PostgreSQL-compatible AWS database services: + Amazon Relational Database Service (Amazon RDS) for PostgreSQL + Amazon Aurora PostgreSQL-Compatible Edition PostgreSQL doesn’t support functions or procedures that have more than 100 arguments. As a workaround, you can define a new data type that has type fields that match the source function’s arguments. Then, you can create and run a PL/pgSQL function that uses the custom data type as an argument. ## Prerequisites and limitations **Prerequisites ** + An active AWS account + An [Amazon RDS Oracle database (DB) instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) + An [Amazon RDS for PostgreSQL DB instance](https://aws.amazon.com/getting-started/hands-on/create-connect-postgresql-db/) or an [Aurora PostgreSQL-Compatible DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) **Product versions** + Amazon RDS Oracle DB instance versions 10.2 and later + Amazon RDS PostgreSQL DB instance versions 9.4 and later, or Aurora PostgreSQL-Compatible DB instance versions 9.4 and later + Oracle SQL Developer version 18 and later + pgAdmin version 4 and later ## Architecture **Source technology stack ** + Amazon RDS Oracle DB instance versions 10.2 and later **Target technology stack ** + Amazon RDS PostgreSQL DB instance versions 9.4 and later, or Aurora PostgreSQL-Compatible DB instance versions 9.4 and later ## Tools **AWS services** + [Amazon Relational Database Service (Amazon RDS) for PostgreSQL ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html)helps you set up, operate, and scale a PostgreSQL relational database in the AWS Cloud. + [Amazon Aurora PostgreSQL-Compatible Edition ](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.AuroraPostgreSQL.html)is a fully managed, ACID-compliant relational database engine that helps you set up, operate, and scale PostgreSQL deployments. **Other services** + [Oracle SQL Developer](https://www.oracle.com/database/technologies/appdev/sqldeveloper-landing.html) is an integrated development environment that simplifies the development and management of Oracle databases in both traditional and cloud-based deployments. + [pgAdmin](https://www.pgadmin.org/) is an open-source management tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects. ## Best practices Make sure that the data type that you create matches the type fields that are included in the source Oracle function or procedure. ## Epics ### Run an Oracle function or procedure that has more than 100 arguments | Task | Description | Skills required | | --- | --- | --- | | Create or identify an existing Oracle/PLSQL function or procedure that has more than 100 arguments. | Create an Oracle/PLSQL function or procedure that has more than 100 arguments.-or-Identify an existing Oracle/PLSQL function or procedure that has more than 100 arguments.For more information, see sections [14.7 CREATE FUNCTION Statement](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/lnpls/CREATE-FUNCTION-statement.html#GUID-B71BC5BD-B87C-4054-AAA5-213E856651F2) and [14.11 CREATE PROCEDURE Statement](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/lnpls/CREATE-PROCEDURE-statement.html#GUID-5F84DB47-B5BE-4292-848F-756BF365EC54) in the Oracle Database documentation. | Oracle/PLSQL knowledge | | Compile the Oracle/PLSQL function or procedure. | Compile the Oracle/PLSQL function or procedure.For more information, see [Compiling a function](https://docs.oracle.com/cd/E37097_01/doc.42/e35128/GUID-6B7B6F82-616D-4915-82BE-D4AE7F59CF37.htm#AEUTL165) in the Oracle Database documentation. | Oracle/PLSQL knowledge | | Run the Oracle/PLSQL function. | Run the Oracle/PLSQL function or procedure. Then, save the output. | Oracle/PLSQL knowledge | ### Define a new data type that matches the source function's or procedure's arguments | Task | Description | Skills required | | --- | --- | --- | | Define a new data type in PostgreSQL. | Define a new data type in PostgreSQL that includes all of the same fields that appear in the source Oracle function’s or procedure’s arguments.For more information, see [CREATE TYPE](https://www.postgresql.org/docs/current/sql-createtype.html) in the PostgreSQL documentation. | PostgreSQL PL/pgSQL knowledge | ### Create a PostgreSQL function that includes the new TYPE argument | Task | Description | Skills required | | --- | --- | --- | | Create a PostgreSQL function that includes the new data type. | Create a PostgreSQL function that includes the new `TYPE` argument.To review an example function, see the **Additional information** section of this pattern. | PostgreSQL PL/pgSQL knowledge | | Compile the PostgreSQL function. | Compile the function in PostgreSQL. If the new data type fields match the source function’s or procedure’s arguments, then the function successfully compiles. | PostgreSQL PL/pgSQL knowledge | | Run the PostgreSQL function. | Run the PostgreSQL function. | PostgreSQL PL/pgSQL knowledge | ## Troubleshooting | Issue | Solution | | --- | --- | | The function returns the following error:**ERROR: syntax error near "
GRANT CONNECT to| DBA | | Prepare the on-premises source database for AWS DMS CDC migration. | (Optional) Prepare the on-premises source Oracle database for AWS DMS CDC migration after completion of Oracle Data Pump Full Load:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link.html) | DBA | | Install and configure SQL Developer. | Install and configure [SQL Developer](https://www.oracle.com/database/sqldeveloper/) to connect and run SQL queries on the source and target databases. | DBA, Migration engineer | | Generate a script to create the tablespaces. | Use the following example SQL query to generate the script on the source database:;
GRANT DATAPUMP_EXP_FULL_DATABASE to;
GRANT SELECT ANY TABLE to;
SELECTThe script will be applied on the target database. | DBA | | Generate a script to create users, profiles, roles, and privileges. | To generate a script to create the database users, profiles, roles, and privileges, use the scripts from the Oracle Support document [How to Extract DDL for User including Privileges and Roles Using dbms\$1metadata.get\$1ddl (Doc ID 2739952.1)](https://support.oracle.com/epmos/faces/DocumentDisplay?id=2739952.1) (Oracle account required).The script will be applied on the target database. | DBA | ### Prepare the target Amazon RDS for Oracle instance | Task | Description | Skills required | | --- | --- | --- | | Create a database link to the source database and verify connectivity. | To create a database link to the on-premises source database, you can use the following example command:
'CREATE TABLESPACE ' tablespace_name ' DATAFILE SIZE 1G AUTOEXTEND ON MAXSIZE UNLIMITED;'
from dba_tablespaces
where tablespace_name not in ('SYSTEM', 'SYSAUX','TEMP','UNDOTBS1')
order by 1;
CREATE DATABASE LINK link2srcTo verify connectivity, run the following SQL command:
CONNECT TOIDENTIFIED BY
USING '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=)
(PORT=))(CONNECT_DATA=(SID= )))';
select * from dual@link2src;Connectivity is successful if the response is `X`. | DBA | | Run the scripts to prepare the target instance. | Run the previously generated scripts to prepare the target Amazon RDS for Oracle instance:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link.html)This helps ensure that the Oracle Data Pump migration can create the schemas and their objects. | DBA, Migration engineer | ### Perform a full-load migration by using Oracle Data Pump Import over a database link | Task | Description | Skills required | | --- | --- | --- | | Migrate the required schemas. | To migrate the required schemas from the source on-premises database to the target Amazon RDS instance, use the code in the [Additional information](#migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link-additional) section:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link.html)To tune the performance of the migration, you can adjust the number of parallel processes by running the following command:
DBMS_DATAPUMP.SET_PARALLEL (handle => v_hdnl, degree => 4);| DBA | | Gather schema statistics to improve performance. | The Gather Schema Statistics command returns the Oracle query optimizer statistics gathered for database objects. By using this information, the optimizer can select the best execution plan for any query against these objects:
EXECUTE DBMS_STATS.GATHER_SCHEMA_STATS(ownname => '| DBA | ### Perform a full-load migration and CDC replication by using Oracle Data Pump and AWS DMS | Task | Description | Skills required | | --- | --- | --- | | Capture the SCN on the source on-premises Oracle database. | Capture the [system change number (SCN)](https://docs.oracle.com/cd/E11882_01/server.112/e40540/transact.htm) on the source on-premises Oracle database. You will use the SCN for full-load import and as the starting point for CDC replication.To generate the current SCN on the source database, run the following SQL statement:');
SELECT current_scn FROM V$DATABASE;| DBA | | Perform the full-load migration of the schemas. | To migrate the required schemas (`FULL LOAD`) from the source on-premises database to the target Amazon RDS instance, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link.html)In the code, replace `
DBMS_DATAPUMP.SET_PARAMETER (handle => v_hdnl, name => 'FLASHBACK_SCN', value =>To tune the performance of the migration, you can adjust the number of parallel processes:);
DBMS_DATAPUMP.SET_PARALLEL (handle => v_hdnl, degree => 4);| DBA | | Disable the triggers under the migrated schemas. | Before you begin the AWS DMS CDC-only task, disable the `TRIGGERS` under the migrated schemas. | DBA | | Gather schema statistics to improve performance. | The Gather Schema Statistics command returns the Oracle query optimizer statistics gathered for database objects:
EXECUTE DBMS_STATS.GATHER_SCHEMA_STATS(ownname => 'By using this information, the optimizer can select the best execution plan for any query against these objects. | DBA | | Use AWS DMS to perform an ongoing replication from the source to target. | Use AWS DMS to perform an ongoing replication from the source Oracle database to the target Amazon RDS for Oracle instance.For more information, see [Creating tasks for ongoing replication using AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html) and the blog post [How to work with native CDC support in AWS DMS](https://aws.amazon.com/blogs/database/aws-dms-now-supports-native-cdc-support/). | DBA, Migration engineer | ### Cut over to Amazon RDS for Oracle | Task | Description | Skills required | | --- | --- | --- | | Enable Multi-AZ on the instance 48 hours before cutover. | If this is a production instance, we recommend enabling [Multi-AZ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) deployment on the Amazon RDS instance to provide the benefits of high availability (HA) and disaster recovery (DR). | DBA, Migration engineer | | Stop the AWS DMS CDC-only task (if CDC was turned on). | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link.html) | DBA | | Enable the triggers. | Enable the `TRIGGERS` that you disabled before the CDC task was created. | DBA | ## Related resources **AWS** + [Preparing an Oracle self-managed source database for CDC using AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Self-Managed.Configuration) + [Creating tasks for ongoing replication using AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html) + [Multi-AZ deployments for high availability](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html) + [How to work with native CDC support in AWS DMS](https://aws.amazon.com/blogs/database/aws-dms-now-supports-native-cdc-support/) (blog post) **Oracle documentation** + [DBMS\$1DATAPUMP](https://docs.oracle.com/database/121/ARPLS/d_datpmp.htm) ## Additional information *Code 1: Full-load migration only, single application schema* ``` DECLARE v_hdnl NUMBER; BEGIN v_hdnl := DBMS_DATAPUMP.OPEN(operation => 'IMPORT', job_mode => 'SCHEMA', remote_link => '');
hdbnsutil -sr_enable --name=siteA| SAP Basis administration | | Register the target SAP HANA DB as a secondary system. | To register the target SAP HANA DB as a secondary system to source for SAP HSR, choose **async** replication.
(sid)adm $> HDB stopAlternatively, you can choose the `–online` option to register. In that case, you don’t need to stop and start the SAP HANA DB. | SAP Basis administration | | Validate synchronization. | On the source SAP HANA DB, verify that all the logs are applied on the target system (because it is async replication).To verify the replication, on the source, run the following commands.
(sid)adm $> hdbnsutil -sr_register –name=siteB –remotehost=hdbhost /
--remoteInstance=00 –replicationMode=async –operationMode=logreplay
(sid)adm $> HDB start
(sid)adm $> cdpy| SAP Basis administration | | Shut down the source SAP application and SAP HANA DB. | During the migration cutover, perform a shutdown of the source system (the SAP application and SAP HANA database. | SAP Basis administration | | Perform a takeover at the target. | To perform a takeover at the target on AWS, run the command `hdbnsutil -sr_takeover`. | SAP Basis administration | | On the target SAP HANA DB, turn off replication. | To clear the replication metadata, stop replication on the target system by running the command `hdbnsutil -sr_disable`. This is in accordance with SAP Note 2693441 – Failed to rename an SAP HANA System due to error. | SAP Basis administration | | Back up the target SAP HANA DB. | After the takeover is successful, we recommend performing a full SAP HANA DB backup. | SAP Basis administration | ### Revert to the original hostname in the target system | Task | Description | Skills required | | --- | --- | --- | | Revert the target SAP HANA DB hostname to the original. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-sap-hana-to-aws-using-sap-hsr-with-the-same-hostname.html)You can validate other options as needed. However, be sure that you don’t mix up the host rename with a SID change (SAP Note 2598814 – hdblcm: SID rename fails). | SAP Basis administration | | Adjust hdbuserstore. | Adapt the `hdbuserstore` details pointing to the source `schema/user` details. For detailed steps, see the [SAP documentation](https://help.sap.com/viewer/b3ee5778bc2e4a089d3299b82ec762a7/2.0.02/en-US/ddbdd66b632d4fe7b3c2e0e6e341e222.html?q=hdbuserstore). To validate this step, run the command `R3trans -d`. The result should reflect a successful connection to the SAP HANA database. | SAP Basis administration | | Start up client connections. | In the target environment, start up the SAP application servers and other client connections. | SAP Basis administration | ## Related resources **SAP references** SAP documentation references are frequently updated by SAP. To stay up to date, see SAP Note 2407186 – How-To Guides & Whitepapers For SAP HANA High Availability. *Additional SAP notes* + 2550327 – How-To Rename an SAP HANA System + 1999880 – FAQ: SAP HANA System Replication + 2078425 – Troubleshooting note for SAP HANA platform lifecycle management tool hdblcm + 2592227 – FQDN suffix change in HANA systems + 2048681 – Performing SAP HANA platform lifecycle management administration tasks on multiple-host systems without SSH or root credentials *SAP documents* + [System Replication Network Connection](https://help.sap.com/docs/SAP_HANA_PLATFORM/4e9b18c116aa42fc84c7dbfd02111aba/47190b425eb1433697b026ecd46ff5f9.html) + [Host Name Resolution for System Replication](https://help.sap.com/viewer/eb3777d5495d46c5b2fa773206bbfb46/1.0.12/en-US/c0cba1cb2ba34ec89f45b48b2157ec7b.html) **AWS references** + [Migrating SAP HANA from Other Platforms to AWS](https://docs.aws.amazon.com/sap/latest/sap-hana/migrating-hana-hana-to-aws.html) ## Additional information The changes performed by `hdblcm` as part of the hostname rename activity are consolidated in the following verbose log. ![\[Code showing processes stopped on temp-host, starting on hdbhost, and SAP HANA DB system renamed.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/004781c1-96df-43dd-a52e-ed1db5bdf9ef/images/9e0c11ca-6555-484f-9639-107f60f725f5.png) # Migrate Microsoft SQL Server Always On availability group using AWS Application Migration Service *Sreenivas Nettem, Bharath Kumar Pammi Ramesh, Anantharaman Seshadri, and Gireesh Sreekantan, Amazon Web Services* ## Summary AWS Application Migration Service (AWS MGN) is a preferred tool for rehosting existing environments in the AWS Cloud, which allows customers to move away from on-premises data centers. This pattern outlines the process of using AWS MGN to migrate Windows clusters with Microsoft SQL Server Always On availability groups. ## Prerequisites and limitations **Prerequisites** + An active AWS account. + An AWS Identity and Access Management (IAM) role for AWS MGN orchestration. + Access to source database servers (SQL Server Always On availability group). + Active Directory in an AWS landing zone to retain DNS names. + A staging subnet with closed network communication to Active Directory. + Target subnets that can communicate with Active Directory. + Two reserved IP addresses for the Windows cluster in a target subnet (one in each Availability Zone). + Two reserved IP addresses for the SQL Always On listener in a target subnet (one in each Availability Zone). **Product versions** + Windows Server 2012 or later + SQL Server 2012 or later ## Architecture **Source technology stack** Microsoft Windows cluster (on-premises physical or virtual machine) Microsoft SQL Server Always On availability group **Target technology stack** Amazon EC2 Windows instance **Target architecture** ![\[AWS architecture for migrating SQL Server Always On availability using AWS MGN.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/aa94040b-5ecf-42f9-90e3-929d0fa5e715/images/0b85c613-51df-475b-9598-3da3f9cd47c6.png) ## Tools *AWS services* + [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down. + [AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) helps you rehost (lift and shift) applications to the AWS Cloud without change and with minimal downtime. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. *Other tools* + [Microsoft SQL Server Management Studio (SSMS)](https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms) is a tool for managing SQL Server, including accessing, configuring, and administering SQL Server components. ## Best practices For AWS MGN, see [Best practices for AWS Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/best_practices_mgn.html). ## Epics ### Prepare the target account | Task | Description | Skills required | | --- | --- | --- | | Initialize AWS MGN. | Initialize AWS MGN in the target AWS Region. This creates the required IAM roles and policies. For more information, see [Initializing Application Migration Service with the console](https://docs.aws.amazon.com/mgn/latest/ug/mgn-initialize-console.html). | Cloud administrator | | Create replication and launch templates. | Configure the replication and launch templates for use with AWS MGN. For more information, see [Configuring the templates](https://docs.aws.amazon.com/mgn/latest/ug/mgn-initialization-templates.html) in the AWS documentation. | Cloud administrator | | Allow communication ports. | To enable network communications for AWS MGN, allow traffic over TCP ports 443 and 1500. For more information, see [Network requirements for Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/Network-Requirements.html) in the AWS documentation. | Cloud administrator, Network administrator | ### Prepare the source server | Task | Description | Skills required | | --- | --- | --- | | Verify AWS MGN prerequisites. | Verify that the source servers meet the prerequisites for the AWS MGN agent installation. For more information, see [Installation requirements](https://docs.aws.amazon.com/mgn/latest/ug/installation-requirements.html) in the AWS documentation. | Migration engineer | | Install AWS MGN agent. | Install the AWS MGN agent on the source servers. During the installation, choose the AWS Region where the servers will be migrated. After installation, the agent communicates with the service and starts replication. For more information, see [Installing the AWS Replication Agent on Windows servers](https://docs.aws.amazon.com/mgn/latest/ug/windows-agent.html). | Migration engineer | | Check source server status. | In the AWS MGN console, check the status of source servers. The servers display **Ready for testing** when replication begins.If you encounter any errors, see [Troubleshooting communication errors](https://docs.aws.amazon.com/mgn/latest/ug/Troubleshooting-Communication-Errors.html) in the AWS MGN documentation. | Cloud administrator, Migration engineer | | Optimize replication settings. | SQL Always On clusters use high I/O synchronous replication from the primary server to the secondary server. To optimize replication and avoid lag, use a [dedicated replication server](https://docs.aws.amazon.com/mgn/latest/ug/replication-settings-template.html) for each SQL Always On server.If the database is greater than 5 TB, consider choosing a larger replication server instance size such as **m5.large** instead of the default **t3.small**. | Cloud administrator, Migration engineer | | Update launch template. | Update the [launch settings](https://docs.aws.amazon.com/mgn/latest/ug/launch-settings.html) and choose subnets for the SQL Always On servers. The SQL Always On cluster servers are spread across different AWS Availability Zones for high availability. | Migration engineer, Migration lead | | Update launch settings. | Based on your size and performance requirements, update the instance type and input/output operations per second (IOPS) in the launch settings.(Optional) Choose an existing elastic network interface in the launch settings. | Migration engineer, Migration lead | ### Test cutover | Task | Description | Skills required | | --- | --- | --- | | Verify source servers. | In the AWS MGN console, verify that the source server status is **Ready for testing**. | Cloud administrator, Migration engineer | | Launch the test instances. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-always-on-group-using-mgn.html) | Cloud administrator, Migration engineer | | Test for connectivity and database integrity. | Test the connectivity and database integrity of the test instances. Then mark the source servers as **Ready for cutover** in the AWS MGN console. | Cloud administrator, Migration engineer | ### Pre-migration tasks | Task | Description | Skills required | | --- | --- | --- | | Test the database integrity. | This helps to ensure there are no database integrity issues at the source prior to migration. Run `DBCC CHECKDB` and specify `WITH_PHYSICAL_ONLY`. Running this check without `WITH_PHYSICAL_ONLY` can cause performance issues at the source. To maintain database integrity, run a weekly full check of the database.These commands check the database’s logical and physical integrity by detecting potential corruption issues. The check verifies the database’s structure, such as pages, rows, indexes, and system tables. | Data engineer, DBA | | Test the connections to linked servers. | Test the connections among all existing servers, and document their status. This helps to ensure that linked servers work as intended after the migration. | Data engineer, DBA | | Verify the backups. | Confirm the integrity of source backups. | Data engineer, DBA | ### AWS MGN cutover | Task | Description | Skills required | | --- | --- | --- | | Stop the SQL Server and cluster services. | Stop the SQL Server and Microsoft cluster services on all SQL cluster nodes. | DBA, Migration engineer | | Verify the servers. | In the AWS MGN console, verify that the status of the source servers is **Ready for cutover** and that the data replication status is **Healthy**. | Migration engineer | | Launch the cutover. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-always-on-group-using-mgn.html)For more information, see [Launching a cutover instance](https://docs.aws.amazon.com/mgn/latest/ug/launch-cutover-gs.html) in the AWS MGN documentation. | Migration engineer | | Test the launched servers. | Log in to the launched Amazon EC2 instances and validate the cluster’s health. Verify that the servers are in the correct subnets, the instance size and IOPS settings are correct, and the witness server is accessible. | DBA, Migration engineer | ### Database post-cutover tasks | Task | Description | Skills required | | --- | --- | --- | | Update the cluster IP address. | Update the cluster IP address for the Windows cluster using the two reserved IP addresses in the target subnets. For more information, see [Change the IP Address of a Failover Cluster Instance](https://learn.microsoft.com/en-us/sql/sql-server/failover-clusters/windows/change-the-ip-address-of-a-failover-cluster-instance?view=sql-server-2016). | DBA, Migration engineer | | Update the Always On availability group listener IPs. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-always-on-group-using-mgn.html) | DBA, Migration engineer | | Verify the connection. | Using SSMS, connect to the Always On availability group listener, and confirm that the connection succeeds. | DBA, Migration engineer | | Check the health of Always On availability group. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-microsoft-sql-server-always-on-group-using-mgn.html) | DBA, Migration engineer | | Check the error log. | Open the error log, and verify any errors reported for the SQL Server instance. Ensure that the recovery is complete for all databases. | DBA, Migration engineer | | Test the linked servers. | Test the connectivity of any linked servers. In case of any connectivity issues, ensure that the target server and port are accessible. | DBA, Migration engineer | ### Finalize the cutover | Task | Description | Skills required | | --- | --- | --- | | Finalize the cutover. | After you validate the target SQL Always On clusters, [finalize](https://docs.aws.amazon.com/mgn/latest/ug/launch-cutover-gs.html#revert-finalize-cutover-gs) the cutover through the AWS MGN console. This stops data replication from the source servers and discards the data from the replication server. It also removes the replication server and its associated resources. | Cloud administrator, Migration engineer | ## Troubleshooting | Issue | Solution | | --- | --- | | AWS MGN troubleshooting | For common issues and resolutions, see the [Troubleshooting](https://docs.aws.amazon.com/mgn/latest/ug/troubleshooting.html) and [FAQ](https://docs.aws.amazon.com/mgn/latest/ug/FAQ.html) sections in the AWS MGN documentation. | ## Related resources *AWS resources* + [Option-1 Rehost - AWS Application Migration Service (AWS)](https://catalog.us-east-1.prod.workshops.aws/workshops/c6bdf8dc-d2b2-4dbd-b673-90836e954745/en-US/04-application-migration/01-mgn) + [What Is AWS Application Migration Service?](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html) *SQL Server resources* + [What is SQL Server Management Studio (SSMS)?](https://learn.microsoft.com/en-us/ssms/sql-server-management-studio-ssms) ## Additional information For standard security requirements for migrating workloads to the AWS Cloud, see the [Best Practices for Security, Identity, and Compliance](https://aws.amazon.com/architecture/security-identity-compliance/) on the AWS website. # Migrate SQL Server to AWS using distributed availability groups *Praveen Marthala, Amazon Web Services* ## Summary Microsoft SQL Server Always On availability groups provide a high availability (HA) and disaster recovery (DR) solution for SQL Server. An availability group consists of a primary replica that accepts read/write traffic, and up to eight secondary replicas that accept read traffic. An availability group is configured on a Windows Server Failover Cluster (WSFC) with two or more nodes. Microsoft SQL Server Always On distributed availability groups provide a solution to configure two separate availability groups between two independent WFSCs. The availability groups that are part of the distributed availability group don’t have to be in the same data center. One availability group can be on premises, and the other availability group can be on the Amazon Web Services (AWS) Cloud on Amazon Elastic Compute Cloud (Amazon EC2) instances in a different domain. This pattern outlines steps for using a distributed availability group to migrate on-premises SQL Server databases that are part of an existing availability group to SQL Server with availability groups set up on Amazon EC2. By following this pattern, you can migrate the databases to the AWS Cloud with minimal downtime during cutover. The databases are highly available on AWS immediately after the cutover. You can also use this pattern to change the underlying operating system from on-premises to AWS while keeping the same version of SQL Server. ## Prerequisites and limitations **Prerequisites** + An active AWS account + AWS Direct Connect or AWS Site-to-Site VPN + The same version of SQL Server installed on-premises and on the two nodes on AWS **Product versions** + SQL Server version 2016 and later + SQL Server Enterprise Edition ## Architecture **Source technology stack** + Microsoft SQL Server database with Always On availability groups on premises **Target technology stack** + Microsoft SQL Server database with Always On availability groups on Amazon EC2 on the AWS Cloud **Migration architecture ** ![\[SQL Server with synchronous replication in availability groups on premises and on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6e229e30-9b11-4ccb-bccd-cbe6601139c0/images/79ee7911-d68f-4db7-9b94-113dcf09c28b.png) *Terminology* + WSFC 1 – WSFC on premises + WSFC 2 – WSFC on the AWS Cloud + AG 1 – First availability group, which is in WSFC 1 + AG 2 – Second availability group, which is in WSFC 2 + SQL Server primary replica – Node in AG 1 that is considered the global primary for all writes + SQL Server forwarder – Node in AG 2 that receives data asynchronously from the SQL Server primary replica + SQL Server secondary replica – Nodes in AG 1 or AG 2 that receive data synchronously from the primary replica or the forwarder ## Tools + [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) – AWS Direct Connect links your internal network to an AWS Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create *virtual interfaces* directly to public AWS services, bypassing internet service providers in your network path. + [Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/concepts.html) – Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the AWS Cloud. You can use Amazon EC2 to launch as many or as few virtual servers as you need, and you can scale out or scale in. + [AWS Site-to-Site VPN](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html) – AWS Site-to-Site VPN supports creating a site-to-site virtual private network (VPN). You can configure the VPN to pass traffic between instances that you launch on AWS and your own remote network. + [Microsoft SQL Server Management Studio](https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-ssms?view=sql-server-ver15) – Microsoft SQL Server Management Studio (SSMS) is an integrated environment for managing SQL Server infrastructure. It provides a user interface and a group of tools with rich script editors that interact with SQL Server. ## Epics ### Set up a second availability group on AWS | Task | Description | Skills required | | --- | --- | --- | | Create a WSFC on AWS. | Create WSFC 2 on Amazon EC2 instances with two nodes for HA. You will use this failover cluster to create the second availability group (AG 2) on AWS. | Systems administrator, SysOps administrator | | Create the second availability group on WSFC 2. | Using SSMS, create AG 2 on two nodes in WSFC 2. The first node in WSFC 2 will act as the forwarder. The second node in WSFC 2 will act as the secondary replica of AG 2.At this stage, no databases are available in AG 2. This is the starting point for setting up the distributed availability group. | DBA, Developer | | Create databases with no recovery option on AG 2. | Back up databases on the on-premises availability group (AG 1). Restore the databases to both the forwarder and the secondary replica of AG 2 with no recovery option. While restoring the databases, specify a location with enough disk space for the database data files and the log files.At this stage, the databases are in the restoring state. They are not part of AG 2 or the distributed availability group, and they are not synchronizing. | DBA, Developer | ### Configure the distributed availability group | Task | Description | Skills required | | --- | --- | --- | | Create the distributed availability group on AG 1. | To create the distributed availability group on AG 1, use the `CREATE AVAILABILITY GROUP` with the `DISTRIBUTED` option.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-sql-server-to-aws-using-distributed-availability-groups.html) | DBA, Developer | | Create the distributed availability group on AG 2. | To create the distributed availability group on AG 2, use `ALTER AVAILABILITY GROUP` with the `DISTRIBUTED` option.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-sql-server-to-aws-using-distributed-availability-groups.html)The distributed availability group is created between AG 1 and AG 2.The databases in AG 2 are not yet configured to take part in the data flow from AG 1 to AG 2. | DBA, Developer | | Add databases to the forwarder and secondary replica on AG 2. | Add the databases to the distributed availability group by using `ALTER DATABASE` with the `SET HADR` `AVAILABILITY GROUP` option in both the forwarder and the secondary replica on AG 2. This starts asynchronous data flow between databases on AG 1 and AG 2. The global primary takes writes, sends data synchronously to the secondary replica on AG 1, and sends data asynchronously to the forwarder on AG 2. The forwarder on AG 2 sends data synchronously to the secondary replica on AG 2. | DBA, Developer | ### Monitor asynchronous data flow between AG 1 and AG 2 | Task | Description | Skills required | | --- | --- | --- | | Use DMVs and SQL Server logs. | Monitor the status of the data flow between two availability groups by using dynamic management views (DMVs) and SQL Server logs. DMVs that are of interest for monitoring include `sys.dm_hadr_availability_replica_states` and `sys.dm_hadr_automatic_seeding`.For the status of forwarder synchronization, monitor the *synchronized state* in the SQL Server log on the forwarder. | DBA, Developer | ### Perform cutover activities for final migration | Task | Description | Skills required | | --- | --- | --- | | Stop all traffic to the primary replica. | Stop incoming traffic to the primary replica in AG 1 so that no write activity occurs on the databases and the databases are ready for migration. | App owner, Developer | | Change the availability mode of the distributed availability group on AG 1. | On the primary replica, set the availability mode of the distributed availability group to synchronous. After you change the availability mode to synchronous, the data are sent synchronously from the primary replica in AG 1 to the forwarder in AG 2. | DBA, Developer | | Check the LSNs in both availability groups. | Check the last Log Sequence Numbers (LSNs) in both AG 1 and AG 2. Because no writes are happening in the primary replica in AG 1, the data are synchronized, and last LSNs for both availability groups should match. | DBA, Developer | | Update AG 1 to the secondary role. | When you update AG 1 to the secondary role, AG 1 loses the primary replica role and doesn't accept writes, and the data flow between two availability groups stops. | DBA, Developer | ### Fail over to the second availability group | Task | Description | Skills required | | --- | --- | --- | | Manually fail over to AG 2. | On the forwarder in AG 2, alter the distributed availability group to allow data loss. Because you already checked and confirmed that the last LSNs on AG 1 and AG 2 match, data loss is not a concern.When you allow data loss on the forwarder in AG 2, the roles of AG 1 and AG 2 change:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-sql-server-to-aws-using-distributed-availability-groups.html) | DBA, Developer | | Change the availability mode of the distributed availability group on AG 2. | On the primary replica in AG 2, change the availability mode to asynchronous.This changes the data movement from AG 2 to AG 1, from synchronous to asynchronous. This step is required to avoid network latency between AG 2 and AG 1, if any, and will not impact the performance of the database. | DBA, Developer | | Start sending traffic to the new primary replica. | Update the connection string to use the listener URL endpoint on AG 2 for sending traffic to the databases.AG 2 now accepts writes and sends data to the forwarder in AG 1, along with sending data to its own secondary replica in AG 2. Data moves asynchronously from AG 2 to AG 1. | App owner, Developer | ### Perform post-cutover activities | Task | Description | Skills required | | --- | --- | --- | | Drop the distributed availability group on AG 2. | Monitor the migration for the planned amount of time. Then drop the distributed availability group on AG 2 to remove distributed availability group setup between AG 2 and AG 1. This removes the distributed availability group configuration, and data flow from AG 2 to AG 1 stops. At this point, AG 2 is highly available on AWS, with a primary replica that takes writes and a secondary replica in the same availability group. | DBA, Developer | | Decommission the on-premises servers. | Decommission the on-premises servers in WSFC 1 that are part of AG 1. | Systems administrator, SysOps administrator | ## Related resources + [Distributed availability groups](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-sql-server/distributed-groups.html) + [SQL Docs: Distributed availability groups](https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/distributed-availability-groups?view=sql-server-ver15) + [SQL Docs: Always On availability groups: a high-availability and disaster-recovery solution](https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/always-on-availability-groups-sql-server?view=sql-server-ver15) # Migrate a relational database to MongoDB Atlas on AWS *Battulga Purevragchaa and Igor Alekseev, Amazon Web Services* *Babu Srinivasan, MongoDB* ## Summary This pattern describes the steps for migrating from a relational database such as SQL Server, MySQL, or PostgreSQL to MongoDB Atlas in the AWS Cloud. It uses [MongoDB Relational Migrator](https://www.mongodb.com/products/relational-migrator) to help accelerate the data migration from relational databases to MongoDB Atlas. The pattern accompanies the guide [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/) on the AWS Prescriptive Guidance website. It provides the implementation steps for one of the migration scenarios that are discussed in that guide. For additional migration scenarios, see the following patterns on the AWS Prescriptive Guidance website: + [Migrate a self-hosted MongoDB environment to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-self-hosted-mongodb-environment-to-mongodb-atlas-on-the-aws-cloud.html) + [Stream data from IBM Db2, SAP, Sybase, and other databases to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/stream-data-from-ibm-db2-to-mongodb-atlas.html) The pattern is intended for [AWS System Integrator (SI) Partners](https://aws.amazon.com/managed-services/partners/) and AWS users. ## Prerequisites and limitations **Prerequisites** + A source relational database (Oracle Database, SQL Server, PostgreSQL, MySQL, SAP/Sybase ASE, and others) to migrate to MongoDB Atlas. + Familiarity with relational databases, MongoDB Atlas, and AWS services. This pattern explains some of the migration steps at a high level. Additional details will be added in future versions. **Product versions** + MongoDB version 5.0 or later ## Architecture The following diagram shows the migration from a relational database management system (RDBMS) database to MongoDB Atlas on AWS. ![\[Architecture for migrating from RDBMS to MongoDB Atlas on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4e3ea0f1-21e8-4641-a9ee-732355f20baf/images/8eacf3ec-f480-4912-9002-6a50800fe9bf.png) For MongoDB Atlas reference architectures that support different usage scenarios, see[ Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/architecture.html) on the AWS Prescriptive Guidance website. ## Tools + [MongoDB Atlas](https://www.mongodb.com/atlas) is a fully managed database as a service (DBaaS) for deploying and managing MongoDB databases in the cloud. + [MongoDB Relational Migrator](https://www.mongodb.com/products/relational-migrator) provides a smooth transition of data from traditional relational databases to MongoDB. It helps automate the conversion process and transforms the structured data model of relational databases into the flexible document format provided by MongoDB. The Relational Migrator preserves data integrity and relationships to simplify the migration. Organizations can take advantage of the scalability, performance, and versatility benefits that MongoDB provides while retaining the familiarity of their existing data. ## Best practices For best practices for using MongoDB on AWS, see the posts on the [AWS Partner Network Blog](https://aws.amazon.com/blogs/apn/tag/mongodb-atlas/). ## Epics ### Discovery and assessment | Task | Description | Skills required | | --- | --- | --- | | Determine the parameters and size of the relational database. | Estimate the working set size by using the Relational Migrator recommendations and information from `db.stats()` for the total index space. Assume that a percentage of your data space will be accessed frequently. This task should take approximately one week. For more information and examples for this and the other stories in this epic, see the [Related resources](#migrate-relational-database-to-mongodb-atlas-resources) section. | App owner, DBA | | Estimate network bandwidth requirements. | To estimate your network bandwidth requirements, multiply the average document size by the number of documents served per second. Consider the maximum traffic that any node on your cluster will bear as the basis. To calculate downstream data transfer rates from your cluster to client applications, use the sum of the total documents returned over a period of time. If your applications read from secondary nodes, divide this number of total documents by the number of nodes that can serve read operations. To find the average document size for a database, use the `db.stats().avgObjSize` command. This task will typically take one day. | DBA | | Select the Atlas tier. | Follow the instructions in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/sizing-tier-selection/) to select the correct Atlas cluster tier. | DBA | | Plan for cutover. | Plan for application cutover. | DBA, App owner | ### Set up a new MongoDB Atlas environment on AWS | Task | Description | Skills required | | --- | --- | --- | | Create a new MongoDB Atlas cluster on AWS. | In MongoDB Atlas, choose **Build a Cluster**. In the **Create New Cluster** dialog box, select AWS as the cloud provider. | DBA | | Select AWS Regions and global cluster configuration. | Select from the list of available AWS Regions for your Atlas cluster. Configure global clusters if required. | DBA | | Select the cluster tier. | Select your preferred cluster tier. Your tier selection determines factors such as memory, storage, and IOPS specification. | DBA | | Configure additional cluster settings. | Configure additional cluster settings such as MongoDB version, backup, and encryption options. For more information about these options, see the [Related resources](#migrate-relational-database-to-mongodb-atlas-resources) section. | DBA | ### Configure security and compliance | Task | Description | Skills required | | --- | --- | --- | | Configure the access list. | To connect to the Atlas cluster, you must add an entry to the project’s access list. Atlas uses TLS/SSL to encrypt the connections to the virtual private cloud (VPC) for your database. To set up the access list for the project and for more information about the stories in this epic, see the [Related resources](#migrate-relational-database-to-mongodb-atlas-resources) section. | DBA | | Authenticate and authorize users. | You must create and authenticate the database users who will access the MongoDB Atlas clusters. To access clusters in a project, users must belong to that project, and they can belong to multiple projects. | DBA | | Create custom roles. | (Optional) Atlas supports creating custom roles in cases where the built-in Atlas database user permissions don’t cover the set of permissions you want. | DBA | | Set up VPC peering. | (Optional) Atlas supports [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) with other VPCs on AWS. | AWS administrator | | Set up an AWS PrivateLink endpoint. | (Optional) You can set up private endpoints on AWS by using AWS PrivateLink. For more information, see the [Amazon VPC documentation](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-services-overview.html). | AWS administrator | | Enable two-factor authentication. | (Optional) Atlas supports two-factor authentication (2FA) to help users control access to their Atlas accounts. | AWS administrator | | Set up user authentication and authorization with LDAP. | (Optional) Atlas supports performing user authentication and authorization with Lightweight Directory Access Protocol (LDAP). | DBA | | Set up unified AWS access. | (Optional) Some Atlas features, including Atlas Data Lake and encryption at rest using customer key management, use AWS Identity and Access Management (IAM) roles for authentication. | AWS administrator | | Set up encryption at rest using AWS KMS. | (Optional) Atlas supports using AWS Key Management Service (AWS KMS) to encrypt storage engines and cloud provider backups. | AWS administrator | | Set up client-side field-level encryption. | (Optional) Atlas supports client-side field level encryption, including automatic encryption of fields. | AWS administrator | ### Migrate data | Task | Description | Skills required | | --- | --- | --- | | Add MongoDB Relational Migrator to the access list. | Add the Relational Migrator to the access list for your source database. This helps prepare the source environment to connect to the target Atlas cluster. | DBA | | Assess the relational database objects. | Launch MongoDB Relational Migrator and connect to your relational database. Start the assessment. | DBA | | Accept the migration patterns or choose to alter them based on your business needs. | Either accept the database patterns recommended by the Relational Migrator based on the initial assessment and performance parameters, or choose to alter them based on your business requirements. | DBA | | Launch your target replica set in MongoDB Atlas. | Launch your target replica set in MongoDB Atlas. In Relational Migrator, choose **I'm ready to migrate**. | DBA | ### Configure operational integration | Task | Description | Skills required | | --- | --- | --- | | Connect to the MongoDB Atlas cluster. | Make sure that MongoDB Atlas cluster connectivity works as expected. | App owner | | Interact with cluster data. | Verify the cluster data. | DBA | | Monitor your clusters. | Verify that your clusters are set up correctly. | DBA | | Back up and restore cluster data. | Schedule backups with a regular cadence for cluster data. | DBA | ## Related resources All of the following links, unless noted otherwise, go to webpages in the MongoDB documentation. **Migration guide** + [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/) (AWS Prescriptive Guidance) **Discovery and assessment** + [Memory](https://docs.atlas.mongodb.com/sizing-tier-selection/#memory) + [Sizing example with Atlas sample data sets](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#example--the-service-sample-data-sets) + [Sizing example for mobile applications](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#example--mobile-app) + [Network Traffic](https://docs.atlas.mongodb.com/sizing-tier-selection/#network-traffic) + [Cluster Auto-Scaling](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#cluster-auto-scaling) + [Atlas sizing template](https://view.highspot.com/viewer/5f438f47a4dfa042e97130c5) **Configuring security and compliance** + [Configure IP Access List Entries](https://docs.atlas.mongodb.com/security/ip-access-list/) + [Configure Database Users](https://docs.atlas.mongodb.com/security-add-mongodb-users/) + [Configure Access to the Atlas UI](https://docs.atlas.mongodb.com/organizations-projects/) + [Configure Custom Database Roles](https://docs.atlas.mongodb.com/security-add-mongodb-roles) + [Configure Database Users](https://docs.atlas.mongodb.com/security-add-mongodb-users/#atlas-user-privileges) + [Set up a Network Peering Connection](https://docs.atlas.mongodb.com/security-vpc-peering/) + [Learn About Private Endpoints in Atlas](https://docs.atlas.mongodb.com/security-private-endpoint/) + [Manage Your Multi-Factor Authentication Options](https://docs.atlas.mongodb.com/security-two-factor-authentication/) + [Set up User Authentication and Authorization with LDAP](https://docs.atlas.mongodb.com/security-ldaps/) + [Atlas Data Lake](https://docs.mongodb.com/datalake/) + [Encryption at Rest using Customer Key Management](https://docs.atlas.mongodb.com/security-kms-encryption/) + [Methods to assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) (IAM documentation) + [Client-Side Field Level Encryption](https://docs.mongodb.com/manual/core/security-client-side-encryption) + [Automatic Encryption](https://docs.mongodb.com/manual/core/security-automatic-client-side-encryption) + [MongoDB Atlas Security Controls](https://webassets.mongodb.com/_com_assets/cms/MongoDB_Atlas_Security_Controls-v7k3rbhi3p.pdf) + [MongoDB Trust Center](https://www.mongodb.com/cloud/trust) + [Configure Security Features for Clusters](https://docs.atlas.mongodb.com/setup-cluster-security/) **Setting up a new MongoDB Atlas environment on ****AWS** + [Cloud Providers and Regions](https://docs.atlas.mongodb.com/cloud-providers-regions/) + [Manage Global Clusters](https://docs.atlas.mongodb.com/global-clusters/) + [Select Cluster Tier](https://www.mongodb.com/docs/atlas/manage-clusters/#select-cluster-tier) + [Configure Additional Settings](https://docs.atlas.mongodb.com/cluster-additional-settings/) + [Get Started with Atlas](https://docs.atlas.mongodb.com/getting-started/) + [Configure Access to the Atlas UI](https://docs.atlas.mongodb.com/organizations-projects/) **Migrating data** + [Migrate or Import Data](https://www.mongodb.com/docs/atlas/import/) **Monitoring clusters** + [Monitor Your Clusters](https://docs.atlas.mongodb.com/monitoring-alerts/) **Integrating operations** + [Connect to a Cluster](https://docs.atlas.mongodb.com/connect-to-cluster/) + [Interact with Your Data](https://docs.atlas.mongodb.com/data-explorer/) + [Monitor Your Clusters](https://docs.atlas.mongodb.com/monitoring-alerts/) + [Back Up, Restore, and Archive Data](https://docs.atlas.mongodb.com/backup-restore-cluster/) # Migrate a self-hosted MongoDB environment to MongoDB Atlas on AWS *Battulga Purevragchaa and Igor Alekseev, Amazon Web Services* *Babu Srinivasan, MongoDB* ## Summary This pattern describes the steps for migrating from a self-managed MongoDB environment (including MongoDB Community Server, Enterprise Server, Enterprise Advanced, mLab, or any managed MongoDB cluster) to MongoDB Atlas in the AWS Cloud. It uses the [Atlas Live Migration Service](https://www.mongodb.com/cloud/atlas/migrate) to help accelerate the data migration from MongoDB to MongoDB Atlas. The pattern accompanies the guide [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/) on the AWS Prescriptive Guidance website. It provides the implementation steps for one of the migration scenarios that are discussed in that guide. For additional migration scenarios, see the following patterns on the AWS Prescriptive Guidance website: + [Migrate a relational database to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-relational-database-to-mongodb-atlas.html) + [Stream data from IBM Db2, SAP, Sybase, and other databases to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/stream-data-from-ibm-db2-to-mongodb-atlas.html) The pattern is intended for [AWS Systems Integrator (SI) Partners](https://aws.amazon.com/managed-services/partners/) and AWS users. ## Prerequisites and limitations **Prerequisites** + A source MongoDB Enterprise Advanced, Community Server, or other self-managed MongoDB environment to migrate to MongoDB Atlas. + Familiarity with MongoDB, MongoDB Atlas, and AWS services. This pattern explains some of the migration steps at a high level. Additional details will be added in future versions. **Product versions** + MongoDB version 6.0.13 or later ## Architecture The following diagram shows the Atlas Live Migration Service that is used to migrate data from a MongoDB Enterprise Advanced database and a MongoDB Community database to MongoDB Atlas on AWS. Use this service when you need to migrate large, complex databases to MongoDB Atlas with minimal downtime and continuous data synchronization. This pattern uses Atlas Live Migration Service. ![\[Migrating data with MongoDB Atlas Live Migration Service.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/52cdb923-64ff-4ee2-b129-93b9a139e24b/images/372134c4-ba47-4e48-bd0d-8b43017773b8.png) The following diagram shows the MongoDB mirror service (`mongomirror`) that you can also use to migrate data from a MongoDB Enterprise Advanced database and a MongoDB Community database to MongoDB Atlas on AWS over a secured [AWS PrivateLink](https://aws.amazon.com/privatelink/) connection. Use `mongomirror` for ongoing data replication between on-premises MongoDB and MongoDB Atlas. This tool is ideal for disaster recovery or phased migrations but is outside the scope of this pattern. ![\[Migrating data with mongomirror tool.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/52cdb923-64ff-4ee2-b129-93b9a139e24b/images/53488a9b-2210-4b3d-b517-b618c1e0182c.png) For more MongoDB Atlas reference architectures that support different usage scenarios, see [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/architecture.html) on the AWS Prescriptive Guidance website. ## Tools + [MongoDB Atlas](https://www.mongodb.com/atlas) is a fully managed database as a service (DbaaS) for deploying and managing MongoDB databases in the cloud. + [Atlas Live Migration Service](https://www.mongodb.com/cloud/atlas/migrate) is a free MongoDB utility that helps migrate databases to Atlas. This service keeps the source database in sync with the destination database until cutover. When you’re ready to cut over, you stop your application instances, point them to the destination Atlas cluster, and restart them. To access this service, choose **Database options** from your MongoDB Atlas cluster. + [mongomirror](https://www.mongodb.com/docs/atlas/import/mongomirror/) is a tool for manually migrating data from an existing MongoDB replica set to a MongoDB Atlas replica set. `mongomirror` doesn’t require you to shut down your existing replica set or applications, doesn’t import user or role data, or copy the configuration database. You can download `mongomirror` from the [MongoDB documentation](https://www.mongodb.com/docs/atlas/import/mongomirror/#download-mongomirror). ## Best practices For best practices for using MongoDB on AWS, see the posts on the [AWS Partner Network Blog](https://aws.amazon.com/blogs/apn/tag/mongodb-atlas/). ## Epics ### Discovery and assessment | Task | Description | Skills required | | --- | --- | --- | | Determine the cluster size. | Estimate the working set size by using the information from `db.stats()` for the total index space. Assume that a percentage of your data space will be accessed frequently. Or, you can estimate your memory requirements based on your own assumptions. This task should take approximately one week. For more information and examples for this and the other stories in this epic, see the [Related resources](#migrate-a-self-hosted-mongodb-environment-to-mongodb-atlas-on-the-aws-cloud-resources) section. | DBA, App owner | | Estimate network bandwidth requirements. | To estimate your network bandwidth requirements, multiply the average document size by the number of documents served per second. Consider the maximum traffic that any node on your cluster will bear as the basis. To calculate downstream data transfer rates from your cluster to client applications, use the sum of the total documents returned over a period of time. If your applications read from secondary nodes, divide this number of total documents by the number of nodes that can serve read operations. To find the average document size for a database, use the `db.stats().avgObjSize` command. This task will typically take one day. | DBA | | Select the Atlas tier. | Follow the instructions in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/sizing-tier-selection/) to select the correct Atlas cluster tier. | DBA | | Plan for cutover. | Plan for application cutover. | DBA, App owner | ### Set up a new MongoDB Atlas environment on AWS | Task | Description | Skills required | | --- | --- | --- | | Create a new MongoDB Atlas cluster on AWS. | Log in to Atlas and open the **Overview** page for your project. Choose the **Create** button to create a cluster. For more information, see the [MongoDB documentation](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster/). | DBA | | Select AWS Regions and global cluster configuration. | Select from the list of available AWS Regions for your Atlas cluster. Configure global clusters if required. For more information, see the [MongoDB documentation](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster/#select-your-preferred-region.). | DBA | | Select the cluster tier. | Select your preferred cluster tier. Your tier selection determines factors such as memory, storage, and IOPS specification. | DBA | | Configure additional cluster settings. | Configure additional cluster settings such as MongoDB version, backup, and encryption options. For more information about these options, see the [Related resources](#migrate-a-self-hosted-mongodb-environment-to-mongodb-atlas-on-the-aws-cloud-resources) section. | DBA | ### Configure security and compliance | Task | Description | Skills required | | --- | --- | --- | | Authenticate and authorize users. | You must create and authenticate the database users who will access the MongoDB Atlas clusters. To access clusters in a project, users must belong to that project, and they can belong to multiple projects. Atlas also supports authentication based on AWS Identity and Access Management (IAM). For more information, see the [MongoDB documentation](https://www.mongodb.com/docs/atlas/security/aws-iam-authentication/#set-up-authentication-with-aws-iam). | DBA | | Create custom roles. | (Optional) Atlas supports creating custom roles in cases where the built-in Atlas database user permissions don’t cover the set of permissions you want. | DBA | | Set up VPC peering. | (Optional) Atlas supports [virtual private cloud (VPC) peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) with other VPCs on AWS. | AWS administrator | | Set up an AWS PrivateLink endpoint. | (Optional) You can set up private endpoints on AWS by using AWS PrivateLink. For more information, see the [Amazon VPC documentation](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-services-overview.html). | AWS administrator | | Enable two-factor authentication. | (Optional) Atlas supports two-factor authentication (2FA) to help users control access to their Atlas accounts. | AWS administrator | | Set up user authentication and authorization with LDAP. | (Optional) Atlas supports performing user authentication and authorization with Lightweight Directory Access Protocol (LDAP). | AWS administrator | | Set up unified AWS access. | (Optional) Some Atlas features, including Atlas Data Lake and encryption at rest using customer key management, use IAM roles for authentication. | AWS administrator | | Set up encryption at rest using AWS KMS. | (Optional) Atlas supports using AWS Key Management Service (AWS KMS) to encrypt storage engines and cloud provider backups. | AWS administrator | | Set up client-side field-level encryption. | (Optional) Atlas supports client-side field level encryption, including automatic encryption of fields. | AWS administrator | ### Migrate data | Task | Description | Skills required | | --- | --- | --- | | Select your target replica set in MongoDB Atlas. | Navigate to the destination Atlas cluster and choose the ellipsis (...) button. On the cluster list, this button appears below the cluster name. In cluster details, the button appears on the right, next to the **Connect** and **Configuration** buttons. For more information, see the [MongoDB documentation](https://www.mongodb.com/docs/atlas/import/c2c-pull-live-migration/#procedure). | DBA | | Add the Atlas Live Migration Service to the access list. | Add the Atlas Live Migration Service to the access list in your AWS source cluster. This helps prepare the source environment to connect to the target Atlas cluster. | DBA | | Perform migration with Atlas Live Migration Service. | Choose **Start migration**. When the **Prepare to Cutover** button turns green, perform the cutover. Review Atlas cluster performance metrics. Consider updating the database connection in all application layers to point to the new database. | DBA | ### Configure operational integration | Task | Description | Skills required | | --- | --- | --- | | Connect to the MongoDB Atlas cluster. | Make sure that MongoDB Atlas cluster connectivity works as expected. | App owner | | Interact with cluster data. | Test the cluster data. | DBA | | Monitor your clusters. | Verify that your clusters are set up correctly. | DBA | | Back up and restore cluster data. | Schedule backups with a regular cadence for cluster data. | DBA | ## Troubleshooting | Issue | Solution | | --- | --- | | Error: Could not reach specified source | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-self-hosted-mongodb-environment-to-mongodb-atlas-on-the-aws-cloud.html) | | Error: Could not resolve hostname | No IP address was found for the given hostname. Confirm that the given hostname is correct and publicly accessible. | | Any other error | If you encounter any other errors, see [Troubleshoot Live Migration (Pull)](https://www.mongodb.com/docs/atlas/import/live-import-troubleshooting/) in the MongoDB documentation. | ## Related resources All of the following links, unless noted otherwise, go to webpages in the MongoDB documentation. **Migration guide** + [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/) (AWS Prescriptive Guidance) **Legacy migration** + [Migration of older versions of MongoDB](https://www.mongodb.com/docs/atlas/legacy-migration/) **Discovery and assessment** + [Memory](https://docs.atlas.mongodb.com/sizing-tier-selection/#memory) + [Sizing example with Atlas sample data sets](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#example--the-service-sample-data-sets) + [Sizing example for mobile applications](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#example--mobile-app) + [Network Traffic](https://docs.atlas.mongodb.com/sizing-tier-selection/#network-traffic) + [Cluster Auto-Scaling](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#cluster-auto-scaling) + [Atlas sizing template](https://view.highspot.com/viewer/5f438f47a4dfa042e97130c5) **Configuring security and compliance** + [Configure IP Access List Entries](https://docs.atlas.mongodb.com/security/ip-access-list/) + [Configure Database Users](https://docs.atlas.mongodb.com/security-add-mongodb-users/) + [Configure Access to the Atlas UI](https://docs.atlas.mongodb.com/organizations-projects/) + [Configure Custom Database Roles](https://docs.atlas.mongodb.com/security-add-mongodb-roles) + [Configure Database Users](https://docs.atlas.mongodb.com/security-add-mongodb-users/#atlas-user-privileges) + [Set up a Network Peering Connection](https://docs.atlas.mongodb.com/security-vpc-peering/) + [Learn About Private Endpoints in Atlas](https://docs.atlas.mongodb.com/security-private-endpoint/) + [Manage Your Multi-Factor Authentication Options](https://docs.atlas.mongodb.com/security-two-factor-authentication/) + [Set up User Authentication and Authorization with LDAP](https://docs.atlas.mongodb.com/security-ldaps/) + [Atlas Data Lake](https://docs.mongodb.com/datalake/) + [Encryption at Rest using Customer Key Management](https://docs.atlas.mongodb.com/security-kms-encryption/) + [Methods to assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) (IAM documentation) + [Client-Side Field Level Encryption](https://docs.mongodb.com/manual/core/security-client-side-encryption) + [Automatic Encryption](https://docs.mongodb.com/manual/core/security-automatic-client-side-encryption) + [MongoDB Atlas Security Controls](https://webassets.mongodb.com/_com_assets/cms/MongoDB_Atlas_Security_Controls-v7k3rbhi3p.pdf) + [MongoDB Trust Center](https://www.mongodb.com/cloud/trust) + [Configure Security Features for Clusters](https://docs.atlas.mongodb.com/setup-cluster-security/) **Setting up a new MongoDB Atlas environment on ****AWS** + [Cloud Providers and Regions](https://docs.atlas.mongodb.com/cloud-providers-regions/) + [Manage Global Clusters](https://docs.atlas.mongodb.com/global-clusters/) + [Select Cluster Tier](https://www.mongodb.com/docs/atlas/manage-clusters/#select-cluster-tier) + [Configure Additional Settings](https://docs.atlas.mongodb.com/cluster-additional-settings/) + [Get Started with Atlas](https://docs.atlas.mongodb.com/getting-started/) + [Configure Access to the Atlas UI](https://docs.atlas.mongodb.com/organizations-projects/) **Migrating data** + [Migrate or Import Data](https://www.mongodb.com/docs/atlas/import/) **Monitoring clusters** + [Monitor Your Clusters](https://docs.atlas.mongodb.com/monitoring-alerts/) **Integrating operations** + [Connect to a Cluster](https://docs.atlas.mongodb.com/connect-to-cluster/) + [Interact with Your Data](https://docs.atlas.mongodb.com/data-explorer/) + [Monitor Your Clusters](https://docs.atlas.mongodb.com/monitoring-alerts/) + [Back Up, Restore, and Archive Data](https://docs.atlas.mongodb.com/backup-restore-cluster/) **Training** + [Live Migration with MongoDB Atlas](https://learn.mongodb.com/courses/live-migration-with-mongodb-atlas) ## Additional information For additional information, see the following topics in the MongoDB documentation: + To move data to a serverless instance, [use Compass to export and import data](https://www.mongodb.com/docs/compass/current/import-export/), or migrate data with self-managed tools. To learn more, see [Serverless Instance Limitations](https://www.mongodb.com/docs/atlas/reference/serverless-instance-limitations/). + To load data into a new cluster in Atlas, see [Load Data into Atlas.](https://www.mongodb.com/docs/atlas/sample-data/#std-label-sample-data) + To make a copy of your cluster for testing purposes, see [Backup Methods for a Self-Managed Deployment](https://www.mongodb.com/docs/manual/core/backups/). + If the application that you want to migrate requires a near-continuous uptime, contact [MongoDB Support](https://www.mongodb.com/docs/atlas/support/#std-label-request-support) and share your uptime requirements and cluster configuration. + For additional information, see [Migrate or Import Data](https://www.mongodb.com/docs/atlas/import/). # Migrate an Oracle database to Amazon DynamoDB using AWS DMS *Rambabu Karnena, Amazon Web Services* ## Summary This pattern walks you through the steps for migrating an Oracle database to [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) using AWS Database Migration Service ([AWS DMS](https://aws.amazon.com/dms/)). It covers three types of source databases: + On-premises Oracle databases + Oracle databases on Amazon Elastic Compute Cloud ([Amazon EC2](https://aws.amazon.com/ec2/)) + Amazon Relational Database Service ([Amazon RDS](https://aws.amazon.com/rds/)) for Oracle DB instances In this proof of concept, this pattern focuses on migrating from an Amazon RDS for Oracle DB instance. ## Prerequisites and limitations **Prerequisites** + An active AWS account + An application connecting to an Amazon RDS for Oracle database + A table created in the source Amazon RDS for Oracle database with a primary key and sample data **Limitations** + Oracle database objects, such as procedures, functions, packages, and triggers, are not considered for migration because Amazon DynamoDB does not support these database objects. **Product versions** + This pattern applies to all editions and versions of Oracle databases that are supported by AWS DMS. For more information, see using an [Oracle database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) and using an [Amazon DynamoDB database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.DynamoDB.html). We recommend that you use the latest versions of AWS DMS for the most comprehensive version and feature support. ## Architecture **Source technology stack** + Amazon RDS for Oracle DB instances, Oracle on Amazon EC2, or on-premises Oracle databases **Target technology stack** + Amazon DynamoDB **AWS data migration architecture** ![\[Data moves from Oracle DB to AWS DMS to Amazon DynamoDB.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/463fc7d4-ec8d-473b-8c7f-1df31800ee03/images/180e7340-3887-455d-a591-b5850e22770a.png) ## Tools + [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups. + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. This pattern uses Amazon RDS for Oracle. ## Epics ### Plan the migration | Task | Description | Skills required | | --- | --- | --- | | Create a VPC. | In your AWS account, create a virtual private cloud (VPC) and a private subnet. | Systems administrator | | Create security groups and network access control lists. | For more information, see the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html). | Systems administrator | | Configure and start the Amazon RDS for Oracle DB instance. | For more information, see the [AWS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html). | DBA, Systems administrator | ### Migrate data | Task | Description | Skills required | | --- | --- | --- | | Create an IAM role to access DynamoDB. | In the AWS Identity and Access Management (IAM) console, create the role, attach the policy `AmazonDynamoDBFullAccess to it`, and select AWS DMS as the service. | Systems administrator | | Create an AWS DMS replication instance for migration. | The replication instance should be in the same Availability Zone and VPC as the source database. | Systems administrator | | Create source and target endpoints in AWS DMS. | To create the source database endpoint, you have two options:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-oracle-database-to-amazon-dynamodb-using-aws-dms.html)To create the target database endpoint, choose the role Amazon Resource Name (ARN) from the previous task to access DynamoDB. | Systems administrator | | Create an AWS DMS task to load the source Oracle database tables to DynamoDB. | Choose the source and destination endpoint names and the replication instance from the previous steps. The type can be full load. Choose the Oracle schema and specify **%** to select all tables. | Systems administrator | | Validate the tables in DynamoDB. | To view the migration results, choose **Tables** from the left navigation pane in the DynamoDB console. | DBA | ### Migrate the application | Task | Description | Skills required | | --- | --- | --- | | Modify the application code. | To connect to and retrieve data from DynamoDB, update the application code. | App owner, DBA, Systems administrator | ### Cut over | Task | Description | Skills required | | --- | --- | --- | | Switch the application clients to use DynamoDB. | | DBA, App owner, Systems administrator | ### Close the project | Task | Description | Skills required | | --- | --- | --- | | Shut down AWS resources. | For example, the shut down the Amazon RDS for Oracle instance, DynamoDB, and the AWS DMS replication instance. | DBA, Systems administrator | | Gather metrics. | Metrics include the time to migrate, the percentages of manual work and work performed by the tool, and cost savings. | DBA, App owner, Systems administrator | ## Related resources + [AWS Database Migration Service and Amazon DynamoDB: What You Need to Know](https://aws.amazon.com/blogs/database/aws-database-migration-service-and-amazon-dynamodb-what-you-need-to-know/) (blog post) + [Using an Oracle Database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html) + [Using an Amazon DynamoDB database as a target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.DynamoDB.html) + [Best Practices for Migrating from RDBMS to Amazon DynamoDB](https://docs.aws.amazon.com/whitepapers/latest/best-practices-for-migrating-from-rdbms-to-dynamodb/welcome.html) (whitepaper) # Migrate from Oracle 8i or 9i to Amazon RDS for Oracle using SharePlex and AWS DMS *Ramu Jagini, Amazon Web Services* ## Summary This pattern describes how to migrate an on-premises Oracle 8i or 9i database to an Amazon Relational Database Service (Amazon RDS) for Oracle database. You can use this pattern to complete your migration with reduced downtime by using Quest SharePlex for synchronous replication. You must use an intermediate Oracle database instance for your migration because AWS Database Migration Service (AWS DMS) doesn’t support Oracle 8i or 9i as a source environment. You can use [SharePlex 7.6.3](https://www.quest.com/community/shareplex/f/forum/20700/where-can-download-7-6-3-or-support-9i-shareplex) to replicate from previous Oracle database versions to later Oracle database versions. The intermediate Oracle database instance is compatible as a target for SharePlex 7.6.3 and supported as a source for AWS DMS or newer releases of SharePlex. This support enables onward replication of data to the Amazon RDS for Oracle target environment. Consider that several deprecated data types and features can impact a migration from Oracle 8i or 9i to the latest version of Oracle Database. To mitigate this impact, this pattern uses Oracle 11.2.0.4 as an intermediate database version to help optimize the schema code prior to migrating to the Amazon RDS for Oracle target environment. ## Prerequisites and limitations **Prerequisites** + An active AWS account + A source Oracle 8i or 9i database in an on-premises environment + [Oracle Database 12c Release 2](https://docs.oracle.com/en/database/oracle/oracle-database/12.2/index.html) (12CR2) for staging on Amazon Elastic Compute Cloud (Amazon EC2) + Quest SharePlex 7.6.3 (commercial grade) **Limitations** + [RDS for Oracle limitations](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Concepts.limitations.html) **Product versions** + Oracle 8i or 9i for the source database + Oracle 12CR2 for the staging database (must match the Amazon RDS for Oracle version) + Oracle 12CR2 or later for the target database (Amazon RDS for Oracle) ## Architecture **Source technology stack** + Oracle 8i or 9i database + SharePlex **Target technology stack** + Amazon RDS for Oracle **Migration architecture** The following diagram shows how to migrate an Oracle 8i or 9i database from an on-premises environment to an Amazon RDS for Oracle DB instance in the AWS Cloud. ![\[Workflow for migrating an on-premises Oracle database to Amazon RDS on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/6e07d586-fd74-4f3d-8e81-79dd55c445c3/images/36e1a5ff-908b-4cb7-96f7-997eb105f1d6.png) The diagram shows the following workflow: 1. Enable the Oracle source database with archive log mode, force logging, and supplemental logging. 1. Restore the Oracle staging database from the Oracle source database by using Recovery Manager (RMAN) point-in-time recovery and [FLASHBACK\$1SCN](https://docs.oracle.com/database/121/SUTIL/GUID-D408B112-1A81-4F68-BEFF-7403A9588DDB.htm#SUTIL849). 1. Configure SharePlex to read redo logs from the Oracle source database by using `FLASHBACK_SCN` (used in RMAN). 1. Start SharePlex replication to synchronize data from the Oracle source database to the Oracle staging database. 1. Restore the Amazon RDS for Oracle target database by using EXPDP and IMPDP with `FLASHBACK_SCN`. 1. Configure AWS DMS and its source tasks as the Oracle staging database and Amazon RDS for Oracle as the target database by using `FLASHBACK_SCN` (used in EXPDP). 1. Start AWS DMS tasks to synchronize data from the Oracle staging database to the Oracle target database. ## Tools + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. + [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups. + [Quest SharePlex](https://support.quest.com/shareplex/11.0/technical-documents) is an Oracle-to-Oracle data replication tool for moving data with minimal downtime and no data loss. + [Recovery Manager (RMAN)](https://docs.oracle.com/cd/E11882_01/backup.112/e10642/rcmquick.htm) is an Oracle Database client that performs backup and recovery tasks on your databases. It greatly simplifies backing up, restoring, and recovering database files. + [Data Pump Export](https://docs.oracle.com/cd/E11882_01/server.112/e22490/dp_export.htm#SUTIL823) helps you upload data and metadata into a set of operating system files called a dump file set. The dump file set can be imported only by the [Data Pump Import](https://docs.oracle.com/cd/E11882_01/server.112/e22490/dp_import.htm#SUTIL300) utility or [DBMS\$1DATAPUMP](https://docs.oracle.com/database/121/ARPLS/d_datpmp.htm#ARPLS356) package. ## Epics ### Set up SharePlex and the Oracle staging database on Amazon EC2 | Task | Description | Skills required | | --- | --- | --- | | Create an EC2 instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-8i-or-9i-to-amazon-rds-for-oracle-using-shareplex-and-aws-dms.html) | Oracle administration | | Prepare the staging database. | Prepare the Oracle staging database for restore as an upgrade on Oracle 12CR2 by taking the RMAN backup from the Oracle 8i or 9i database source environment.For more information, see [Oracle 9i Recovery Manager User's Guide](https://docs.oracle.com/cd/B10500_01/server.920/a96566/toc.htm) and [Database Backup and Recovery User's Guide](https://docs.oracle.com/database/121/BRADV/rcmcomre.htm#BRADV8005) in the Oracle documentation. | Oracle administration | | Configure SharePlex. | Configure the SharePlex source as an on-premises Oracle 8i or 9i database, and configure the target as the Oracle 12CR2 staging database hosted on Amazon EC2. | SharePlex, Oracle administration | ### Set up Amazon RDS for Oracle as your target environment | Task | Description | Skills required | | --- | --- | --- | | Create an Oracle DB instance. | Create an Amazon RDS for Oracle database, and then connect Oracle 12CR2 to the database.For more information, see [Creating an Oracle DB instance and connecting to a database on an Oracle DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.Oracle.html) in the Amazon RDS documentation. | DBA | | Restore Amazon RDS for Oracle from the staging database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-8i-or-9i-to-amazon-rds-for-oracle-using-shareplex-and-aws-dms.html)For more information, see [54 DBMS\$1DATAPUMP](https://docs.oracle.com/en/database/oracle/oracle-database/21/arpls/DBMS_DATAPUMP.html#GUID-AEA7ED80-DB4A-4A70-B199-592287206348) in the Oracle documentation. | DBA | ### Set up AWS DMS | Task | Description | Skills required | | --- | --- | --- | | Create endpoints for the databases. | Create a source endpoint for the Oracle staging database and a target endpoint for the Amazon RDS for Oracle database.For more information, see [How do I create source or target endpoints using AWS DMS?](https://aws.amazon.com/premiumsupport/knowledge-center/create-source-target-endpoints-aws-dms/) in the AWS Knowledge Center. | DBA | | Create a replication instance. | Use AWS DMS to launch a replication instance for the Oracle staging database to the Amazon RDS for Oracle database.For more information, see [How do I create an AWS DMS replication instance?](https://aws.amazon.com/premiumsupport/knowledge-center/create-aws-dms-replication-instance/) in the AWS Knowledge Center. | DBA | | Create and start replication tasks. | Create AWS DMS replication tasks for change data capture (CDC) by using `FLASHBACK_SCN` from EXPDP (since the full load already happened through EXPDP).For more information, see [Creating a task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.Creating.html) in the AWS DMS documentation. | DBA | ### Cut over to Amazon RDS for Oracle | Task | Description | Skills required | | --- | --- | --- | | Stop the application workload. | Stop the application servers and its applications during the planned cutover window. | App developer, DBA | | Validate the synching of the on-premises Oracle staging database with the EC2 instance. | Confirm that all messages have been posted for replication tasks from the SharePlex replication instance to the Oracle staging database on Amazon EC2 by performing a few log switches on the on-premises source database.For more information, see [6.4.2 Switching a Log File](https://docs.oracle.com/database/121/ADMQS/GUID-E30B4C65-2AC7-4A44-A58C-D3C121EB152F.htm#ADMQS12075) in the Oracle documentation. | DBA | | Validate the synching of the Oracle staging database with the Amazon RDS for Oracle database. | Confirm that all your AWS DMS tasks have no lag and no errors, and then check the validation state of the tasks. | DBA | | Stop the replication of SharePlex and Amazon RDS. | If both the SharePlex and AWS DMS replications are not showing any errors, then stop both replications. | DBA | | Remap the application to Amazon RDS. | Share the Amazon RDS for Oracle endpoint details with the application server and its applications, and then start the application to resume business operations. | App developer, DBA | ### Test the AWS target environment | Task | Description | Skills required | | --- | --- | --- | | Test the Oracle staging database environment on AWS. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-8i-or-9i-to-amazon-rds-for-oracle-using-shareplex-and-aws-dms.html) | SharePlex, Oracle administration | | Test the Amazon RDS environment. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-from-oracle-8i-or-9i-to-amazon-rds-for-oracle-using-shareplex-and-aws-dms.html)For more information, see [Amazon RDS for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) in the Amazon RDS documentation. | Oracle administration | ## Related resources + [Migrate with confidence](https://aws.amazon.com/cloud-migration/) + [Amazon EC2](https://aws.amazon.com/ec2/) + [Amazon RDS for Oracle](https://aws.amazon.com/rds/oracle/) + [AWS Database Migration Service](https://aws.amazon.com/dms/) + [Debugging Your AWS DMS Migrations: What to Do When Things Go Wrong (Part 1)](https://aws.amazon.com/blogs/database/debugging-your-aws-dms-migrations-what-to-do-when-things-go-wrong-part-1/) + [Debugging Your AWS DMS Migrations: What to Do When Things Go Wrong (Part 2)](https://aws.amazon.com/blogs/database/debugging-your-aws-dms-migrations-what-to-do-when-things-go-wrong-part-2/) + [Debugging Your AWS DMS Migrations: What to Do When Things Go Wrong? (Part 3)](https://aws.amazon.com/blogs/database/debugging-your-aws-dms-migrations-what-to-do-when-things-go-wrong-part-3/) + [SharePlex for Database Replication](https://aws.amazon.com/marketplace/pp/B07943W4MJ) + [SharePlex: database replication for any environment](https://www.youtube.com/watch?v=ygS_ouUaNus) # Migrate an on-premises MySQL database to Amazon EC2 *Lorenzo Mota, Amazon Web Services* ## Summary This pattern provides guidance for migrating an on-premises MySQL database to a MySQL database on an Amazon Elastic Compute Cloud (Amazon EC2) instance. The pattern discusses the use of AWS Database Migration Service (AWS DMS) or native MySQL tools such as **mysqldump** for the migration. It focuses on full database migration to a MySQL DB instance. The pattern is primarily for DBAs and solutions architects. It can be used in small or large projects, in a testing or final migration phase. We recommend that you run at least one testing cycle before you use this pattern in a production environment. ## Prerequisites and limitations **Prerequisites** + An active AWS account + A MySQL source database in an on-premises data center **Product versions** + MySQL version 5.5 and later + A target operating system supported by Amazon EC2; see [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/) ## Architecture **Source technology stack ** + An on-premises MySQL database **Target technology stack ** + A MySQL database instance on Amazon EC2 **AWS data migration methods** + AWS DMS + Native MySQL tools such as [mysqldump](https://dev.mysql.com/doc/refman/en/mysqldump.html), or third-party tools such as [Percona XtraBackup](https://www.percona.com/mysql/software/percona-xtrabackup) **Target architecture** The following diagram illustrates the target Amazon EC2 implementation after cutover. ![\[A MySQL DB instance on Amazon EC2 with replication to a standby MySQL DB instance.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/d22b3e25-4d3b-4bd7-ad07-501748d67752/images/34cab6f9-9107-4c3b-98ec-a6d7fa9f298a.png) **AWS data migration architecture** *Using AWS DMS:* The following diagram illustrates the data migration workflow based on AWS DMS for sending full and incremental changes to the target MySQL database until cutover. The network connection from on premises to AWS depends on the requirements of the SQL client and is outside the scope of this pattern. ![\[Using AWS DMS to send data to a target MySQL DB on Amazon EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/d22b3e25-4d3b-4bd7-ad07-501748d67752/images/c906c45d-fac5-4bb9-b8c8-55e2f9f05fd8.png) *Using other MySQL tools:* The following diagram illustrates the data migration workflow based on using MySQL tools to generate export dump files from the on-premises database. These files are moved to Amazon Simple Storage Service (Amazon S3) and imported into the target MySQL database before cutover. The network connection from on premises to AWS depends on the requirements of the SQL client and is outside the scope of this pattern. ![\[Using native MySQL tools to send data to a target MySQL DB on Amazon EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/d22b3e25-4d3b-4bd7-ad07-501748d67752/images/18e88877-7879-4a99-b985-25c56bf7c35f.png) Notes: + Depending on down time considerations and the size of the database for the final cutover, you can use AWS DMS or another change data capture (CDC) tool to minimize the cutover time. When you use a CDC tool such as AWS DMS, you can migrate to the target database in minutes. + An offline strategy with **mysqldump** can suffice if the size of the database and network latency allow for a short cutover migration window. (We recommend that you perform testing to get an approximate time.) + Usually a CDC strategy through AWS DMS requires more monitoring and complexity than an offline option. ## Tools **AWS services** + [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) supports several source and target databases. For information about MySQL source and target databases supported by AWS DMS, see [Using a MySQL-compatible database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html) and [Using a MySQL-compatible database as a target for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html). If your source database isn't supported by AWS DMS, you must choose another method to migrate your data. **Other tools** + [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) is a MySQL utility that creates a dump file from a MySQL database for backup or migration purposes. + [Percona XtraBackup](https://www.percona.com/mysql/software/percona-xtrabackup) is an open source utility for performing non-blocking backups on MySQL databases. ## Epics ### Plan the migration | Task | Description | Skills required | | --- | --- | --- | | Validate the database versions. | Validate the versions of the source and target databases. For information about MySQL versions supported by AWS DMS, see [Sources for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html) and [Targets for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html) in the AWS DMS documentation. | DBA | | Identify the target operating system. | Determine the version of the target operating system. For a list of target operating systems supported by Amazon EC2, see [Amazon EC2 FAQs](https://aws.amazon.com/ec2/faqs/). | DBA, Systems administrator | | Identify hardware requirements. | Determine the hardware requirements for the [target server instance](https://aws.amazon.com/rds/instance-types/) based on the MySQL compatibility list and capacity requirements. | DBA, Systems administrator | | Identify storage requirements. | Determine the storage type and capacity for the target database. | DBA, Systems administrator | | Identify network requirements. | Determine networking requirements such as latency and bandwidth. | DBA, Systems administrator | | Choose the target instance type. | Choose the [target instance type](https://aws.amazon.com/rds/instance-types/) based on capacity, storage features, and network features. | DBA, Systems administrator | | Identify security requirements. | Determine the network or host access security requirements for the source and target databases. | DBA, Systems administrator | | Identify users. | Determine the list of operating system users for the MySQL software installation. For more information, see the [MySQL documentation](https://dev.mysql.com/doc/mysql-security-excerpt/en/access-control.html). | DBA, Systems administrator | | Determine a backup strategy. | | DBA | | Determine availability requirements. | | DBA | | Identify the application migration or switchover strategy. | | DBA, Systems administrator | ### Configure the infrastructure | Task | Description | Skills required | | --- | --- | --- | | Create a virtual private cloud (VPC) and subnets. | Configure route tables, internet gateway, NAT gateways, and subnets. For more information, see [VPC configuration options](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc-options.html) in the Amazon VPC documentation. | Systems administrator | | Create security groups and network access control lists (ACLs). | Configure ports (default for MySQL is 3306) and CIDR ranges or specific IPs depending on your requirements. | Systems administrator | | Configure and start an EC2 instance. | For instructions, see [Launch an EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html) in the Amazon EC2 documentation. | Systems administrator | ### Install MySQL software | Task | Description | Skills required | | --- | --- | --- | | Create users and groups. | Create the operating system users and groups who need access to the server and database. For more information, see [Access Control and Account Management](https://dev.mysql.com/doc/refman/en/access-control.html) in the MySQL documentation. | DBA, Systems administrator | | Download MySQL. | Download the MySQL software. For instructions and binaries, see [Installing MySQL](https://dev.mysql.com/doc/refman/en/installing.html) in the MySQL documentation. | DBA, Systems administrator | | Install MySQL on the EC2 instance and configure the server. | Connect to your EC2 instance and install the MySQL software. For more information, see [Connect to your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect.html) in the Amazon EC2 documentation. | DBA, Systems administrator | ### Migrate data – option 1 | Task | Description | Skills required | | --- | --- | --- | | Migrate data with native MySQL or third-party tools. | This option uses native MySQL tools or third-party tools to migrate database objects and data. For instructions, see the documentation for [mysqldump](https://dev.mysql.com/doc/refman/en/mysqldump.html) or [Percona XtraBackup](https://docs.percona.com/percona-xtrabackup/2.4/index.html) (for physical migration). For more information about using these tools, see the AWS blog post [Migration options for MySQL to Amazon RDS for MySQL or Amazon Aurora MySQL](https://aws.amazon.com/blogs/database/migration-options-for-mysql-to-amazon-rds-for-mysql-or-amazon-aurora-mysql/). | DBA | ### Migrate data – option 2 | Task | Description | Skills required | | --- | --- | --- | | Migrate data with AWS DMS. | For more information, see [High-level view of AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.HighLevelView.html) in the AWS DMS documentation. | DBA | ### Prepare for cutover | Task | Description | Skills required | | --- | --- | --- | | Gather object counts. | Gather object counts from source database and new target databases. Fix any discrepancies in the target database. | DBA | | Check dependencies. | Confirm that dependencies (links) to and from other databases are still valid and work correctly. | DBA | | Test. | If this is a testing cycle, perform query testing, gather metrics, and fix any issues. | DBA | ### Cut over | Task | Description | Skills required | | --- | --- | --- | | Move clients. | Switch the application clients over to the new infrastructure. | DBA, App owner, Systems administrator | | Provide support. | Provide support during functional application testing. | DBA | ### Close the project | Task | Description | Skills required | | --- | --- | --- | | Shut down resources. | Shut down the AWS DMS replication instance and other temporary AWS resources. | DBA, Systems administrator | | Review and project documents. | Review and validate the project documents. | DBA, App owner, Systems administrator | | Collect metrics. | Collect metrics such as time to migrate, percentage of manual changes compared with tool-aided changes, and cost savings. | DBA, App owner, Systems administrator | | Close out project. | Close out the migration project and provide feedback. | DBA, App owner, Systems administrator | | Decommission source database. | Decommission the on-premises MySQL database. | DBA, Systems administrator | ## Related resources **References** + [Amazon EC2 documentation](https://docs.aws.amazon.com/ec2/) + [AWS DMS documentation](https://docs.aws.amazon.com/dms/) + [Amazon EC2 Pricing](https://aws.amazon.com/ec2/pricing/) + [AWS DMS Step-by-Step Walkthroughs](https://docs.aws.amazon.com/dms/latest/sbs/DMS-SBS-Welcome.html) + [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) + [Percona XtraBackup](https://www.percona.com/mysql/software/percona-xtrabackup) **Tutorials and videos** + [Getting Started with AWS DMS](https://aws.amazon.com/dms/getting-started/) + [Introduction to Amazon EC2 – Elastic Cloud Server & Hosting with AWS](https://www.youtube.com/watch?v=TsRBftzZsQo) (video) # Monitor Amazon Aurora for instances without encryption *Mansi Suratwala, Amazon Web Services* ## Summary This pattern provides an Amazon Web Services (AWS) CloudFormation template that you can deploy to set up automatic notifications when an Amazon Aurora instance is created without encryption turned on. Aurora is a fully managed relational database engine that's compatible with MySQL and PostgreSQL. With some workloads, Aurora can deliver up to five times the throughput of MySQL and up to three times the throughput of PostgreSQL without requiring changes to most of your existing applications. The CloudFormation template creates an Amazon CloudWatch Events event and an AWS Lambda function. The event uses AWS CloudTrail to monitor for any Aurora instance creation or a point in time restoration of an existing instance. The Cloudwatch Events event initiates the Lambda function, which checks whether encryption is enabled. If encryption is not turned on, the Lambda function sends an Amazon Simple Notification Service (Amazon SNS) notification informing you of the violation. ## Prerequisites and limitations **Prerequisites**** ** + An active AWS account **Limitations ** + This service control works with Amazon Aurora instances only. It does not support other Amazon Relational Database Service (Amazon RDS) instances. + The CloudFormation template must be deployed for `CreateDBInstance`** **and `RestoreDBClusterToPointInTim`**e **only. **Product versions** + PostgreSQL versions that are supported in Amazon Aurora + MySQL versions that are supported in Amazon Aurora ## Architecture **Target technology stack ** + Amazon Aurora + AWS CloudTrail + Amazon CloudWatch + AWS Lambda + Amazon Simple Storage Service (Amazon S3) + Amazon SNS **Target architecture ** ![\[Aurora launch without encryption invoking CloudTrail, CloudWatch Events, Lambda, and an SNS message.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/de1528b8-a5a4-4c66-8ab3-7d9863572cbc/images/7dcab41a-d805-4bb0-99d1-1dcef37c4e55.png) **Automation and scale** You can use the CloudFormation template multiple times for different Regions and accounts. You need to run it only once in each Region or account. ## Tools **Tools** + [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) – Amazon Aurora is a fully managed relational database engine that's compatible with MySQL and PostgreSQL. + [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html) – AWS CloudTrail helps you manage governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, a role, or an AWS service are recorded as events in CloudTrail. + [Amazon CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html) – Amazon CloudWatch Events delivers a near-real-time stream of system events that describe changes in AWS resources. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) – AWS Lambda is a compute service that supports running code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. + [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html) – Amazon Simple Storage Service (Amazon S3) is a highly scalable object storage service that you can use for a wide range of storage solutions, including websites, mobile applications, backups, and data lakes. + [Amazon SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) – Amazon Simple Notification Service (Amazon SNS) is a managed service that provides message delivery using Lambda, HTTP, email, mobile push notifications, and mobile text messages (SMS). **Code ** A .zip file of the project is available as an attachment. ## Epics ### Create the S3 bucket for the Lambda script | Task | Description | Skills required | | --- | --- | --- | | Define the S3 bucket. | Open the Amazon S3 console, and choose or create an S3 bucket. This S3 bucket will host the Lambda code .zip file. Your S3 bucket needs to be in the same Region as Aurora. The S3 bucket name cannot contain leading slashes. | Cloud architect | ### Upload the Lambda code to the S3 bucket | Task | Description | Skills required | | --- | --- | --- | | Upload the Lambda code. | Upload the Lambda code .zip file provided in the *Attachments* section to the S3 bucket that you defined. | Cloud architect | ### Deploy the CloudFormation template | Task | Description | Skills required | | --- | --- | --- | | Deploy the CloudFormation template. | On the CloudFormation console, deploy the `RDS_Aurora_Encryption_At_Rest.yml` CloudFormation template that's provided as an attachment to this pattern. In the next epic, provide values for the template parameters. | Cloud architect | ### Complete the parameters in the CloudFormation template | Task | Description | Skills required | | --- | --- | --- | | Provide the S3 bucket name. | Enter the name of the S3 bucket that you created or chose in the first epic. | Cloud architect | | Provide the S3 key. | Provide the location of the Lambda code .zip file in your S3 bucket, without leading slashes (for example, `
(sidadm $> python systemReplicationStatus.py
sudo yum install -y awslogs| AWS general | | Edit the agent configuration files. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/monitor-oracle-goldengate-logs-by-using-amazon-cloudwatch.html) | AWS general | | Start the CloudWatch agent. | To start the agent, use the following command.
$ sudo service awslogsd startAfter you start the agent, you can view the log group in the CloudWatch console. The log stream will have the contents of the file. | AWS general | ### Step 6. Create metric filters for the log group | Task | Description | Skills required | | --- | --- | --- | | Create metric filters for the keywords ABEND and STOPPED. | When you create metric filters for the log group, whenever the filters are identified in the error log, it starts an alarm and sends an email notification based on the Amazon SNS configuration.To create metric filters:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/monitor-oracle-goldengate-logs-by-using-amazon-cloudwatch.html) | CloudWatch | ### Step 7. Set up Amazon SNS | Task | Description | Skills required | | --- | --- | --- | | Create an SNS topic. | In this step, you configure Amazon SNS to create alarms for the metric filters.To create an SNS topic:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/monitor-oracle-goldengate-logs-by-using-amazon-cloudwatch.html) | Amazon SNS | | Create a subscription. | To create a subscription to the topic:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/monitor-oracle-goldengate-logs-by-using-amazon-cloudwatch.html)Amazon SNS opens your web browser and displays a subscription confirmation with your subscription ID. | Amazon SNS | ### Step 8. Create an alarm to send notifications for the metric filters | Task | Description | Skills required | | --- | --- | --- | | Create an alarm for the SNS topic. | To create an alarm based on a log group-metric filter:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/monitor-oracle-goldengate-logs-by-using-amazon-cloudwatch.html)After these steps, whenever these patterns are detected in the GoldenGate error log file (`ggserr.log`) that you are monitoring, you will get an email notification. | CloudWatch | ## Troubleshooting | Issue | Solution | | --- | --- | | The log stream from the GoldenGate error log doesn’t flow into CloudWatch. | Check the `/etc/awslogs/awslogs.conf` file to verify the file name, log group name, and the date/time format. You must specify the date/time to match the date format in `ggserror.log`. Otherwise, the log stream won’t flow into CloudWatch. | ## Related resources + [Amazon CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) + [Collecting metrics and logs with the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) + [Amazon SNS documentation](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) # Replatform Oracle Database Enterprise Edition to Standard Edition 2 on Amazon RDS for Oracle *Lanre (Lan-Ray) showunmi and Tarun Chawla, Amazon Web Services* ## Summary Oracle Database Enterprise Edition (EE) is a popular choice for running applications in many enterprises. In some cases, however, applications use few or no Oracle Database EE features, so there is a lack of justification for incurring huge licensing costs. You can achieve cost savings by downgrading such databases to Oracle Database Standard Edition 2 (SE2) when you migrate to Amazon RDS. This pattern describes how to downgrade from Oracle Database EE to Oracle Database SE2 when migrating from on premises to [Amazon RDS for Oracle](https://aws.amazon.com/rds/oracle/). The steps presented in this pattern also apply if your EE Oracle database is already running on Amazon RDS or on an [Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) (Amazon EC2) instance. For more information, see the AWS Prescriptive Guidance guide on how to [Evaluate downgrading Oracle databases to Standard Edition 2 on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/evaluate-downgrading-oracle-edition/welcome.html). ## Prerequisites and limitations **Prerequisites** + An active AWS account + Oracle Database Enterprise Edition + A client tool, such as [Oracle SQL Developer](https://www.oracle.com/database/sqldeveloper/) or SQL\$1Plus, for connecting to and running SQL commands on Oracle database + Database user for performing the assessment; for example, one of the following: + User with sufficient [privileges](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Permissions) for running [AWS Schema Conversion Tool (AWS SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) assessment + User with sufficient privileges to run SQL queries on Oracle database dictionary tables + Database user for performing database migration; for example, one of the following: + User with sufficient [privileges](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.Self-Managed) for running [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) + User with sufficient [privileges for performing Oracle Data Pump export and import](https://docs.oracle.com/database/121/SUTIL/GUID-8B6975D3-3BEC-4584-B416-280125EEC57E.htm#SUTIL807) + User with sufficient [privileges for running Oracle GoldenGate](https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/user_assignment.htm#GIORA546) **Limitations ** + Amazon RDS for Oracle has a maximum database size. For more information, see [Amazon RDS DB instance storage](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). **Product versions** The general logic described in this document applies to Oracle versions from 9i and later. For supported versions of self-managed and Amazon RDS for Oracle databases, see the [AWS DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html). To identify feature usage in cases where AWS SCT is not supported, run SQL queries on the source database. To migrate from earlier versions of Oracle where AWS DMS and Oracle Data Pump are not supported, use [Oracle Export and Import utilities](https://docs.oracle.com/cd/B19306_01/server.102/b14215/exp_imp.htm). For a current list of supported versions and editions, see [Oracle on Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.html) in the AWS documentation. For details on pricing and supported instance classes, see [Amazon RDS for Oracle pricing](https://aws.amazon.com/rds/oracle/pricing/). ## Architecture **Source technology stack ** + Oracle Database Enterprise Edition running on premises or on Amazon EC2 **Target technology stack using native Oracle tools ** + Amazon RDS for Oracle running Oracle Database SE2 ![\[Three-step process for migrating from on-premises Oracle DB to Amazon RDS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a1b28050-9bab-4de6-b2a9-b97b3e5070bd/images/bf765c5b-4b12-4a8c-b27c-c5e0bd605dd1.png) 1. Export data by using Oracle Data Pump. 1. Copy dump files to Amazon RDS through a database link. 1. Import dump files to Amazon RDS by using Oracle Data Pump. **Target technology stack using AWS DMS ** + Amazon RDS for Oracle running Oracle Database SE2 + AWS DMS ![\[Four-step process for migrating from on-premises Oracle DB to Amazon RDS using AWS DMS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a1b28050-9bab-4de6-b2a9-b97b3e5070bd/images/fef4eced-1acb-4303-baaa-5c1c29650935.png) 1. Export data by using Oracle Data Pump with FLASHBACK\$1SCN. 1. Copy dump files to Amazon RDS through a database link. 1. Import dump files to Amazon RDS by using Oracle Data Pump. 1. Use AWS DMS [change data capture (CDC)](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html). ## Tools **AWS services** + [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. This pattern uses Amazon RDS for Oracle. + [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html)** **provides a project-based user interface to automatically assess, convert, and copy the database schema of your source Oracle database into a format compatible with Amazon RDS for Oracle. AWS SCT enables you to analyze potential cost savings that can be achieved by changing your license type from Enterprise to Standard Edition of Oracle. The **License Evaluation and Cloud Support** section of the AWS SCT report provides detailed information about Oracle features in use so you can make an informed decision while migrating to Amazon RDS for Oracle. **Other tools** + Native Oracle import and export utilities support moving Oracle data in and out of Oracle databases. Oracle offers two types of database import and export utilities: [Original Export and Import](https://docs.oracle.com/cd/B19306_01/server.102/b14215/exp_imp.htm) (for earlier releases) and [Oracle Data Pump Export and Import](https://docs.oracle.com/cd/B19306_01/server.102/b14215/part_dp.htm#CEGJCCHC) (available in Oracle Database 10g release 1 and later). + [Oracle GoldenGate](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.OracleGoldenGate.html) offers real-time replication capabilities so that you can synchronize your target database after an initial load. This option can help reduce application downtime during go-live. ## Epics ### Make a pre-migration assessment | Task | Description | Skills required | | --- | --- | --- | | Validate database requirements for your applications. | Ensure that your applications are certified to run on Oracle Database SE2. Check directly with the software vendor, developer, or application documentation. | App developer, DBA, App owner | | Investigate use of EE features directly in the database. | To determine EE feature use, do one of the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replatform-oracle-database-enterprise-edition-to-standard-edition-2-on-amazon-rds-for-oracle.html) | App owner, DBA, App developer | | Identify use of EE features for operational activities. | Database or application administrators sometimes rely on EE-only features for operational activities. Common examples include online maintenance activities (index rebuild, table move) and use of parallelism by batch jobs.These dependencies can be mitigated by modifying your operations where possible. Identify the use of these features and make a decision based on cost compared with benefits.Use the [Comparing Oracle Database EE and SE2 features ](https://docs.aws.amazon.com/prescriptive-guidance/latest/evaluate-downgrading-oracle-edition/compare-features.html)table as a guide to identify features that are available in Oracle Database SE2. | App developer, DBA, App owner | | Review workload patterns of the EE Oracle database. | Oracle Database SE2 automatically restricts usage to a maximum of 16 CPU threads at any time.If your Oracle EE database is licensed to use the Oracle Diagnostic Pack, use the Automatic Workload Repository (AWR) tool, or DBA\$1HIST\$1\$1 views, to analyze database workload patterns to determine whether the maximum limit of 16 CPU threads will negatively impact service levels when you downgrade to SE2.Ensure that your assessment covers periods of peak activity, such as end of day, month, or year processing. | App owner, DBA, App developer | ### Prepare the target infrastructure on AWS | Task | Description | Skills required | | --- | --- | --- | | Deploy and configure networking infrastructure. | Create a [virtual private cloud (VPC) and subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html), [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html), and [network access control lists](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html). | AWS administrator, Cloud architect, Network administrator, DevOps engineer | | Provision the Amazon RDS for Oracle SE2 database. | Provision the target [Amazon RDS for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.Oracle.html) SE2 database to meet your applications’ performance, availability, and security requirements. We recommend Multi-AZ configuration for production workloads. However, to improve migration performance, you can defer [enabling Multi-AZ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/create-multi-az-db-cluster.html) until after data migration. | Cloud administrator, Cloud architect, DBA, DevOps engineer, AWS administrator | | Customize the Amazon RDS environment. | Configure custom [parameters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) and [options](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithOptionGroups.html), and enable additional [monitoring](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.html). For more information, see [Best practices for migrating to Amazon RDS for Oracle](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/best-practices.html). | AWS administrator, AWS systems administrator, Cloud administrator, DBA, Cloud architect | ### Perform the migration dry run and application testing | Task | Description | Skills required | | --- | --- | --- | | Migrate the data (dry run). | Migrate data from the source Oracle EE database to the Amazon RDS for Oracle SE2 database instance using the approach best suited to your specific environment. Select a migration strategy based on factors such as size, complexity, and the available downtime window. Use one or a combination of the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replatform-oracle-database-enterprise-edition-to-standard-edition-2-on-amazon-rds-for-oracle.html) | DBA | | Validate the target database. | Perform post-migration validation of database storage and code objects. Review migration logs, and fix any identified issues. For more information, see the guide [Migrating Oracle databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/best-practices.html#post-import). | DBA | | Test the applications. | Application and database administrators should conduct functional, performance, and operational tests as appropriate. For more information, see [Best practices for migrating to Amazon RDS for Oracle](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/best-practices.html#test-migration).Lastly, obtain sign-offs on test-results from stakeholders. | App developer, App owner, DBA, Migration engineer, Migration lead | ### Cut over | Task | Description | Skills required | | --- | --- | --- | | Refresh data from Oracle Database EE. | Select a data refresh approach based on the application availability requirement. For more information, see the migration methods in [Strategies for Migrating Oracle Databases to AWS](https://docs.aws.amazon.com/whitepapers/latest/strategies-migrating-oracle-db-to-aws/data-migration-methods.html).For example, you can achieve near-zero downtime by using tools such as Oracle GoldenGate or AWS DMS with ongoing replication. If the downtime window permits, you can perform the final data cutover using offline methods such as Oracle Data Pump or Original Export-Import utilities. | App owner, Cutover lead, DBA, Migration engineer, Migration lead | | Point applications to the target database instance. | Update connection parameters in applications and other clients to point to the Amazon RDS for Oracle SE2 database. | App developer, App owner, Migration engineer, Migration lead, Cutover lead | | Perform post-migration activities. | Perform post data migration tasks such as enabling Multi-AZ, data validation, and other checks. | DBA, Migration engineer | | Perform post-cutover monitoring. | Use tools such as [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/monitoring-cloudwatch.html) and Amazon [RDS Performance Insights](https://aws.amazon.com/rds/performance-insights/) to monitor the Amazon RDS for Oracle SE2 database. | App developer, App owner, AWS administrator, DBA, Migration engineer | ## Related resources **AWS Prescriptive Guidance** + [Migrating Oracle databases to the AWS Cloud](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/welcome.html) (guide) + [Evaluate downgrading Oracle databases to Standard Edition 2 on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/evaluate-downgrading-oracle-edition/welcome.html) (guide) + [Migrate an on-premises Oracle database to Amazon RDS for Oracle](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.html?did=pg_card&trk=pg_card) (pattern) + [Migrate an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-using-oracle-data-pump.html?did=pg_card&trk=pg_card) (pattern) **Blog posts** + [Migrating Oracle databases with near-zero downtime using AWS DMS](https://aws.amazon.com/blogs/database/migrating-oracle-databases-with-near-zero-downtime-using-aws-dms/) + [Analyzing performance management in Oracle SE using Amazon RDS for Oracle](https://aws.amazon.com/blogs/database/analyzing-performance-management-in-oracle-se-using-amazon-rds-for-oracle/) + [Managing your SQL plan in Oracle SE with Amazon RDS for Oracle](https://aws.amazon.com/blogs/database/managing-your-sql-plan-in-oracle-se-with-amazon-rds-for-oracle/) + [Implementing table partitioning in Oracle Standard Edition: Part 1](https://aws.amazon.com/blogs/database/implementing-table-partitioning-in-oracle-standard-edition-part-1/) # Replicate mainframe databases to AWS by using Precisely Connect *Lucio Pereira, Sayantan Giri, and Balaji Mohan, Amazon Web Services* ## Summary This pattern outlines steps for replicating data from mainframe databases to Amazon data stores in near real time by using Precisely Connect. It implements an event-based architecture with Amazon Managed Streaming for Apache Kafka (Amazon MSK) and custom database connectors in the cloud to improve scalability, resilience, and performance. Precisely Connect is a replication tool that captures data from legacy mainframe systems and integrates it into cloud environments. Data is replicated from mainframes to AWS through change data capture (CDC) by using near real-time message flows with low-latency and high-throughput heterogeneous data pipelines. This pattern also covers a disaster recovery strategy for resilient data pipelines with multi-Region data replication and failover routing. ## Prerequisites and limitations **Prerequisites** + An existing mainframe database—for example, IBM DB2, IBM Information Management System (IMS), or Virtual Storage Access Method (VSAM)—that you want to replicate to the AWS Cloud + An active [AWS account](https://aws.amazon.com/account/) + [AWS Direct Connect](https://aws.amazon.com/directconnect/) or [AWS Virtual Private Network (AWS VPN](https://aws.amazon.com/vpn/)) from your corporate environment to AWS + A [virtual private cloud](https://aws.amazon.com/vpc/) with a subnet that is reachable by your legacy platform ## Architecture **Source technology stack** A mainframe environment that includes at least one of the following databases: + IBM IMS database + IBM DB2 database + VSAM files **Target technology stack** + Amazon MSK + Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon EKS Anywhere + Docker + An AWS relational or NoSQL database such as the following: + Amazon DynamoDB + Amazon Relational Database Service (Amazon RDS) for Oracle, Amazon RDS for PostgreSQL, or Amazon Aurora + Amazon ElastiCache for Redis + Amazon Keyspaces (for Apache Cassandra) **Target architecture** *Replicating mainframe data to AWS databases* The following diagram illustrates the replication of mainframe data to an AWS database such as DynamoDB, Amazon RDS, Amazon ElastiCache, or Amazon Keyspaces. The replication occurs in near real time by using Precisely Capture and Publisher in your on-premises mainframe environment, Precisely Dispatcher on Amazon EKS Anywhere in your on-premises distributed environment, and Precisely Apply Engine and database connectors in the AWS Cloud. ![\[Replicating mainframe data to AWS databases\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/17ac53b7-86d5-4a8c-a55f-56b6338a1af3/images/777dd7da-48ed-4510-b8fa-9041be042671.png) The diagram shows the following workflow: 1. Precisely Capture gets mainframe data from CDC logs and maintains the data in internal transient storage. 1. Precisely Publisher listens for changes in the internal data storage and sends CDC records to Precisely Dispatcher through a TCP/IP connection. 1. Precisely Dispatcher receives the CDC records from Publisher and sends them to Amazon MSK. Dispatcher creates Kafka keys based on the user configuration and multiple worker tasks to push data in parallel. Dispatcher sends an acknowledgment back to Publisher when records have been stored in Amazon MSK. 1. Amazon MSK holds the CDC records in the cloud environment. The partition size of topics depends on your transaction processing system (TPS) requirements for throughput. The Kafka key is mandatory for further transformation and transaction ordering. 1. The Precisely Apply Engine listens to the CDC records from Amazon MSK and transforms the data (for example, by filtering or mapping) based on target database requirements. You can add customized logic to the Precisely SQD scripts. (SQD is Precisely’s proprietary language.) The Precisely Apply Engine transforms each CDC record to Apache Avro or JSON format and distributes it to different topics based on your requirements. 1. The target Kafka topics hold CDC records in multiple topics based on the target database, and Kafka facilitates transaction ordering based on the defined Kafka key. The partition keys align with the corresponding partitions to support a sequential process. 1. Database connectors (customized Java applications) listen to the CDC records from Amazon MSK and store them in the target database. 1. You can select a target database based on your requirements. This pattern supports both NoSQL and relational databases. *Disaster recovery* Business continuity is key to your organization’s success. The AWS Cloud provides capabilities for high availability (HA) and disaster recovery (DR), and supports your organization’s failover and fallback plans. This pattern follows an active/passive DR strategy and provides high-level guidance for implementing a DR strategy that meets your RTO and RPO requirements. The following diagram illustrates the DR workflow. ![\[Disaster recovery workflow for replicating mainframe data on AWS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/17ac53b7-86d5-4a8c-a55f-56b6338a1af3/images/9cccba7a-7a25-411e-829f-7cd5a7a20ab4.png) The diagram shows the following: 1. A semi-automated failover is required if any failure happens in AWS Region 1. In the case of failure in Region 1, the system must initiate routing changes to connect Precisely Dispatcher to Region 2. 1. Amazon MSK replicates data through mirroring between Regions, For this reason, during failover, the Amazon MSK cluster in Region 2 has to be promoted as the primary leader. 1. The Precisely Apply Engine and database connectors are stateless applications that can work in any Region. 1. Database synchronization depends on the target database. For example, DynamoDB can use global tables, and ElastiCache can use global datastores. *Low-latency and high-throughput processing through database connectors* Database connectors are critical components in this pattern. Connectors follow a listener-based approach to collect data from Amazon MSK and send transactions to the database through high-throughput and low-latency processing for mission-critical applications (tiers 0 and 1). The following diagram illustrates this process. ![\[Using database connectors to replicate mainframe data on AWS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/17ac53b7-86d5-4a8c-a55f-56b6338a1af3/images/79479634-becb-4212-bbfc-1a3b17ae1bed.png) This pattern supports the development of a customized application with single-threaded consumption through a multithreaded processing engine. 1. The connector main thread consumes CDC records from Amazon MSK and sends them to the thread pool for processing. 1. Threads from the thread pool process CDC records and send them to the target database. 1. If all threads are busy, the CDC records are kept on hold by the thread queue. 1. The main thread waits to get all the records cleared from the thread queue and commits offsets into Amazon MSK. 1. The child threads handle failures. If failures happen during processing, the failed messages are sent to the DLQ (dead letter queue) topic. 1. The child threads initiate conditional updates (see [Condition expressions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html) in the DynamoDB documentation), based on the mainframe timestamp, to avoid any duplication or out-of-order updates in the database. For information about how to implement a Kafka consumer application with multi-threading capabilities, see the blog post [Multi-Threaded Message Consumption with the Apache Kafka Consumer](https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/) on the Confluent website. ## Tools **AWS services** + [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) is a fully managed service that helps you build and run applications that use Apache Kafka to process streaming data. + [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) helps you run Kubernetes on AWS without having to install or maintain your own Kubernetes control plane or nodes. + [Amazon EKS Anywhere](https://anywhere.eks.amazonaws.com/docs/) helps you deploy, use, and manage Kubernetes clusters that run in your own data centers. + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) is a fully managed NoSQL database service that provides fast, predictable, and scalable performance. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. + [Amazon ElastiCache](https://docs.aws.amazon.com/elasticache/) helps you set up, manage, and scale distributed in-memory cache environments in the AWS Cloud. + [Amazon Keyspaces (for Apache Cassandra)](https://docs.aws.amazon.com/keyspaces/latest/devguide/what-is-keyspaces.html) is a managed database service that helps you migrate, run, and scale your Cassandra workloads in the AWS Cloud. **Other tools** + [Precisely Connect](https://www.precisely.com/product/precisely-connect/connect) integrates data from legacy mainframe systems such as VSAM datasets or IBM mainframe databases into next-generation cloud and data platforms. ## Best practices + Find the best combination of Kafka partitions and multi-threaded connectors to balance optimal performance and cost. Multiple Precisely Capture and Dispatcher instances can increase cost because of higher MIPS (million instructions per second) consumption. + Avoid adding data manipulation and transformation logic to the database connectors. For this purpose, use the Precisely Apply Engine, which provides processing times in microseconds. + Create periodic request or health check calls to the database (*heartbeats*) in database connectors to warm up the connection frequently and reduce latency. + Implement thread pool validation logic to understand the pending tasks in the thread queue and wait for all threads to be completed before the next Kafka polling. This helps avoid data loss if a node, container, or process crashes. + Expose latency metrics through health endpoints to enhance observability capabilities through dashboards and tracing mechanisms. ## Epics ### Prepare the source environment (on premises) | Task | Description | Skills required | | --- | --- | --- | | Set up the mainframe process (batch or online utility) to start the CDC process from mainframe databases. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Mainframe engineer | | Activate mainframe database log streams. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Mainframe DB specialist | | Use the Capture component to capture CDC records. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Mainframe engineer, Precisely Connect SME | | Configure the Publisher component to listen to the Capture component. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Mainframe engineer, Precisely Connect SME | | Provision Amazon EKS Anywhere in the on-premises distributed environment. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | DevOps engineer | | Deploy and configure the Dispatcher component in the distributed environment to publish the topics in the AWS Cloud. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | DevOps engineer, Precisely Connect SME | ### Prepare the target environment (AWS) | Task | Description | Skills required | | --- | --- | --- | | Provision an Amazon EKS cluster in the designated AWS Region. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | DevOps engineer, Network administrator | | Provision an MSK cluster and configure applicable Kafka topics. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | DevOps engineer, Network administrator | | Configure the Apply Engine component to listen to the replicated Kafka topics. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Precisely Connect SME | | Provision DB instances in the AWS Cloud. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Data engineer, DevOps engineer | | Configure and deploy database connectors to listen to the topics published by the Apply Engine. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | App developer, Cloud architect, Data engineer | ### Set up business continuity and disaster recovery | Task | Description | Skills required | | --- | --- | --- | | Define disaster recovery goals for your business applications. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Cloud architect, Data engineer, App owner | | Design disaster recovery strategies based on defined RTO/RPO. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | Cloud architect, Data engineer | | Provision disaster recovery clusters and configurations. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | DevOps engineer, Network administrator, Cloud architect | | Test the CDC pipeline for disaster recovery. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/replicate-mainframe-databases-to-aws-by-using-precisely-connect.html) | App owner, Data engineer, Cloud architect | ## Related resources **AWS resources** + [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html) + [Condition expressions with Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html) + [Amazon EKS](https://docs.aws.amazon.com/eks/index.html) + [Amazon EKS Anywhere](https://anywhere.eks.amazonaws.com/docs/) + [Amazon ElasticCache](https://docs.aws.amazon.com/elasticache/index.html) + [Amazon Keyspaces](https://docs.aws.amazon.com/keyspaces/?icmpid=docs_homepage_databases) + [Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html) + [Amazon RDS and Amazon Aurora](https://docs.aws.amazon.com/rds/index.html) + [Amazon VPC](https://docs.aws.amazon.com/vpc/index.html) **Precisely Connect resources** + [Precisely Connect Overview](https://www.precisely.com/product/precisely-connect/connect) + [Change Data Capture with Precisely Connect](https://help.precisely.com/r/Connect-CDC-SQData/4.1/en-US/Connect-CDC-SQData-Installation/Connect-CDC-SQData-Architecture) **Confluent resources** + [Multi-Threaded Message Consumption with the Apache Kafka Consumer](https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/) # Schedule jobs for Amazon RDS for PostgreSQL and Aurora PostgreSQL by using Lambda and Secrets Manager *Yaser Raja, Amazon Web Services* ## Summary For on-premises databases and databases that are hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances, database administrators often use the **cron** utility to schedule jobs. For example, a job for data extraction or a job for data purging can easily be scheduled using **cron**. For these jobs, database credentials are typically either hard-coded or stored in a properties file. However, when you migrate to Amazon Relational Database Service (Amazon RDS) or Amazon Aurora PostgreSQL-Compatible Edition, you lose the ability to log in to the host instance to schedule **cron** jobs. This pattern describes how to use AWS Lambda and AWS Secrets Manager to schedule jobs for Amazon RDS for PostgreSQL and Aurora PostgreSQL-Compatible databases after migration. ## Prerequisites and limitations **Prerequisites** + An active AWS account + An Amazon RDS for PostgreSQL or Aurora PostgreSQL-Compatible database **Limitations** + A job must complete within 15 minutes, which is the Lambda function timeout limit. For other limits, see the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/limits.html). + Job code must be written in a [language supported by Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html). ## Architecture **Source technology stack** This stack features jobs written in languages such as Bash, Python, and Java. Database credentials are stored in the properties file, and the job is scheduled using Linux **cron**. **Target technology stack** This stack has a Lambda function that uses the credentials stored in Secrets Manager to connect to the database and to perform the activity. The Lambda function is initiated at the scheduled interval by using Amazon CloudWatch Events. **Target architecture** ![\[CloudWatch event starting a Lambda function that schedules jobs for the RDS DB instance.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8e0d1c90-0599-4909-a800-26a89b87f686/images/61f9ca34-9157-4565-96ba-5234d389ac2a.png) ## Tools + [Amazon CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html) delivers a near real-time stream of system events that describe changes in AWS resources. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams. CloudWatch Events becomes aware of operational changes as they occur. It responds to these operational changes and takes corrective action as necessary, by sending messages to respond to the environment, activating functions, making changes, and capturing state information. You can also use CloudWatch Events to schedule automated actions that self-initiate at certain times using **cron** or **rate** expressions. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time you consume; there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service with zero administration. Lambda runs your code on a high-availability compute infrastructure and manages all the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring, and logging. All you need to do is provide your code in one of the [languages that Lambda supports](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html). + [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you protect secrets for accessing your applications, services, and IT resources. You can easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. Users and applications retrieve secrets by calling Secrets Manager APIs, which eliminates the need to hard-code sensitive information in plain text. Secrets Manager offers secret rotation with built-in integration for Amazon RDS, Amazon Redshift, and Amazon DocumentDB. The service is extensible to other types of secrets, including API keys and OAuth tokens. Secrets Manager enables you to control access to secrets using fine-grained permissions and to audit secret rotation centrally for resources in the AWS Cloud, third-party services, and on premises. ## Epics ### Store database credentials in Secrets Manager | Task | Description | Skills required | | --- | --- | --- | | Create a database user for the Lambda function. | It is a good practice to use separate database users for different parts of your application. If a separate database user already exists for your cron jobs, use that. Otherwise, create a new database user. For more information, see [Managing PostgreSQL users and roles](https://aws.amazon.com/blogs/database/managing-postgresql-users-and-roles/) (AWS blog post). | DBA | | Store database credentials as a secret in Secrets Manager. | Follow the instructions in [Create a database secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_database_secret.html) (Secrets Manager documentation). | DBA, DevOps | ### Author the code for the Lambda function | Task | Description | Skills required | | --- | --- | --- | | Choose a programming language supported by Lambda. | For a list of supported languages, see [Lambda runtimes](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) (Lambda documentation). | Developer | | Write the logic to fetch the database credentials from Secrets Manager. | For sample code, see [How to securely provide database credentials to Lambda functions by using AWS Secrets Manager](https://aws.amazon.com/blogs/security/how-to-securely-provide-database-credentials-to-lambda-functions-by-using-aws-secrets-manager/) (AWS blog post). | Developer | | Write the logic to perform the scheduled database activity. | Migrate your existing code for the scheduling job that you're using on premises to the Lambda function. For more information, see [Deploying Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-deploy-functions.html) (Lambda documentation). | Developer | ### Deploy the code and create the Lambda function | Task | Description | Skills required | | --- | --- | --- | | Create the Lambda function deployment package. | This package contains the code and its dependencies. For more information, see [Deployment packages](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html) (Lambda documentation). | Developer | | Create the Lambda function. | In the Lambda console, choose **Create function**, enter a function name, choose the runtime environment, and then choose **Create function**. | DevOps | | Upload the deployment package. | Choose the Lambda function you created to open its configuration. You can write your code directly in the code section or upload your deployment package. To upload your package, go to the **Function code** section, choose the **Code entry type** to upload a .zip file, and then select the package. | DevOps | | Configure the Lambda function per your requirements. | For example, you can set the **Timeout** parameter to the duration you expect your Lambda function to take. For more information, see [Configuring function options](https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html) (Lambda documentation). | DevOps | | Set permissions for the Lambda function role to access Secrets Manager. | For instructions, see [Use secrets in AWS Lambda functions](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html) (Secrets Manager documentation). | DevOps | | Test the Lambda function. | Initiate the Lambda function manually to make sure it works as expected. | DevOps | ### Schedule the Lambda function by using CloudWatch Events | Task | Description | Skills required | | --- | --- | --- | | Create a rule to run your Lambda function on a schedule. | Schedule the Lambda function by using CloudWatch Events. For instructions, see [Schedule Lambda functions using CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html) (CloudWatch Events tutorial). | DevOps | ## Related resources + [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) + [Getting started with Lambda](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html) + [Creating a CloudWatch Events Rule That Triggers on an Event](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Rule.html) + [AWS Lambda Limits](https://docs.aws.amazon.com/lambda/latest/dg/limits.html) + [Query your AWS database from your serverless application](https://aws.amazon.com/blogs/database/query-your-aws-database-from-your-serverless-application/) (blog post) # Send notifications for an Amazon RDS for SQL Server database instance by using an on-premises SMTP server and Database Mail *Nishad Mankar, Amazon Web Services* ## Summary [Database Mail](https://learn.microsoft.com/en-us/sql/relational-databases/database-mail/database-mail?view=sql-server-ver16) (Microsoft documentation) sends email messages, such as notifications or alerts, from a Microsoft SQL Server database by using a Simple Mail Transfer Protocol (SMTP) server. The Amazon Relational Database Service (Amazon RDS) for Microsoft SQL Server documentation provides instructions for using Amazon Simple Email Service (Amazon SES) as the SMTP server for Database Mail. For more information, see [Using Database Mail on Amazon RDS for SQL Server](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.DBMail.html). As an alternative configuration, this pattern explains how to configure Database Mail to send email from an Amazon RDS for SQL Server database (DB) instance by using an on-premises SMTP server as the mail server. ## Prerequisites and limitations **Prerequisites** + An active AWS account + An Amazon RDS DB instance running a Standard or Enterprise edition of SQL Server + The IP address or hostname of the on-premises SMTP server + An inbound [security group rule](https://docs.aws.amazon.com/vpc/latest/userguide/security-group-rules.html#working-with-security-group-rules) that allows connections to the Amazon RDS for SQL Server DB instance from the IP address of the SMTP server + A connection, such as an [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) connection, between your on-premises network and the virtual private cloud (VPC) that contains the Amazon RDS DB instance **Limitations** + Express editions of SQL Server aren't supported. + For more information about limitations, see [Limitations](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.FeatureSupport.Limits) in *Using Database Mail on Amazon RDS for SQL Server* in the Amazon RDS documentation. **Product versions** + Standard and Enterprise editions of [SQL Server versions supported in RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html#SQLServer.Concepts.General.VersionSupport) ## Architecture **Target technology stack** + Amazon RDS for SQL Server database instance + Amazon Route 53 forwarding rule + Database Mail + On-premises SMTP server + Microsoft SQL Server Management Studio (SSMS) **Target architecture** The following image shows the target architecture for this pattern. When an event or action occurs that initiates a notification or alert regarding the database instance, Amazon RDS for SQL Server uses Database Mail to send an email notification. Database Mail uses the on-premises SMTP server to send the email. ![\[Amazon RDS for SQL server using an on-premises SMTP server to send email notifications to users.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e5599724-43cf-4fe1-8c5a-8fca1a424993/images/47efb12f-3505-4a60-ac43-194a176e71c8.png) ## Tools **AWS services** + [Amazon Relational Database Service (Amazon RDS) for Microsoft SQL Server](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html) helps you set up, operate, and scale a SQL Server relational database in the AWS Cloud. + [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) is a highly available and scalable DNS web service. **Other tools** + [Database Mail](https://learn.microsoft.com/en-us/sql/relational-databases/database-mail/database-mail) is a tool that sends e-mail messages, such as notifications and alerts, from the SQL Server Database Engine to users. + [Microsoft SQL Server Management Studio (SSMS)](https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-ssms) is a tool for managing SQL Server, including accessing, configuring, and administering SQL Server components. In this pattern, you use SSMS to run the SQL commands to set up Database Mail on an Amazon RDS for SQL Server DB instance. ## Epics ### Enable network connectivity with the on-premises SMTP server | Task | Description | Skills required | | --- | --- | --- | | Remove Multi-AZ from the RDS DB instance. | If you're using a Multi-Zone RDS DB instance, convert the Multi-AZ instance to a Single-AZ instance. When you have finished configuring Database Mail, you will convert the DB instance back to a Multi-AZ deployment. The Database Mail configuration then works in both the primary and secondary nodes. For instructions, see [Removing Multi-AZ from a Microsoft SQL Server DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServerMultiAZ.html#USER_SQLServerMultiAZ.Removing). | DBA | | Create an allow list for the Amazon RDS endpoint or IP address on the on-premises SMTP server. | The SMTP server is outside of the AWS network. On the on-premises SMTP server, create an allow list that permits the server to communicate with the outbound endpoint or IP address for the Amazon RDS instance or the Amazon Elastic Compute Cloud (Amazon EC2) instance hosted on Amazon RDS. This procedure varies from organization to organization. For more information about the DB instance endpoint, see [Finding the DB instance endpoint and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToMicrosoftSQLServerInstance.html#sqlserver-endpoint). | DBA | | Remove port 25 restrictions. | By default, AWS restricts port 25 on EC2 instances. To remove the port 25 restriction, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.html)[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.html) | General AWS | | Add a Route 53 rule to resolve DNS queries for the SMTP server. | Use Route 53 to resolve DNS queries between your AWS resources and the on-premises SMTP server. You must create a rule that forwards the DNS queries to the SMTP server domain, such as `example.com`. For instructions, see [Creating forwarding rules](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-rules-managing.html#resolver-rules-managing-creating-rules) in the Route 53 documentation. | Network administrator | ### Set up Database Mail on the Amazon RDS for SQL Server DB instance | Task | Description | Skills required | | --- | --- | --- | | Enable Database Mail. | Create a parameter group for Database Mail, set the `database mail xps` parameter to `1`, and then associate the Database Mail parameter group with the target RDS DB instance. For instructions, see [Enabling Database Mail](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.DBMail.html#SQLServer.DBMail.Enable) in the Amazon RDS documentation. Do not proceed to the *Configuring Database Mail* section in these instructions. The configuration for the on-premises SMTP server differs from Amazon SES. | DBA | | Connect to the DB instance. | From a bastion host, use Microsoft SQL Server Management Studio (SSMS) to connect to the Amazon RDS for SQL Server database instance. For instructions, see [Connecting to a DB instance running the Microsoft SQL Server database engine](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToMicrosoftSQLServerInstance.html). If you encounter any errors, see the connection troubleshooting references in the [Related resources](#send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail-resources) section. | DBA | | Create the profile. | In SSMS, enter the following SQL statement to create the Database Mail profile. Replace the following values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.html)For more information about this stored procedure and its arguments, see [sysmail\$1add\$1profile\$1sp](https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sysmail-add-profile-sp-transact-sql) in the Microsoft documentation.
EXECUTE msdb.dbo.sysmail_add_profile_sp| DBA | | Add principals to the profile. | Enter the following SQL statement to add public or private principals to the Database Mail profile. A *principal* is an entity that can request SQL Server resources. Replace the following values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.html)For more information about this stored procedure and its arguments, see [sysmail\$1add\$1principalprofile\$1sp](https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sysmail-add-principalprofile-sp-transact-sql) in the Microsoft documentation.
@profile_name = 'SQL Alerts profile',
@description = 'Profile used for sending outgoing notifications using OM SMTP Server.';
EXECUTE msdb.dbo.sysmail_add_principalprofile_sp| DBA | | Create the account. | Enter the following SQL statement to create the Database Mail account. Replace the following values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.html)For more information about this stored procedure and its arguments, see [sysmail\$1add\$1account\$1sp](https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sysmail-add-account-sp-transact-sql) in the Microsoft documentation.
@profile_name = 'SQL Alerts profile',
@principal_name = 'public',
@is_default = 1 ;
EXECUTE msdb.dbo.sysmail_add_account_sp| DBA | | Add the account to the profile. | Enter the following SQL statement to add the Database Mail account to the Database Mail profile. Replace the following values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.html)For more information about this stored procedure and its arguments, see [sysmail\$1add\$1profileaccount\$1sp](https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sysmail-add-profileaccount-sp-transact-sql) in the Microsoft documentation.
@account_name = 'SQL Alerts account',
@description = 'Database Mail account for sending outgoing notifications.',
@email_address = 'xyz@example.com',
@display_name = 'xyz@example.com',
@mailserver_name = 'test_smtp.example.com',
@port = 25,
@enable_ssl = 1,
@username = 'SMTP-username',
@password = 'SMTP-password';
EXECUTE msdb.dbo.sysmail_add_profileaccount_sp| DBA | | (Optional) Add Multi-AZ to the RDS DB instance. | If you want to add Multi-AZ with Database Mirroring (DBM) or Always On Availability Groups (AGs), see the instructions in [Adding Multi-AZ to a Microsoft SQL Server DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServerMultiAZ.html#USER_SQLServerMultiAZ.Adding). | DBA | ## Related resources + [Using Database Mail on Amazon RDS for SQL Server](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.DBMail.html) (Amazon RDS documentation) + [Working with file attachments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.DBMail.html#SQLServer.DBMail.MAZ) (Amazon RDS documentation) + [Troubleshooting connections to your SQL Server DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToMicrosoftSQLServerInstance.html#USER_ConnectToMicrosoftSQLServerInstance.Troubleshooting) (Amazon RDS documentation) + [Can't connect to Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Troubleshooting.html#CHAP_Troubleshooting.Connecting) (Amazon RDS documentation) # Set up disaster recovery for SAP on IBM Db2 on AWS *Ambarish Satarkar and Debasis Sahoo, Amazon Web Services* ## Summary This pattern outlines the steps to set up a disaster recovery (DR) system for SAP workloads with IBM Db2 as the database platform, running on the Amazon Web Services (AWS) Cloud. The objective is to provide a low-cost solution for providing business continuity in the event of an outage. The pattern uses the [pilot light approach](https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot-light-and-warm-standby/). By implementing pilot light DR on AWS, you can reduce downtime and maintain business continuity. The pilot light approach focuses on setting up a minimal DR environment in AWS, including an SAP system and a standby Db2 database, that is synchronized with the production environment. This solution is scalable. You can extend it to a full-scale disaster recovery environment as needed. ## Prerequisites and limitations **Prerequisites** + An SAP instance running on an Amazon Elastic Compute Cloud (Amazon EC2) instance + An IBM Db2 database + An operating system that is supported by the SAP Product Availability Matrix (PAM) + Different physical database hostnames for production and standby database hosts + An Amazon Simple Storage Service (Amazon S3) bucket in each AWS Region with [Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) enabled **Product versions** + IBM Db2 Database version 11.5.7 or later ## Architecture **Target technology stack** + Amazon EC2 + Amazon Simple Storage Service (Amazon S3) + Amazon Virtual Private Cloud (VPC peering) + Amazon Route 53 + IBM Db2 High Availability Disaster Recovery (HADR) **Target architecture** This architecture implements a DR solution for SAP workloads with Db2 as the database platform. The production database is deployed in AWS Region 1 and a standby database is deployed in a second Region. The standby database is referred to as the DR system. Db2 Database supports multiple standby databases (up to three). It uses Db2 HADR for setting up the DR database and automating log shipping between the production and standby databases. In the event of a disaster that makes Region 1 unavailable, the standby database in the DR Region takes over the production database role. SAP application servers can be built in advance or by using [AWS Elastic Disaster Recovery](https://aws.amazon.com/disaster-recovery/) or an Amazon Machine Image (AMI) to meet the recovery time objective (RTO) requirements. This pattern uses an AMI. Db2 HADR implements a production-standby setup, where production acts as the primary server, and all users are connected to it. All transactions are written to log files, which are transferred to the standby server by using TCP/IP. The standby server updates its local database by rolling forward the transferred log records, which helps to ensure that it is kept in sync with the production server. VPC peering is used so that instances in the production Region and DR Region can communicate with each other. Amazon Route 53 routes end users to internet applications. ![\[Db2 on AWS with cross-Region replication\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/06edfa4c-0827-4d05-95cf-2d2651e74323/images/e77c1e4e-36f3-4af4-89d0-8eec72348f0a.png) 1. [Create an AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html#creating-an-ami) of the application server in Region 1 and [copy the AMI](https://repost.aws/knowledge-center/copy-ami-region) to Region 2. Use the AMI to launch servers in Region 2 in the event of a disaster. 1. Set up Db2 HADR replication between the production database (in Region 1) and the standby database (in Region 2). 1. Change the EC2 instance type to match the production instance in the event of a disaster. 1. In Region 1, `LOGARCHMETH1` is set to `db2remote: S3 path`. 1. In Region 2, `LOGARCHMETH1` is set to `db2remote: S3 path`. 1. Cross-Region Replication is performed between the S3 buckets. ## Tools **AWS services** + [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down. + [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) is a highly available and scalable DNS web service. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. + [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. This pattern uses [VPC peering](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-peering.html). ## Best practices + The network plays a key role in deciding the HADR replication mode. For DR across AWS Regions, we recommend that you use Db2 HADR ASYNC or SUPERASYNC mode. + For more information about replication modes for Db2 HADR, see the [IBM documentation](https://ibm.github.io/db2-hadr-wiki/hadrSyncMode.html#Description_of_the_Modes). + You can use the AWS Management Console or the AWS Command Line Interface (AWS CLI) to [create a new AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html#creating-an-ami) of your existing SAP system. You can then use the AMI to recover your existing SAP system or to create a clone. + [AWS Systems Manager Automation](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html) can help with the common maintenance and deployment tasks of EC2 instances and other AWS resources. + AWS provides multiple native services to monitor and manage your infrastructure and applications on AWS. Services such as Amazon CloudWatch and AWS CloudTrail can be used to monitor your underlying infrastructure and API operations, respectively. For more details, see [SAP on AWS – IBM Db2 HADR with Pacemaker](https://docs.aws.amazon.com/sap/latest/sap-AnyDB/sap-ibm-pacemaker.html). ## Epics ### Prepare the environment | Task | Description | Skills required | | --- | --- | --- | | Check the system and logs. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | AWS administrator, SAP Basis administrator | ### Set up the servers and replication | Task | Description | Skills required | | --- | --- | --- | | Create the SAP and database servers. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html)The rollforward pending state is set by default after the full backup is restored. The rollforward pending state indicates that the database is in the process of being restored and that some changes might need to be applied. For more information, see the [IBM documentation](https://www.ibm.com/docs/en/db2/11.5?topic=commands-rollforward-database). | SAP Basis administrator | | Check the configuration. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | AWS administrator, SAP Basis administrator | | Set up replication from the production DB to the DR DB (using ASYNC mode). | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | SAP Basis administrator | ### Test DR failover tasks | Task | Description | Skills required | | --- | --- | --- | | Plan the production business downtime for the DR test. | Make sure that you plan the required business downtime on production environment for testing the DR failover scenario. | SAP Basis administrator | | Create a test user. | Create a test user (or any test changes) that can be validated in the DR host to confirm log replication after DR failover. | SAP Basis administrator | | On the console, stop the production EC2 instances. | Ungraceful shutdown is initiated in this step to mimic a disaster scenario. | AWS systems administrator | | Scale up the DR EC2 instance to match the requirements. | On the EC2 console, change the instance type in the DR Region.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | SAP Basis Admin | | Initiate takeover. | From the DR system (`host2`), initiate the take-over process and bring up the DR database as the primary.
@profile_name = 'SQL Alerts profile',
@account_name = 'SQL Alerts account',
@sequence_number = 1;
db2 takeover hadr on databaseOptionally, you can set the following parameters to adjust database memory allocation automatically based on the instance type. The `INSTANCE_MEMORY` value can be decided based on the dedicated portion of memory to be allocated to the Db2 database.by force
db2 update db cfg forVerify the change by using the following commands.using INSTANCE_MEMORY IMMEDIATE;
db2 get db cfg for| grep -i DATABASE_MEMORY AUTOMATIC IMMEDIATE;
db2 update db cfg forusing self_tuning_mem ON IMMEDIATE;
db2 get db cfg for| SAP Basis administrator | | Launch the application server for SAP in the DR Region. | Using the AMI that you made of the production system, [launch a new additional application server](https://aws.amazon.com/premiumsupport/knowledge-center/launch-instance-custom-ami/) in the DR Region. | SAP Basis administrator | | Perform validation before starting the SAP application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | AWS administrator, SAP Basis administrator | | Start the SAP application on the DR system. | Start the SAP application on the DR system by using `| grep -i MEMORY
db2 get db cfg for| grep -i self_tuning_mem
sapconrol -nr XX -function StartService| SAP Basis administrator | | Perform SAP validation. | This is performed as a DR test to provide evidence or to check the data replication success to the DR Region. | Test engineer | ### Perform DR failback tasks | Task | Description | Skills required | | --- | --- | --- | | Start the production SAP and database servers. | On the console, start the EC2 instances that host SAP and the database in the production system. | SAP Basis administrator | | Start the production database and set up HADR. | Log in to production system (`host1`) and verify that the DB is in recovery mode by using the following command.
sapconrol -nr XX -function StartSystem
sapconrol -nr YY -function StartService
sapconrol -nr YY -function StartSystem
db2startVerify that the HADR status is `connected`. Replication status should be `peer`.
db2 start HADR on db P3V as standby
db2 connect to
db2pd -dIf the database is not inconsistent and is not at `connected` and `peer` status, a backup and restore might be required to bring the database (on `host1`) in sync with the currently active database (`host2` in the DR Region). In that case, restore the DB backup from the database in the `host2` DR Region to the database in the `host1` production Region. | SAP Basis administrator | | Fail back the database to the production Region. | In a normal business-as-usual scenario, this step is performed in a scheduled downtime. Applications running on the DR system are stopped, and the database is failed back to the production Region (Region 1) to resume operations from the production Region.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | SAP Basis administrator | | Perform validation before starting the SAP application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | AWS administrator, SAP Basis administrator | | Start the SAP application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | SAP Basis administrator | ## Troubleshooting | Issue | Solution | | --- | --- | | Key log files and commands to troubleshoot HADR-related issues | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.html) | | SAP note for troubleshooting HADR issues on Db2 UDB | Refer to SAP [Note 1154013 - DB6: DB problems in HADR environment](https://service.sap.com/sap/support/notes/1154013). (You need SAP portal credentials to access this note.) | ## Related resources + [Disaster recovery approaches for Db2 databases on AWS](https://aws.amazon.com/blogs/architecture/disaster-recovery-approaches-for-db2-databases-on-aws/) (blog post) + [SAP on AWS – IBM Db2 HADR with Pacemaker](https://docs.aws.amazon.com/sap/latest/sap-AnyDB/sap-ibm-pacemaker.html) + [Step by Step Procedure to set up HADR replication between DB2 databases](https://www.ibm.com/support/pages/step-step-procedure-set-hadr-replication-between-db2-databases) + [Db2 HADR Wiki](https://ibm.github.io/db2-hadr-wiki/index.html) ## Additional information Using this pattern, you can set up a disaster recovery system for an SAP system running on the Db2 database. In a disaster situation, business should be able to continue within your defined recovery time objective (RTO) and recovery point objective (RPO) requirements: + **RTO** is the maximum acceptable delay between the interruption of service and restoration of service. This determines what is considered an acceptable time window when service is unavailable. + **RPO** is the maximum acceptable amount of time since the last data recovery point. This determines what is considered an acceptable loss of data between the last recovery point and the interruption of service. For FAQs related to HADR, see [SAP note \$11612105 - DB6: FAQ on Db2 High Availability Disaster Recovery (HADR)](https://launchpad.support.sap.com/#/notes/1612105). (You need SAP portal credentials to access this note.) # Set up a CI/CD pipeline for database migration by using Terraform *Dr. Rahul Sharad Gaikwad, Ashish Bhatt, Aniket Dekate, Ruchika Modi, Tamilselvan P, Nadeem Rahaman, Aarti Rajput, and Naveen Suthar, Amazon Web Services* ## Summary This pattern is about establishing a continuous integration and continuous deployment (CI/CD) pipeline for managing database migrations in a reliable and automated manner. It covers the process of provisioning the necessary infrastructure, migrating data, and customizing schema changes by using Terraform, which is an infrastructure as code (IaC) tool. Specifically, the pattern sets up a CI/CD pipeline to migrate an on-premises Microsoft SQL Server database to Amazon Relational Database Service (Amazon RDS) on AWS. You can also use this pattern to migrate a SQL Server database that's on a virtual machine (VM) or in another cloud environment to Amazon RDS. This pattern addresses the following challenges associated with database management and deployment: + Manual database deployments are time-consuming, error-prone, and lack consistency across environments. + Coordinating infrastructure provisioning, data migrations, and schema changes can be complex and difficult to manage. + Ensuring data integrity and minimizing downtime during database updates is crucial for production systems. This pattern provides the following benefits: + Streamlines the process of updating and deploying database changes by implementing a CI/CD pipeline for database migrations. This reduces the risk of errors, ensures consistency across environments, and minimizes downtime. + Helps improve reliability, efficiency, and collaboration. Enables faster time to market and reduced downtime during database updates. + Helps you adopt modern DevOps practices for database management, which leads to increased agility, reliability, and efficiency in your software delivery processes. ## Prerequisites and limitations **Prerequisites** + An active AWS account + Terraform 0.12 or later installed on your local machine (for instructions, see the [Terraform documentation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)) + Terraform AWS Provider version 3.0.0 or later from HashiCorp (see the [GitHub repository](https://github.com/hashicorp/terraform-provider-aws) for this provider) + Least privilege AWS Identity and Access Management (IAM) policy (see the blog post [Techniques for writing least privilege IAM policies](https://aws.amazon.com/blogs/security/techniques-for-writing-least-privilege-iam-policies/)) ## Architecture This pattern implements the following architecture, which provides the complete infrastructure for the database migration process. ![\[CI/CD pipeline architecture for migrating an on-premises SQL Server database to Amazon RDS on AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/87845d9f-8e6e-4c51-b9ee-9e7833671d05/images/a1e95458-419a-4de9-85ef-b17d8340700a.png) In this architecture: + The source database is a SQL Server database that is on premises, on a virtual machine (VM), or hosted by another cloud provider. The diagram assumes that the source database is in an on-premises data center. + The on-premises data center and AWS are connected through a VPN or AWS Direct Connect connection. This provides secure communications between the source database and the AWS infrastructure. + The target database is an Amazon RDS database that is hosted inside the virtual private cloud (VPC) on AWS with the help of a database provisioning pipeline. + AWS Database Migration Service (AWS DMS) replicates your on-premises database to AWS. It is used to configure the replication of the source database to the target database. The following diagram shows the infrastructure set up with different levels of the database migration process, which involves provisioning, AWS DMS setup, and validation. ![\[CI/CD pipeline details of the migration process from on premises to AWS.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/87845d9f-8e6e-4c51-b9ee-9e7833671d05/images/3aca17e5-6fd7-4317-b578-ab5e485c6efb.png) In this process: + The validation pipeline validates all checks. The integrated pipeline moves to the next step when all necessary validations are complete. + The DB provisioning pipeline consists of various AWS CodeBuild stages that perform Terraform actions on the provided Terraform code for the database. When these steps are complete, it deploys resources in the target AWS account. + The AWS DMS pipeline consists of various CodeBuild stages that perform tests and then provision the AWS DMS infrastructure for performing the migration by using IaC. ## Tools **AWS services and tools** + [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html) is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages. + [AWS CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html) is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. + [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is an object storage service that offers scalability, data availability, security, and performance. + [AWS Database Migration Service (AWS DMS)](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) helps you migrate data stores into the AWS Cloud or between combinations of cloud and on-premises setups. **Other services** + [Terraform](https://www.terraform.io/) is an IaC tool from HashiCorp that helps you create and manage cloud and on-premises resources. **Code repository** The code for this pattern is available in the GitHub [Database Migration DevOps Framework using Terraform samples](https://github.com/aws-samples/aws-terraform-db-migration-framework-samples) repository. ## Best practices + Implement automated tests for your database migration to verify the correctness of schema changes and data integrity. This includes unit tests, integration tests, and end-to-end tests. + Implement a robust backup and restore strategy for your databases, especially before migration. This ensures data integrity and provides a fallback option in case of failures. + Implement a robust rollback strategy to revert database changes in case of failures or issues during migration. This could involve rolling back to a previous database state or reverting individual migration scripts. + Set up monitoring and logging mechanisms to track the progress and status of database migrations. This helps you identify and resolve issues quickly. ## Epics ### Set up your local workstation | Task | Description | Skills required | | --- | --- | --- | | Set up and configure Git on your local workstation. | Install and configure Git on your local workstation by following the instructions in the [Git documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). | DevOps engineer | | Create a project folder and add the files from the GitHub repository. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-ci-cd-pipeline-for-db-migration-with-terraform.html) | DevOps engineer | ### Provision the target architecture | Task | Description | Skills required | | --- | --- | --- | | Update required parameters. | The `ssm-parameters.sh` file stores all required AWS Systems Manager parameters. You can configure these parameters with the custom values for your project.In the `setup/db-ssm-params` folder on your local workstation, open the `ssm-parameters.sh` file and set these parameters before you run the CI/CD pipeline. | DevOps engineer | | Initialize the Terraform configuration. | In the `db-cicd-integration` folder, enter the following command to initialize your working directory that contains the Terraform configuration files:-hadr
terraform init| DevOps engineer | | Preview the Terraform plan. | To create a Terraform plan, enter the following command:
terraform plan -var-file="terraform.sample"Terraform evaluates the configuration files to determine the target state for the declared resources. It then compares the target state against the current state and creates a plan. | DevOps engineer | | Verify the plan. | Review the plan and confirm that it configures the required architecture in your target AWS account. | DevOps engineer | | Deploy the solution. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-ci-cd-pipeline-for-db-migration-with-terraform.html) | DevOps engineer | ### Verify the deployment | Task | Description | Skills required | | --- | --- | --- | | Validate the deployment. | Verify the status of the `db-cicd-integration` pipeline to confirm that the database migration is complete.1. Sign in to the AWS Management Console, and then open the [AWS CodePipeline console](https://console.aws.amazon.com/codesuite/codepipeline/home).2. In the navigation pane, choose **Pipelines**.3. Choose the `db-cicd-integration` pipeline.4. Validate that the pipeline execution has completed successfully. | DevOps engineer | ### Clean up infrastructure after use | Task | Description | Skills required | | --- | --- | --- | | Clean up the infrastructure. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-ci-cd-pipeline-for-db-migration-with-terraform.html) | DevOps engineer | ## Related resources **AWS documentation** + [Getting started with a Terraform product](https://docs.aws.amazon.com/servicecatalog/latest/adminguide/getstarted-Terraform.html) **Terraform documentation** + [Terraform installation](https://learn.hashicorp.com/tutorials/terraform/install-cli) + [Terraform backend configuration](https://developer.hashicorp.com/terraform/language/backend) + [Terraform AWS Provider documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) # Set up a Microsoft SQL Server failover cluster on Amazon EC2 by using FSx for Windows File Server *Sweta Krishna and Ramesh Babu Donti, Amazon Web Services* ## Summary Microsoft SQL Server Standard edition with a failover cluster instance (FCI) can provide a more cost-effective alternative to SQL Server Enterprise. Setting up SQL FCI requires shared file storage between nodes, and [Amazon FSx for Windows File Server](https://aws.amazon.com/fsx/windows/) provides fully managed storage that automatically replicates synchronously across Availability Zones. Amazon FSx reduces storage costs by using built-in data deduplication for general-purpose file shares, which eliminates the need to maintain third-party solutions. Amazon FSx also supports the following: + Pay only for what you use with no upfront fees or commitments. + Set up FCI manually with Amazon FSx as your shared storage. + Use Amazon FSx as the file share witness for your SQL cluster. + Amazon FSx for Windows File Server supports Server Message Block (SMB) 3.0 for continuously available file shares, making it suitable for SQL Server FCI deployments. ## Prerequisites and limitations **Prerequisites** + Active [AWS account](https://aws.amazon.com/account/). + Permissions to create and manage Amazon Virtual Private Cloud (Amazon VPC) resources, Amazon Elastic Compute Cloud (Amazon EC2) instance, security groups, and AWS Identity and Access Management (IAM) roles. + AWS Managed Microsoft AD or your own on-premises Active Directory. + An Active Directory domain user with [necessary permission](https://learn.microsoft.com/en-us/windows-server/failover-clustering/configure-failover-cluster-accounts) to set up a failover cluster. + Security group rules for SQL Server FCI and [Microsoft Active Directory ports](https://docs.aws.amazon.com/whitepapers/latest/access-workspaces-with-access-cards/ip-address-and-port-requirements.html) for secure hybrid connectivity. + A [service account](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-windows-service-accounts-and-permissions?view=sql-server-ver16#sql-server-failover-cluster-instance) in Active Directory for SQL Server that’s configured with appropriate permissions across SQL nodes. + Amazon FSx for Windows File Server in a failover cluster. + SQL Server installation binaries. **Limitations** + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas page](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service. **Product versions** + Amazon EC2 for Windows Server 2012 R2 or later + Amazon FSx for Windows File Server with all current Windows Server versions + Amazon FSx for NetApp ONTAP as an alternative for shared storage + SQL Server 2012/2016/2019/2022 ## Architecture **Technology stack** + Amazon EC2 + Amazon FSx for Windows File Server + Amazon VPC + AWS Directory Service + AWS Systems Manager + IAM **Target architecture** The following diagram shows the high-level architecture of Microsoft SQL Server FCI on Amazon EC2 using Amazon FSx for Windows File Server. ![\[Architecture diagram for Microsoft Server FCI on Amazon EC2 using Amazon FSx for Windows File Server.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/208bf64a-8fef-4019-944a-723372450885/images/ba0c9169-9536-41c3-ae8e-7264dcc3e1ad.png) **Network infrastructure** + Amazon VPC provides a network container that spans three Availability Zones. + Private subnets provide isolated subnets in each Availability Zones for deploying resources. **Compute layer** + Amazon EC2 contains an SQL Server cluster node 1, deployed in Availability Zone 1 as part of the Windows Server Failover Cluster (WSFC). + Amazon EC2 contains an SQL Server cluster node 2, deployed in Availability Zone 2 as part of the WSFC. + The WSFC cluster connects both SQL Server nodes for failover capability. **Storage layer for Amazon FSx for Windows File Server** **Multi-AZ FSx deployment (spanning Availability Zones 1 and 2)** + A primary FSx file system in Availability Zone 1 hosts active SQL Server data and log files. + A secondary FSx file system in Availability Zone 2 provides automatic failover capability. + A shared SMB file share (\$1\$1fsx.domain\$1sqlshare), accessible by both cluster nodes for SQL Server databases. **Single-AZ FSx deployment (in AZ3)** + Amazon FSx file server witness in Availability Zone 3 serves as the cluster quorum witness. + The file share witness (`\\fsx.domain\witness`) maintains the cluster quorum and prevents split-brain scenarios. **Directory services** + AWS Managed Microsoft AD provides Windows authentication and domain services that are required for cluster functionality. **High availability features** + Multi-AZ components provide fault tolerance across Availability Zones. + FSx standby file server provides automatic failover if the primary server fails. + File share witness provides cluster quorum management in Availability Zone 3 to help ensure proper cluster operation during failures. + Domain is integrated with AWS Managed Microsoft AD for seamless Windows authentication. ## Tools **AWS services** + [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down. + [Amazon FSx](https://docs.aws.amazon.com/fsx/?id=docs_gateway) provides file systems that support industry-standard connectivity protocols and offer high availability and replication across AWS Regions. + [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. + [AWS Directory Service for Microsoft Active Directory](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html) enables your directory-aware workloads and AWS resources to use Microsoft Active Directory in the AWS Cloud. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. + [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) helps you manage your applications and infrastructure running in the AWS Cloud. It simplifies application and resource management, shortens the time to detect and resolve operational problems, and helps you manage your AWS resources securely at scale. ## Best practices + Place database instances in private subnets to protect them from being publicly accessible from the internet while still allowing them to connect to AWS services and perform updates. + To use PowerShell to administer your Amazon FSx for Windows File Server, see [Administering FSx for Windows file systems](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/administering-file-systems.html). ## Epics ### Create and configure Amazon EC2 nodes for SQL Server | Task | Description | Skills required | | --- | --- | --- | | Add names and tags. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Choose a Windows AMI. | Choose an Amazon Machine Image (AMI) for Windows that meets your SQL Server requirements. | DBA | | Select an instance type. | Select an Amazon EC2 instance type that meets your requirements. | DBA | | Use a key pair. | You can use a key pair to securely connect to your instance. Ensure that you have access to the selected key pair before you launch the instance. | DBA | | Configure network settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Configure advanced network settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Configure storage. | Configure the required total storage and choose the required storage type. | DBA | | Configure advanced details and launch the instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Create node 2. | Repeat these steps to create and configure node 2. | DBA | ### Install and configure Windows Server failover cluster on nodes 1 and 2 | Task | Description | Skills required | | --- | --- | --- | | Log in to node 1. | Log in to the Windows Amazon EC2 instance as an administrator. | DBA | | Install FCI features on node 1. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html)
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Log in to node 2. | Log in to the Windows Amazon EC2 instance as an administrator. | DBA | | Install FCI features on node 2. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html)
Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Add nodes to the cluster. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Bring the cluster online. | To bring the cluster online, update the static IP addresses of both nodes:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Validate the cluster. | Navigate to **Failover cluster manager** and verify that the cluster core resources are online. | DBA | ### Install SQL Server on nodes 1 and 2 | Task | Description | Skills required | | --- | --- | --- | | Log in to the server. | Log in to the Amazon EC2 instance as an administrator. | DBA | | Mount the SQL binaries. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | | | Add node 2 to the failover cluster. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | ### Configure the Amazon FSx file share witness | Task | Description | Skills required | | --- | --- | --- | | Configure quorum settings. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/microsoft-sql-failover-cluster-on-amazon-ec2.html) | DBA | | Retrieve DNS details. | In the Amazon FSx console, choose **Managed AD** and then **Attach**. The DNS should have the following format: `\\example.example.net\share` | DBA | | Configure the file share witness. | Choose **Amazon FSx file share path** and then **Finish**. | DBA | ## Related resources **AWS resources** + [Amazon FSx for Windows File Server](https://www.youtube.com/watch?v=IMDWTIShlyI) (video) + [Deep dive on Amazon FSx for Windows File Server](https://www.youtube.com/watch?v=_x_Geur93oc) (video) + [How to deploy a SQL Server failover cluster with Amazon EBS Multi-Attach on Windows Server](https://aws.amazon.com/blogs/modernizing-with-aws/how-to-deploy-a-sql-server-failover-cluster-with-amazon-ebs-multi-attach-on-windows-server/) (AWS blog post) + [Simplify your Microsoft SQL Server high availability deployments using Amazon FSx for Windows File Server](https://aws.amazon.com/blogs/storage/simplify-your-microsoft-sql-server-high-availability-deployments-using-amazon-fsx-for-windows-file-server/) (AWS blog post) + [SQL Server high availability deployments using Amazon FSx for NetApp ONTAP](https://aws.amazon.com/blogs/modernizing-with-aws/sql-server-high-availability-amazon-fsx-for-netapp-ontap/) (AWS blog post) + [Using FSx for Windows File Server with Microsoft SQL Server](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/sql-server.html) + [What is FSx for Windows File Server?](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/what-is.html) **Other resources** + [Create a failover cluster](https://learn.microsoft.com/en-us/windows-server/failover-clustering/create-failover-cluster?pivots=windows-admin-center) ## Additional information **Configuring the file share witness** Ensure that you’re connected to the file system from both nodes by adding rules in the Amazon FSx security group that allow inbound connections. The SMB port should be allowed. For example, if the DNS name is `\\example.example.com\share`, use `\\example.example.com\share`. Use the same value for the file share witness in the Always On availability cluster. Complete the following steps to configure the file share witness: 1. Use RDP to connect to your Amazon EC2 instance. 1. Navigate to **Failover cluster manager**. 1. Open the context (right-click) menu and choose **More actions**. 1. Choose **Configure cluster quorum settings**. 1. Choose **Next**. 1. Select **Quorum configuration** and configure a file share witness. 1. Provide the DNS name. 1. Review the summary and then choose **Finish**. The file share witness should be online in the **Cluster core** resources section. # Set up an HA/DR architecture for Oracle E-Business Suite on Amazon RDS Custom with an active standby database *Simon Cunningham, Jaydeep Nandy, and Nitin Saxena, Amazon Web Services* ## Summary This pattern describes how you can architect your Oracle E-Business solution on Amazon Relational Database Service (Amazon RDS) Custom for high availability (HA) and disaster recovery (DR) by setting up an Amazon RDS Custom read replica database in another Amazon Web Services (AWS) Availability Zone and converting it to an active standby database. The creation of the Amazon RDS Custom read replica is fully automated through the AWS Management Console. This pattern doesn’t discuss the steps for adding additional application tiers and shared file systems, which can also be part of an HA/DR architecture. For more information about those topics, see the following Oracle Support Notes: 1375769.1, 1375670.1, and 1383621.1 (section 5, *Advanced Cloning Options*). (Access requires an [Oracle Support](https://support.oracle.com/portal/) account.) To migrate E-Business Suite system to a single-tier, Single-AZ architecture on Amazon Web Services (AWS), see the pattern [Migrate Oracle E-Business Suite to Amazon RDS Custom](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-e-business-suite-to-amazon-rds-custom.html). Oracle E-Business Suite is an Enterprise Resource Planning (ERP) solution for automating enterprise-wide processes such as financials, human resources, supply chains, and manufacturing. It has a three-tier architecture: client, application, and database. Previously, you had to run your E-Business Suite database on a self-managed [Amazon Elastic Compute Cloud (Amazon EC2) instance,](https://aws.amazon.com/ec2/) but you can now benefit from [Amazon RDS Custom](https://aws.amazon.com/rds/custom/). ## Prerequisites and limitations **Prerequisites** + An existing E-Business Suite installation on Amazon RDS Custom; see the pattern [Migrate Oracle E-Business Suite to Amazon RDS Custom](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-e-business-suite-to-amazon-rds-custom.html) + If you want to change the read replica to read-only and use it to offload reporting to the standby, an [Oracle Active Data Guard database license](https://www.oracle.com/corporate/pricing/) (see the *Oracle Technology Commercial Price List*) **Limitations** + Limitations and unsupported configurations for [Oracle databases on Amazon RDS Custom](https://docs.amazonaws.cn/en_us/AmazonRDS/latest/UserGuide/custom-reqs-limits.html#custom-reqs-limits.limits) + Limitations associated with [Amazon RDS Custom for Oracle read replicas](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-rr.html#custom-rr.limitations) **Product versions** For Oracle Database versions and instance classes supported by Amazon RDS Custom, see [Requirements and limitations for Amazon RDS Custom for Oracle](https://docs.amazonaws.cn/en_us/AmazonRDS/latest/UserGuide/custom-reqs-limits.html). ## Architecture The following diagram illustrates a representative architecture for E-Business Suite on AWS that includes multiple Availability Zones and application tiers in an active/passive setup. The database uses an Amazon RDS Custom DB instance and Amazon RDS Custom read replica. The read replica uses Active Data Guard to replicate to another Availability Zone. You can also use the read replica to offload read traffic on the primary database and for reporting purposes. ![\[Multi-AZ architecture for Oracle E-Business Suite on AWS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a17947e8-56b1-4d92-91df-096c02ff4c19/images/ffdaa2d4-123b-44a0-8d52-b1352a4eee44.png) For more information, see [Working with read replicas for Amazon RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-rr.html) in the Amazon RDS documentation. The Amazon RDS Custom read replica is created by default as mounted. However, if you want to offload some of your read-only workloads to the standby database to reduce the load on your primary database, you can manually change the mode of mounted replicas to read-only by following the steps in the [Epics](#set-up-an-ha-dr-architecture-for-oracle-e-business-suite-on-amazon-rds-custom-with-an-active-standby-database-epics) section. A typical use case for this would be to run your reports from the standby database. Changing to read-only requires an active standby database license. When you create a read replica on AWS, the system uses Oracle Data Guard broker under the covers. This configuration is automatically generated and set up in Maximum Performance mode as follows: ``` DGMGRL> show configuration Configuration - rds_dg Protection Mode: MaxPerformance Members: vis_a - Primary database vis_b - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS (status updated 58 seconds ago) ``` ## Tools **AWS services** + [Amazon RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/working-with-custom-oracle.html) is a managed database service for legacy, custom, and packaged applications that require access to the underlying operating system and database environment. It automates database administration tasks and operations while making it possible for you, as a database administrator, to access and customize your database environment and operating system. **Other tools** + Oracle Data Guard is a tool that helps you create and manage Oracle standby databases. This pattern uses Oracle Data Guard to set up an active standby database on Amazon RDS Custom. ## Epics ### Create a read replica | Task | Description | Skills required | | --- | --- | --- | | Create a read replica of the Amazon RDS Custom DB instance. | To create a read replica, follow the instructions in the [Amazon RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) and use the Amazon RDS Custom DB instance you created (see the [Prerequisites](#set-up-an-ha-dr-architecture-for-oracle-e-business-suite-on-amazon-rds-custom-with-an-active-standby-database-prereqs) section) as the source database.By default, the Amazon RDS Custom read replica is created as a physical standby and is in the mounted state. This is intentional to ensure compliance with the Oracle Active Data Guard license. Follow the next steps to convert the read replica to read-only mode. | DBA | ### Change the read replica to a read-only active standby | Task | Description | Skills required | | --- | --- | --- | | Connect to the Amazon RDS Custom read replica. | Use the following commands to convert your physical standby database to an active standby database. These commands require an Oracle active standby license. To get a license, contact your Oracle representative.
$ sudo su - rdsdb| DBA | | Start media recovery with real-time log apply. | To enable the real-time log apply feature, use the following commands. These convert and validate the standby (read replica) as an active standby database, so you can connect and run read-only queries.
-bash-4.2$ sql
SQL> select process,status,sequence# from v$managed_standby;
PROCESS STATUS SEQUENCE#
--------- ------------ ----------
ARCH CLOSING 3956
ARCH CONNECTED 0
ARCH CLOSING 3955
ARCH CLOSING 3957
RFS IDLE 0
RFS IDLE 3958
MRP0 APPLYING_LOG 3958
SQL> select name, database_role, open_mode from v$database;
NAME DATABASE_ROLE OPEN_MODE
--------- ---------------- --------------------
VIS PHYSICAL STANDBY MOUNTED
SQL> alter database recover managed standby database cancel;
Database altered.
Open the standby database
SQL> alter database open;
Database altered.
SQL> select name, database_role, open_mode from v$database;
NAME DATABASE_ROLE OPEN_MODE
--------- ---------------- --------------------
VIS PHYSICAL STANDBY READ ONLY
SQL> alter database recover managed standby database using current logfile disconnect from session;| DBA | | Check the database status. | To check the status of the database, use the following command.
Database altered
SQL> select name, database_role, open_mode from v$database;| DBA | | Check redo apply mode. | To check redo apply mode, use the following command.
NAME DATABASE_ROLE OPEN_MODE
--------- ---------------- --------------------
VIS PHYSICAL STANDBY READ ONLY WITH APPLY
SQL> select process,status,sequence# from v$managed_standby;| DBA | ## Related resources + [Migrate Oracle E-Business Suite to Amazon RDS Custom](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-oracle-e-business-suite-to-amazon-rds-custom.html) (AWS Prescriptive Guidance) + [Working with Amazon RDS Custom](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-custom.html) (Amazon RDS documentation) + [Working with read replicas for Amazon RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-rr.html) (Amazon RDS documentation) + [Amazon RDS Custom for Oracle – New Control Capabilities in Database Environment](https://aws.amazon.com/blogs/aws/amazon-rds-custom-for-oracle-new-control-capabilities-in-database-environment/) (AWS News blog) + [Migrating Oracle E-Business Suite on AWS](https://d1.awsstatic.com/whitepapers/migrate-oracle-e-business-suite.pdf) (AWS whitepaper) + [Oracle E-Business Suite architecture on AWS](https://docs.aws.amazon.com/whitepapers/latest/overview-oracle-e-business-suite/oracle-e-business-suite-architecture-on-aws.html) (AWS whitepaper) # Stream data from IBM Db2, SAP, Sybase, and other databases to MongoDB Atlas on AWS *Battulga Purevragchaa and Igor Alekseev, Amazon Web Services* *Babu Srinivasan, MongoDB* ## Summary This pattern describes the steps for migrating data from IBM Db2 and other databases such as mainframe databases and Sybase to MongoDB Atlas on the AWS Cloud. It uses [AWS Glue](https://aws.amazon.com/glue/) to help accelerate the data migration to MongoDB Atlas. The pattern accompanies the guide [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/) on the AWS Prescriptive Guidance website. It provides the implementation steps for one of the migration scenarios that are discussed in that guide. For additional migration scenarios, see the following patterns on the AWS Prescriptive Guidance website: + [Migrate a self-hosted MongoDB environment to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-a-self-hosted-mongodb-environment-to-mongodb-atlas-on-the-aws-cloud.html) + [Migrate relational databases to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-relational-database-to-mongodb-atlas.html) The pattern is intended for [AWS Managed Services Partners](https://aws.amazon.com/managed-services/partners/) and AWS users. ## Prerequisites and limitations **Prerequisites** + A source database such as SAP, Sybase, IBM Db2, and others to migrate to MongoDB Atlas. + Familiarity with databases such as SAP, Sybase, IBM Db2, MongoDB Atlas, and AWS services. **Product versions** + MongoDB version 5.0 or later. ## Architecture The following diagram illustrates batch data load and data streaming by using AWS Glue Studio, Amazon Kinesis Data Streams, and MongoDB Atlas. This reference architecture uses AWS Glue Studio to create extract, transform, and load (ETL) pipelines to migrate data to MongoDB Atlas. An AWS Glue crawler integrates with MongoDB Atlas to facilitate data governance. The data can be either ported in batch or streamed to MongoDB Atlas by using Amazon Kinesis Data Streams. **Batch data load** ![\[Migrating data to MongoDB Atlas in batch mode.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/805a376f-35f4-44cc-b4b0-8bf4d95c1e5d/images/68d87202-95ba-4e2a-9b3b-27dd6db6165e.png) For more information about the batch data migration, see the AWS blog post [Compose your ETL jobs for MongoDB Atlas with AWS Glue](https://aws.amazon.com/blogs/big-data/compose-your-etl-jobs-for-mongodb-atlas-with-aws-glue/). **Data streaming** ![\[Migrating data to MongoDB Atlas in data streaming mode.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/805a376f-35f4-44cc-b4b0-8bf4d95c1e5d/images/b007a116-f463-418f-9721-647d80177e3b.png) For MongoDB Atlas reference architectures that support different usage scenarios, see [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/architecture.html) on the AWS Prescriptive Guidance website. ## Tools ● [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) is a fully managed ETL service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams. ● [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/) helps you collect and process large streams of data records in real time. ● [MongoDB Atlas](https://www.mongodb.com/atlas) is a fully managed database as a service (DbaaS) for deploying and managing MongoDB databases in the cloud. ## Best practices For guidelines, see [Best Practices Guide for MongoDB](https://github.com/mongodb-partners/mongodb_atlas_as_aws_bedrock_knowledge_base/blob/main/data/MongoDB_Best_Practices_Guide.pdf) in the MongoDB GitHub repository. ## Epics ### Discovery and assessment | Task | Description | Skills required | | --- | --- | --- | | Determine the cluster size. | Estimate the working set size by using the information from `db.stats()` for the total index space. Assume that a percentage of your data space will be accessed frequently. Or, you can estimate your memory requirements based on your assumptions. This task should take approximately one week. For more information and examples for this and the other stories in this epic, see the links in the [Related resources](#stream-data-from-ibm-db2-to-mongodb-atlas-resources) section. | MongoDB DBA, Application architect | | Estimate network bandwidth requirements. | To estimate your network bandwidth requirements, multiply the average document size by the number of documents served per second. Consider the maximum traffic that any node on your cluster will bear as the basis. To calculate downstream data transfer rates from your cluster to client applications, use the sum of the total documents returned over a period of time. If your applications read from secondary nodes, divide this number of total documents by the number of nodes that can serve read operations. To find the average document size for a database, use the `db.stats().avgObjSize` command. This task will typically take one day. | MongoDB DBA | | Select the Atlas tier. | Follow the instructions in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/manage-clusters/) to select the correct Atlas cluster tier. | MongoDB DBA | | Plan for cutover. | Plan for application cutover. | MongoDB DBA, Application architect | ### Set up a new MongoDB Atlas environment on AWS | Task | Description | Skills required | | --- | --- | --- | | Create a new MongoDB Atlas cluster on AWS. | In MongoDB Atlas, choose **Build a Cluster**, and select AWS as the cloud provider. | MongoDB DBA | | Select AWS Regions and global cluster configuration. | Select from the list of available AWS Regions for your Atlas cluster. Configure global clusters if required. | MongoDB DBA | | Select the cluster tier. | Select your preferred cluster tier. Your tier selection determines factors such as memory, storage, and IOPS specification. | MongoDB DBA | | Configure additional cluster settings. | Configure additional cluster settings such as MongoDB version, backup, and encryption options. For more information about these options, see the [Related resources](#stream-data-from-ibm-db2-to-mongodb-atlas-resources) section. | MongoDB DBA | ### Configure security and compliance | Task | Description | Skills required | | --- | --- | --- | | Configure the access list. | To connect to the Atlas cluster, you must add an entry to the [project’s access list](https://www.mongodb.com/docs/atlas/setup-cluster-security/#configure-security-features-for-clusters). Atlas uses Transport Layer Security (TLS) / Secure Sockets Layer (SSL) to encrypt the connections to the virtual private cloud (VPC) for your database. To set up the access list for the project and for more information about the stories in this epic, see the links in the [Related resources](#stream-data-from-ibm-db2-to-mongodb-atlas-resources) section. | MongoDB DBA | | Authenticate and authorize users. | You must create and authenticate the database users who will access the MongoDB Atlas clusters. To access the clusters in a project, users must belong to that project, and they can belong to multiple projects. You can also enable authorization with AWS Identity and Access Management (IAM). For more information, see [Set Up Authentication with IAM](https://www.mongodb.com/docs/atlas/security/aws-iam-authentication/#set-up-authentication-with-aws-iam) in the MongoDB documentation. | MongoDB DBA | | Create custom roles. | (Optional) Atlas supports creating [custom roles](https://www.mongodb.com/docs/atlas/reference/custom-role-actions/) if the built-in Atlas database user privileges don’t cover your desired set of privileges. | MongoDB DBA | | Set up VPC peering. | (Optional) Atlas supports [VPC peering](https://www.mongodb.com/docs/atlas/security-vpc-peering/#set-up-a-network-peering-connection) with other AWS VPCs. | MongoDB DBA | | Set up an AWS PrivateLink endpoint. | (Optional) You can set up private endpoints on AWS by using [AWS PrivateLink](https://www.mongodb.com/docs/atlas/security-private-endpoint/). | MongoDB DBA | | Enable two-factor authentication. | (Optional) Atlas supports two-factor authentication (2FA) to help users control access to their Atlas accounts. | MongoDB DBA | | Set up user authentication and authorization with LDAP. | (Optional) Atlas supports performing user authentication and authorization with Lightweight Directory Access Protocol (LDAP). | MongoDB DBA | | Set up unified AWS access. | (Optional) Some Atlas features, including Atlas Data Lake and encryption at rest using customer key management, use IAM roles for authentication. | MongoDB DBA | | Set up encryption at rest by using AWS KMS. | (Optional) Atlas supports using AWS Key Management Service (AWS KMS) to encrypt storage engines and cloud provider backups. | MongoDB DBA | | Set up CSFLE. | (Optional) Atlas supports [client-side field-level encryption (CSFLE)](https://www.mongodb.com/docs/upcoming/core/csfle/#client-side-field-level-encryption), including automatic encryption of fields. | MongoDB DBA | ### Migrate data | Task | Description | Skills required | | --- | --- | --- | | Launch your target replica set in MongoDB Atlas. | Launch your target replica set in MongoDB Atlas. In Atlas Live Migration Service, choose **I'm ready to migrate**. | MongoDB DBA | | Establish the connection of AWS Glue with MongoDB Atlas. | Use an AWS Glue crawler to connect AWS Glue with MongoDB Atlas (target database). This step helps prepare the target environment for migration. For more information, see the [AWS Glue documentation](https://docs.aws.amazon.com/glue/latest/dg/console-connections.html). | MongoDB DBA | | Establish the connection of AWS Glue with the source database or source stream. | This helps prepare the target environment for migration. | MongoDB DBA | | Set up the data transformation. | Configure the transformation logic to migrate the data from the legacy structured schema to the flexible schema of MongoDB. | MongoDB DBA | | Migrate the data. | Schedule the migration in AWS Glue Studio. | MongoDB DBA | ### Configure operational integration | Task | Description | Skills required | | --- | --- | --- | | Connect to the cluster. | Connect to the MongoDB Atlas cluster. | App developer | | Interact with data. | Interact with cluster data. | App developer | | Monitor the clusters. | Monitor your MongoDB Atlas clusters. | MongoDB DBA | | Back up and restore data. | Back up and restore cluster data. | MongoDB DBA | ## Troubleshooting | Issue | Solution | | --- | --- | | If you encounter issues | See [Troubleshooting](https://github.com/mongodb/mongodbatlas-cloudformation-resources/tree/master#troubleshooting) in the MongoDB Atlas CloudFormation Resources repository. | ## Related resources All of the following links, unless noted otherwise, go to webpages in the MongoDB documentation. **Migration guide** + [Migrating to MongoDB Atlas on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-mongodb-atlas/) (AWS Prescriptive Guidance) **Discovery and assessment** + [Memory](https://docs.atlas.mongodb.com/sizing-tier-selection/#memory) + [Sizing example with Atlas sample data sets](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#example--the-service-sample-data-sets) + [Sizing example for mobile applications](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#example--mobile-app) + [Network Traffic](https://docs.atlas.mongodb.com/sizing-tier-selection/#network-traffic) + [Cluster Auto-Scaling](https://www.mongodb.com/docs/atlas/sizing-tier-selection/#cluster-auto-scaling) + [Atlas sizing template](https://view.highspot.com/viewer/5f438f47a4dfa042e97130c5) **Configuring security and compliance** + [Configure IP Access List Entries](https://docs.atlas.mongodb.com/security/ip-access-list/) + [Configure Database Users](https://docs.atlas.mongodb.com/security-add-mongodb-users/) + [Configure Access to the Atlas UI](https://docs.atlas.mongodb.com/organizations-projects/) + [Configure Custom Database Roles](https://docs.atlas.mongodb.com/security-add-mongodb-roles) + [Configure Database Users](https://docs.atlas.mongodb.com/security-add-mongodb-users/#atlas-user-privileges) + [Set up a Network Peering Connection](https://docs.atlas.mongodb.com/security-vpc-peering/) + [Learn About Private Endpoints in Atlas](https://docs.atlas.mongodb.com/security-private-endpoint/) + [Manage Your Multi-Factor Authentication Options](https://docs.atlas.mongodb.com/security-two-factor-authentication/) + [Set up User Authentication and Authorization with LDAP](https://docs.atlas.mongodb.com/security-ldaps/) + [Atlas Data Lake](https://docs.mongodb.com/datalake/) + [Encryption at Rest using Customer Key Management](https://docs.atlas.mongodb.com/security-kms-encryption/) + [Methods to assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) (IAM documentation) + [Client-Side Field Level Encryption](https://docs.mongodb.com/manual/core/security-client-side-encryption) + [Automatic Encryption](https://docs.mongodb.com/manual/core/security-automatic-client-side-encryption) + [MongoDB Atlas Security Controls](https://webassets.mongodb.com/_com_assets/cms/MongoDB_Atlas_Security_Controls-v7k3rbhi3p.pdf) + [MongoDB Trust Center](https://www.mongodb.com/cloud/trust) + [Configure Security Features for Clusters](https://docs.atlas.mongodb.com/setup-cluster-security/) **Setting up a new MongoDB Atlas environment on ****AWS** + [Cloud Providers and Regions](https://docs.atlas.mongodb.com/cloud-providers-regions/) + [Manage Global Clusters](https://docs.atlas.mongodb.com/global-clusters/) + [Select Cluster Tier](https://www.mongodb.com/docs/atlas/manage-clusters/#select-cluster-tier) + [Configure Additional Settings](https://docs.atlas.mongodb.com/cluster-additional-settings/) + [Get Started with Atlas](https://docs.atlas.mongodb.com/getting-started/) + [Configure Access to the Atlas UI](https://docs.atlas.mongodb.com/organizations-projects/) **Migrating data** + [Migrate or Import Data](https://www.mongodb.com/docs/atlas/import/) **Monitoring clusters** + [Monitor Your Clusters](https://docs.atlas.mongodb.com/monitoring-alerts/) **Integrating operations** + [Connect to a Cluster](https://docs.atlas.mongodb.com/connect-to-cluster/) + [Interact with Your Data](https://docs.atlas.mongodb.com/data-explorer/) + [Monitor Your Clusters](https://docs.atlas.mongodb.com/monitoring-alerts/) + [Backu Up, Restore, and Archive Data](https://docs.atlas.mongodb.com/backup-restore-cluster/) **GitHub repository** + [Stream data to MongoDB Atlas using AWS Glue](https://github.com/mongodb-partners/Stream_Data_into_MongoDB_AWS_Glue?tab=readme-ov-file#troubleshooting) # Transition roles for an Oracle PeopleSoft application on Amazon RDS Custom for Oracle *sampath kathirvel, Amazon Web Services* ## Summary To run the [Oracle PeopleSoft](https://www.oracle.com/applications/peoplesoft/) enterprise resource planning (ERP) solution on Amazon Web Services (AWS), you can use [Amazon Relational Database Service (Amazon RDS)](https://aws.amazon.com/rds/) or [Amazon RDS Custom for Oracle](https://aws.amazon.com/rds/custom/), which supports legacy, custom, and packaged applications that require access to the underlying operating system (OS) and database environment. For key factors to consider when planning a migration, see [Oracle database migration strategies](https://docs.aws.amazon.com/prescriptive-guidance/latest/migration-oracle-database/strategies.html) in AWS Prescriptive Guidance. This pattern focuses on the steps to perform an Oracle Data Guard switchover, or role transition, for a PeopleSoft application database running on Amazon RDS Custom as the primary database with a read replica database. The pattern includes steps to configure [fast-start failover (FSFO)](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/using-data-guard-broker-to-manage-switchovers-failovers.html#GUID-D26D79F2-0093-4C0E-98CD-224A5C8CBFA4). During this process, the databases in the Oracle Data Guard configuration continue to function in their new roles. Typical use cases for Oracle Data Guard switchover are disaster recovery (DR) drills, scheduled maintenance activities on databases, and [Standby-First Patch Apply](https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/upgrading-patching-downgrading-oracle-data-guard-configuration.html#GUID-A5226768-DB6B-4714-BB9A-0A3EF17A01C8) rolling patches. For more information, see the blog post [Reduce database patching downtime in Amazon RDS Custom](https://aws.amazon.com/blogs/database/reduce-database-patching-downtime-in-amazon-rds-custom-for-oracle-using-oracle-data-guard-standby-first-patch-apply/). ## Prerequisites and limitations **Prerequisites ** + Completion of the [Add HA to Oracle PeopleSoft on Amazon RDS Custom by using a read replica](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/add-ha-to-oracle-peoplesoft-on-amazon-rds-custom-by-using-a-read-replica.html) pattern. **Limitations ** + Limitations and unsupported configurations for [RDS Custom for Oracle](https://docs.amazonaws.cn/en_us/AmazonRDS/latest/UserGuide/custom-reqs-limits.html#custom-reqs-limits.limits) + Limitations associated with [Amazon RDS Custom for Oracle read replicas](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-rr.html#custom-rr.limitations) **Product versions** + For Oracle Database versions supported by Amazon RDS Custom, see [RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RDS_Fea_Regions_DB-eng.Feature.RDSCustom.html#Concepts.RDS_Fea_Regions_DB-eng.Feature.RDSCustom.ora). + For Oracle Database instance classes supported by Amazon RDS Custom, see [DB instance class support for RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-reqs-limits.html#custom-reqs-limits.instances). ## Architecture **Technology stack** + Amazon RDS Custom for Oracle **Target architecture** The following diagram shows an Amazon RDS Custom DB instance and an Amazon RDS Custom read replica. Oracle Data Guard provides role transition during failover for DR. ![\[Oracle Data Guard switchover for a primary RDS Custom DB instance with a read replica database.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/da3b011c-1668-4de4-9079-0982888a74b4/images/4e2a2f3b-b5bd-44b7-9b5a-13a663ee3be6.png) For a representative architecture using Oracle PeopleSoft on AWS, see [Set up a highly available PeopleSoft architecture on AWS](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-a-highly-available-peoplesoft-architecture-on-aws.html). ## Tools **AWS services** + [Amazon RDS Custom for Oracle](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/working-with-custom-oracle.html) is a managed database service for legacy, custom, and packaged applications that require access to the underlying OS and database environment. + [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically. In this pattern, you retrieve the database user passwords from Secrets Manager for `RDS_DATAGUARD` with the secret name `do-not-delete-rds-custom-+<
PROCESS STATUS SEQUENCE#
--------- ------------ ----------
ARCH CLOSING 3956
ARCH CONNECTED 0
ARCH CLOSING 3955
ARCH CLOSING 3957
RFS IDLE 0
RFS IDLE 3958
MRP0 APPLYING_LOG 3958
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY WITH APPLY
-bash-4.2$ dgmgrl RDS_DATAGUARD@RDS_CUSTOM_ORCL_A**CDB**
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Nov 28 20:55:50 2022
Version 19.10.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "ORCL_A"
Connected as SYSDG.
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_a - Primary database
orcl_d - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 59 seconds ago)
DGMGRL>
CDB-bash-4.2$ dgmgrl C##RDS_DATAGUARD@RDS_CUSTOM_RDSCDB_A| DBA | | Verify the instance role. | Open the AWS Management Console, and navigate to the Amazon RDS console. In the database’s **Replication** section, on the **Connectivity & security** tab, verify the instance role for the primary and replica.The primary role should match the Oracle Data Guard primary database, and the replica role should match the Oracle Data Guard physical standby database. | Cloud administrator, DBA | | Perform the switchover. | To perform the switchover, connect to `DGMGRL` from the primary node.**Non-CDB**
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Jan 18 06:13:07 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "RDSCDB_A"
Connected as SYSDG.
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_a - Primary database
rdscdb_b - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 52 seconds ago)
DGMGRL>
DGMGRL> switchover to orcl_d;**CDB**
Performing switchover NOW, please wait...
Operation requires a connection to database "orcl_d"
Connecting ...
Connected to "ORCL_D"
Connected as SYSDG.
New primary database "orcl_d" is opening...
Operation requires start up of instance "ORCL" on database "orcl_a"
Starting instance "ORCL"...
Connected to an idle instance.
ORACLE instance started.
Connected to "ORCL_A"
Database mounted.
Database opened.
Connected to "ORCL_A"
Switchover succeeded, new primary is "orcl_d"
DGMGRL>
DGMGRL> switchover to rdscdb_b| DBA | | Verify the Oracle Data Guard connection. | After switchover, verify the Oracle Data Guard connection from the primary node to `DGMGRL`.**Non-CDB**
Performing switchover NOW, please wait...
New primary database "rdscdb_b" is opening...
Operation requires start up of instance "RDSCDB" on database "rdscdb_a"
Starting instance "RDSCDB"...
Connected to an idle instance.
ORACLE instance started.
Connected to "RDSCDB_A"
Database mounted.
Database opened.
Connected to "RDSCDB_A"
Switchover succeeded, new primary is "rdscdb_b"
DGMGRL> show configuration;**CDB**
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_d - Primary database
orcl_a - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 60 seconds ago)
DGMGRL>
DGMGRL> show configuration lag;
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_d - Primary database
orcl_a - Physical standby database
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 44 seconds ago)
DGMGRL>
DGMGRL> show configuration| DBA | | Verify the instance role on the Amazon RDS console. | After you perform the role switch, the Amazon RDS console shows the new roles under the **Replication** section on the **Connectivity & Security** tab under **Databases**. It might take a few minutes for **Replication state** to update from empty to **Replicating**. | DBA | ### Configure FSFO | Task | Description | Skills required | | --- | --- | --- | | Reset the switchover. | Set the switchover back to the primary node. | DBA | | Install and start the observer. | An observer process is a `DGMGRL` client component, typically running in a different machine from the primary and standby databases. The ORACLE HOME installation for the observer can be an Oracle Client Administrator installation, or you can install either Oracle Database Enterprise Edition or Personal Edition. For more information about observer installation for your database release, see [Installing and Starting the Observer](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/using-data-guard-broker-to-manage-switchovers-failovers.html#GUID-11EF3897-8FCA-4A54-B63B-E8C1668AE21B). To configure high availability for the observer process, you might want to do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transition-roles-for-an-oracle-peoplesoft-application-on-amazon-rds-custom-for-oracle.html)For Oracle 12c Release 2 and later, you can deploy up to three observers. One observer is the primary observer, and the rest are backup observers. When the primary observer fails, one of the backup observers takes the primary role. | DBA | | Connect to DGMGRL from the observer host. | The observer host is configured with `tnsnames.ora` entries for primary and standby database connectivity. You can enable FSFO with maximum performance protection mode as long as data loss is within the [FastStartFailoverLagLimit](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/oracle-data-guard-broker-properties.html) configuration (value in seconds), However, you must use maximum availability protection mode to work to achieve zero data loss (RPO=0).**Non-CDB**
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_b - Primary database
rdscdb_a - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 52 seconds ago)
DGMGRL>
DGMGRL> show configuration lag
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_b - Primary database
rdscdb_a - Physical standby database
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 53 seconds ago)
DGMGRL>
DGMGRL> show configuration;**CDB**
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_a - Primary database
orcl_d - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 58 seconds ago)
DGMGRL> show configuration lag
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_a - Primary database
orcl_d - Physical standby database
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 5 seconds ago)
DGMGRL>
-bash-4.2$ dgmgrl C##RDS_DATAGUARD@RDS_CUSTOM_RDSCDB_A| DBA | | Modify the standby database to be the failover target. | Connect from either the primary node or the observer node to one standby database. (Although your onfiguration could have mulitiple standby databases, you need to connect to only one at this time.)**Non-CDB**
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Jan 18 06:55:09 2023
Version 19.16.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Password:
Connected to "RDSCDB_A"
Connected as SYSDG.
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_a - Primary database
rdscdb_b - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 18 seconds ago)
DGMGRL>
DGMGRL> edit database orcl_a set property FastStartFailoverTarget='orcl_d';**CDB**
Property "faststartfailovertarget" updated
DGMGRL> edit database orcl_d set property FastStartFailoverTarget='orcl_a';
Property "faststartfailovertarget" updated
DGMGRL> show database orcl_a FastStartFailoverTarget;
FastStartFailoverTarget = 'orcl_d'
DGMGRL> show database orcl_d FastStartFailoverTarget;
FastStartFailoverTarget = 'orcl_a'
DGMGRL>
DGMGRL> edit database orcl_a set property FastStartFailoverTarget='rdscdb_b';| DBA | | Configure FastStartFailoverThreshold for the connection to DGMGRL. | The default value is 30 seconds in Oracle 19c, and the minimum value is 6 seconds. A lower value can potentially shorten the recovery time objective (RTO) during the failover. A higher value helps reduce the chance of unnecessary failover transient errors on the primary database.The RDS Custom for Oracle automation framework monitors database health and performs corrective actions every few seconds. Therefore, we recommend setting FastStartFailoverThreshold to a value higher than 10 seconds. The following example configures the threshold value at 35 seconds.**Non-CBD or CDB**
Object "orcl_a" was not found
DGMGRL> edit database rdscdb_a set property FastStartFailoverTarget='rdscdb_b';
Property "faststartfailovertarget" updated
DGMGRL> edit database rdscdb_b set property FastStartFailoverTarget='rdscdb_a';
Property "faststartfailovertarget" updated
DGMGRL> show database rdscdb_a FastStartFailoverTarget;
FastStartFailoverTarget = 'rdscdb_b'
DGMGRL> show database rdscdb_b FastStartFailoverTarget;
FastStartFailoverTarget = 'rdscdb_a'
DGMGRL>
DGMGRL> edit configuration set property FastStartFailoverThreshold=35;| DBA | | Enable FSFO by connecting to DGMGRL from the primary or observer node. | If the database doesn’t have [Flashback Database](https://docs.oracle.com/en/database/oracle/oracle-database/19/rcmrf/FLASHBACK-DATABASE.html#GUID-584AC79A-40C5-45CA-8C63-DED3BE3A4511) enabled, the warning message `ORA-16827` appears. The optional flashback database helps automatically reinstate failed primary databases to a point in time before failover if the [FastStartFailoverAutoReinstate](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/oracle-data-guard-broker-properties.html#GUID-824E97C0-EEB0-4E1B-BD4A-F5AE282CEA28) configuration property is set to `TRUE` (which is the default).**Non-CDB**
Property "faststartfailoverthreshold" updated
DGMGRL> show configuration FastStartFailoverThreshold;
FastStartFailoverThreshold = '35'
DGMGRL>
DGMGRL> enable fast_start failover;**CDB**
Warning: ORA-16827: Flashback Database is disabled
Enabled in Zero Data Loss Mode.
DGMGRL>
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_a - Primary database
Warning: ORA-16819: fast-start failover observer not started
orcl_d - (*) Physical standby database
Warning: ORA-16819: fast-start failover observer not started
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
WARNING (status updated 29 seconds ago)
DGMGRL>
DGMGRL> enable fast_start failover;| DBA | | Start the observer for FSFO monitoring, and verify the status. | You can start the observer before or after you enable FSFO. If FSFO is already enabled, the observer immediately begins monitoring the status and connections to the primary and target standby databases. If FSFO is not enabled, the observer doesn’t start monitoring until after FSFO is enabled.When you start the observer, the primary DB configuration will be displayed without any error messages, as evidenced by the previous `show configuration` command.**Non-CDB**
Warning: ORA-16827: Flashback Database is disabled
Enabled in Zero Data Loss Mode.
DGMGRL> show configuration;
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_a - Primary database
Warning: ORA-16819: fast-start failover observer not started
rdscdb_b - (*) Physical standby database
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
WARNING (status updated 11 seconds ago)
DGMGRL>
DGMGRL> start observer;**CDB**
[W000 2022-12-01T06:16:51.271+00:00] FSFO target standby is orcl_d
Observer 'ip-10-0-1-89' started
[W000 2022-12-01T06:16:51.352+00:00] Observer trace level is set to USER
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_a - Primary database
orcl_d - (*) Physical standby database
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
SUCCESS (status updated 56 seconds ago)
DGMGRL>
DGMGRL> show observer
Configuration - rds_dg
Primary: orcl_a
Active Target: orcl_d
Observer "ip-10-0-1-89" - Master
Host Name: ip-10-0-1-89
Last Ping to Primary: 1 second ago
Last Ping to Target: 1 second ago
DGMGRL>
DGMGRL> start observer;| DBA | | Verify the failover. | In this scenario, a failover test can be performed by manually stopping the primary EC2 instance. Before stopping the EC2 instance, use the `tail` command to monitor the observer log file based on your configuration. Use `DGMGRL` to log in to standby database `orcl_d` with user `RDS_DATAGUARD`, and check the Oracle Data Guard status. It should show that `orcl_d` is the new primary database.In this failover testing scenario, `orcl_d` is the non-CDB database.Before failover, the flashback database has been enabled on `orcl_a`. After the former primary database returns online and starts in `MOUNT` state, the observer reinstates it into a new standby database. The reinstated database acts as the FSFO target for the new primary database. You can verify the details in observer logs.
Succeeded in opening the observer file "/home/oracle/fsfo_ip-10-0-1-56.dat".
[W000 2023-01-18T07:31:32.589+00:00] FSFO target standby is rdscdb_b
Observer 'ip-10-0-1-56' started
The observer log file is '/home/oracle/observer_ip-10-0-1-56.log'.
DGMGRL> show configuration
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
rdscdb_a - Primary database
rdscdb_b - (*) Physical standby database
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
SUCCESS (status updated 12 seconds ago)
DGMGRL>
DGMGRL> show observer;
Configuration - rds_dg
Primary: rdscdb_a
Active Target: rdscdb_b
Observer "ip-10-0-1-56" - Master
Host Name: ip-10-0-1-56
Last Ping to Primary: 1 second ago
Last Ping to Target: 2 seconds ago
DGMGRL>
DGMGRL> show configurationThe following shows example output in `observer.log`.
Configuration - rds_dg
Protection Mode: MaxAvailability
Members:
orcl_d - Primary database
Warning: ORA-16824: multiple warnings, including fast-start failover-related warnings, detected for the database
orcl_a - (*) Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: Enabled in Zero Data Loss Mode
Configuration Status:
WARNING (status updated 25 seconds ago)
DGMGRL>
$ tail -f /tmp/observer.log| DBA | ### Configure connectivity between the Oracle Peoplesoft application and the database | Task | Description | Skills required | | --- | --- | --- | | Create and start the service in the primary database. | You can avoid application configuration changes during a role transition by using a TNS entry that contains both the primary and standby database endpoints in the configuration. You can define two role-based database services to support both read/write and read-only workloads. In the following example, `orcl_rw` is the read/write service that’s active on the primary database. `orcl_ro` is the read-only service and is active on the standby database that has been opened in read-only mode.
Unable to connect to database using rds_custom_orcl_a
[W000 2023-01-18T07:50:32.589+00:00] Primary database cannot be reached.
[W000 2023-01-18T07:50:32.589+00:00] Fast-Start Failover threshold has expired.
[W000 2023-01-18T07:50:32.590+00:00] Try to connect to the standby.
[W000 2023-01-18T07:50:32.590+00:00] Making a last connection attempt to primary database before proceeding with Fast-Start Failover.
[W000 2023-01-18T07:50:32.591+00:00] Check if the standby is ready for failover.
[S002 2023-01-18T07:50:32.591+00:00] Fast-Start Failover started...
2023-01-18T07:50:32.591+00:00
Initiating Fast-Start Failover to database "orcl_d"...
[S002 2023-01-18T07:50:32.592+00:00] Initiating Fast-start Failover.
Performing failover NOW, please wait...
Failover succeeded, new primary is "orcl_d"
2023-01-18T07:55:32.101+00:00
[S002 2023-01-18T07:55:32.591+00:00] Fast-Start Failover finished...
[W000 2023-01-18T07:55:32.591+00:00] Failover succeeded. Restart pinging.
[W000 2023-01-18T07:55:32.603+00:00] Primary database has changed to orcl_d.
[W000 2023-01-18T07:55:33.618+00:00] Try to connect to the primary.
[W000 2023-01-18T07:55:33.622+00:00] Try to connect to the primary rds_custom_orcl_d.
[W000 2023-01-18T07:55:33.634+00:00] The standby orcl_a needs to be reinstated
[W000 2023-01-18T07:55:33.654+00:00] Try to connect to the new standby orcl_a.
[W000 2023-01-18T07:55:33.654+00:00] Connection to the primary restored!
[W000 2023-01-18T07:55:35.654+00:00] Disconnecting from database rds_custom_orcl_d.
[W000 2023-01-18T07:55:57.701+00:00] Try to connect to the new standby orcl_a.
ORA-12170: TNS:Connect timeout occurred
SQL> select name,open_mode from v$database;| DBA | | Start the service in the standby database. | To start the service in the read-only standby database, use the following code.
NAME OPEN_MODE
--------- --------------------
ORCL READ WRITE
SQL> exec dbms_service.create_service('orcl_rw','orcl_rw');
PL/SQL procedure successfully completed.
SQL> exec dbms_service.create_service('orcl_ro','orcl_ro');
PL/SQL procedure successfully completed.
SQL> exec dbms_service.start_service('orcl_rw');
PL/SQL procedure successfully completed.
SQL>
SQL> select name,open_mode from v$database;| DBA | | Automate starting the service when the primary DB is restarted. | To automatically start the service in the primary database when it’s restarted, use the following code.
NAME OPEN_MODE
--------- --------------------
ORCL READ ONLY WITH APPLY
SQL> exec dbms_service.start_service('orcl_ro');
PL/SQL procedure successfully completed.
SQL>
SQL> CREATE OR REPLACE TRIGGER TrgDgServices after startup on database| DBA | | Configure a connection between the read/write and read-only databases. | You can use the following application-configuration example for the read/write and read-only connection.
DECLARE
db_role VARCHAR(30);
db_open_mode VARCHAR(30);
BEGIN
SELECT DATABASE_ROLE, OPEN_MODE INTO db_role, db_open_mode FROM V$DATABASE;
IF db_role = 'PRIMARY' THEN
DBMS_SERV 2 ICE.START_SERVICE('orcl_rw');
END IF;
IF db_role = 'PHYSICAL STANDBY' AND db_open_mode LIKE 'READ ONLY%' THEN
DBMS_SERVICE.START_SERVICE('orcl_ro');
END IF;
END;
/
Trigger created.
SQL>
ORCL_RW = (DESCRIPTION =| DBA | ## Related resources + [Enabling High Availability with Data Guard on Amazon RDS Custom for Oracle](https://d1.awsstatic.com/whitepapers/enabling-high-availability-with-data-guard-on-amazon-rds-custom-for-oracle.pdf) (AWS Technical Guide) + [Configuring Amazon RDS as an Oracle PeopleSoft Database](https://d1.awsstatic.com/whitepapers/configuring-amazon-rds-as-peoplesoft-database.pdf) (AWS whitepaper) + [Oracle Data Guard Broker guide ](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/index.html)(Oracle reference documentation) + [Data Guard Concepts and Administration ](https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/index.html)(Oracle reference documentation) + [Oracle Data Guard Specific FAN and FCF Configuration Requirements ](https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/using-data-guard-broker-to-manage-switchovers-failovers.html#GUID-DFFDAA2B-A889-49AD-AB85-747D73FF0FF5)(Oracle reference documentation) # Unload data from an Amazon Redshift cluster across accounts to Amazon S3 *Andrew Kamel, Amazon Web Services* ## Summary When you test applications, it's helpful to have production data in your test environment. Using production data can give you a more accurate assessment of the application that you're developing. This pattern extracts data from an Amazon Redshift cluster in a production environment to an Amazon Simple Storage Service (Amazon S3) bucket in a development environment on Amazon Web Services (AWS). The pattern steps through the setup of both DEV and PROD accounts, including the following: + Required resources + AWS Identity and Access Management (IAM) roles + Network adjustments to subnets, security groups, and the virtual private cloud (VPC) to support the Amazon Redshift connection + An example AWS Lambda function with a Python runtime for testing the architecture To grant access to the Amazon Redshift cluster, the pattern uses AWS Secrets Manager to store the relevant credentials. The benefit is having all the needed information to directly connect to the Amazon Redshift cluster without needing to know where the Amazon Redshift cluster resides. Additionally, you can [monitor use of the secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/monitoring.html). The secret saved in Secrets Manager includes the Amazon Redshift cluster's host, database name, port, and relevant credentials. For information about security considerations when using this pattern, see the [Best practices](#unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3-best-practices) section. ## Prerequisites and limitations **Prerequisites ** + An [Amazon Redshift cluster running](https://docs.aws.amazon.com/redshift/latest/gsg/new-user.html) in the PROD account + An [S3 bucket created](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the DEV account + [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html) between the DEV and PROD accounts, with [route tables adjusted](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-routing.html) accordingly + [DNS hostnames and DNS resolution enabled](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html) for both peered VPCs **Limitations ** + Depending on the amount of data that you want to query, the Lambda function might time out. If your run takes more time than the maximum Lambda timeout (15 minutes), use an asynchronous approach for your Lambda code. The code example for this pattern uses the [psycopg2](https://github.com/psycopg/psycopg2) library for Python, which doesn't currently support asynchronous processing. + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service. ## Architecture The following diagram shows the target architecture, with DEV and PROD accounts. ![\[The Lambda VPC in the DEV account and the Amazon Redshift VPC in the PROD account.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5c83c617-3a85-4aea-a7a7-930f406d1cef/images/fa4d01df-483d-4454-9711-b391ebbe4629.png) The diagram shows the following workflow: 1. The Lambda function in the DEV account assumes the IAM role that's required to access the Amazon Redshift credentials in Secrets Manager in the PROD account. The Lambda function then retrieves the Amazon Redshift cluster secret. 1. The Lambda function in the DEV account uses the information to connect to the Amazon Redshift cluster in the PROD account through the peered VPCs. The Lambda function then sends an unload command to query the Amazon Redshift cluster in the PROD account. 1. The Amazon Redshift cluster in the PROD account assumes the relevant IAM role to access the S3 bucket in the DEV account. The Amazon Redshift cluster unloads the queried data to the S3 bucket in the DEV account. **Querying data from Amazon Redshift** The following diagram shows the roles that are used to retrieve the Amazon Redshift credentials and connect to the Amazon Redshift cluster. The workflow is initiated by the Lambda function. ![\[The three-step process for assuming roles across accounts.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5c83c617-3a85-4aea-a7a7-930f406d1cef/images/ab25b72c-773c-4d58-9012-4a3755c181ff.png) The diagram shows the following workflow: 1. The `CrossAccount-SM-Read-Role` in the DEV account assumes the `SM-Read-Role` in the PROD account. 1. The `SM-Read-Role` role uses the attached policy to retrieve the secret from Secrets Manager. 1. The credentials are used to access the Amazon Redshift cluster. **Uploading data to Amazon S3** The following diagram shows the cross-account read-write process for extracting data and uploading it to Amazon S3. The workflow is initiated by the Lambda function. The pattern [chains IAM roles in Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/mgmt/authorizing-redshift-service.html#authorizing-redshift-service-chaining-roles). The unload command that comes from the Amazon Redshift cluster assumes the `CrossAccount-S3-Write-Role`, and then assumes the `S3-Write-Role`. This role chaining gives Amazon Redshift access to Amazon S3. ![\[The roles that get credentials, access Amazon Redshift, and upload data to Amazon S3.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/5c83c617-3a85-4aea-a7a7-930f406d1cef/images/d2982fc6-1d12-4f9d-9493-a99ce691d693.png) The workflow includes the following steps: 1. The `CrossAccount-SM-Read-Role` in the DEV account assumes the `SM-Read-Role` in the PROD account. 1. The `SM-Read-Role` retrieves the Amazon Redshift credentials from Secrets Manager. 1. The Lambda function connects to the Amazon Redshift cluster and sends a query. 1. The Amazon Redshift cluster assumes the `CrossAccount-S3-Write-Role`. 1. The `CrossAccount-S3-Write-Role` assumes the `S3-Write-Role` in the DEV account. 1. The query results are unloaded to the S3 bucket in the DEV account. ## Tools **AWS services** + [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. + [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html) is a managed petabyte-scale data warehouse service in the AWS Cloud. + [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. **Code repository** The code for this pattern is available in the GitHub [unload-redshift-to-s3-python](https://github.com/aws-samples/unload-redshift-to-s3-python/) repository. ## Best practices **Security disclaimer** Before you implement this solution, consider the following important security recommendations: + Remember that connecting development and production accounts can increase the scope and lower overall security posture. We recommend deploying this solution only temporarily, extracting the required portion of data and then immediately destroying the deployed resources. To destroy the resources, you should delete the Lambda function, remove any IAM roles and policies created for this solution, and revoke any network access that was granted between the accounts. + Consult your security and compliance teams before copying any data from production to development environments. Personally identifiable information (PII), Protected health information (PHI), and other confidential or regulated data should generally not be copied in this manner. Copy only publicly available, non-confidential information (for example, public stock data from a shop frontend). Consider tokenizing or anonymizing data, or generating synthetic test data, instead of using production data whenever possible. One of the [AWS security principles](https://docs.aws.amazon.com/en_us/wellarchitected/2022-03-31/framework/sec-design.html) is to keep people away from data. In other words, developers should not perform operations in the production account. + Restrict access to the Lambda function in the development account because it can read data from the Amazon Redshift cluster in the production environment. + To avoid disrupting the production environment, implement the following recommendations: + Use a separate, dedicated development account for testing and development activities. + Implement strict network access controls and limit traffic between accounts to only what is necessary. + Monitor and audit access to the production environment and data sources. + Implement least-privilege access controls for all resources and services involved. + Regularly review and rotate credentials, such as AWS Secrets Manager secrets and IAM role access keys. + Refer to the following security documentation for the services used in this article: + [AWS Lambda security](https://docs.aws.amazon.com/lambda/latest/dg/lambda-security.html) + [Amazon Redshift security](https://docs.aws.amazon.com/redshift/latest/mgmt/iam-redshift-user-mgmt.html) + [Amazon S3 security](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security.html) + [AWS Secrets Manager security](https://docs.aws.amazon.com/secretsmanager/latest/userguide/security.html) + [IAM security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) Security is a top priority when accessing production data and resources. Always follow best practices, implement least-privilege access controls, and regularly review and update your security measures. ## Epics ### Query data from Amazon Redshift | Task | Description | Skills required | | --- | --- | --- | | Create a secret for the Amazon Redshift cluster. | To create the secret for the Amazon Redshift cluster, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Create a role to access Secrets Manager. | To create the role, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | ### Upload data to Amazon S3 | Task | Description | Skills required | | --- | --- | --- | | Create a role to access the S3 bucket. | To create the role for accessing the S3 bucket, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Create the Amazon Redshift role. | To create the Amazon Redshift role, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | ### Deploy the Lambda function | Task | Description | Skills required | | --- | --- | --- | | Deploy the Lambda function. | To deploy a Lambda function in the peered VPC, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | ### Test the architecture | Task | Description | Skills required | | --- | --- | --- | | Import the required resources. | To import the required resources, run the following commands:
(CONNECT_TIMEOUT= 120)(RETRY_COUNT=20)(RETRY_DELAY=3)(TRANSPORT_CONNECT_TIMEOUT=3)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST=devpsftdb.******.us-west-2.rds.amazonaws.com)(PORT=1521))
(ADDRESS = (PROTOCOL = TCP)(HOST=psftread.******.us-west-2.rds.amazonaws.com)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME = orcl_rw))
)
ORCL_RO = (DESCRIPTION =
(CONNECT_TIMEOUT= 120)(RETRY_COUNT=20)(RETRY_DELAY=3)(TRANSPORT_CONNECT_TIMEOUT=3)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST=devpsftdb.******.us-west-2.rds.amazonaws.com)(PORT=1521))
(ADDRESS = (PROTOCOL = TCP)(HOST=psftread.******.us-west-2.rds.amazonaws.com)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME = orcl_ro))
)
import ast| App developer | | Run the Lambda handler function. | The Lambda function uses AWS Security Token Service (AWS STS) for cross-account access and temporary credential management. The function uses the AssumeRole API operation to temporarily assume the permissions of the `sm_read_role` IAM role.To run the Lambda function, use the following example code:
import boto3
import psycopg2
import base64
from botocore.exceptions import ClientError
def lambda_handler(event, context):| App developer | | Get the secret. | To get the Amazon Redshift secret, use the following example code:
sts_client = boto3.client('sts')
# Secrets Manager Configurations
secret_name = "redshift_creds"
sm_region = "eu-west-1"
sm_read_role = "arn:aws:iam::PROD_ACCOUNT_NUMBER:role/SM-Read-Role"
# S3 Bucket Configurations
s3_bucket_path = "s3://mybucket/"
s3_bucket_region = "eu-west-1"
s3_write_role = "arn:aws:iam::DEV_ACCOUNT_NUMBER:role/S3-Write-Role"
# Redshift Configurations
sql_query = "select * from category"
redshift_db = "dev"
redshift_s3_write_role = "arn:aws:iam::PROD_ACCOUNT_NUMBER:role/CrossAccount-S3-Write-Role"
chained_s3_write_role = "%s,%s" % (redshift_s3_write_role, s3_write_role)
assumed_role_object = sts_client.assume_role(
RoleArn=sm_read_role,
RoleSessionName="CrossAccountRoleAssumption",
ExternalId="YOUR_EXTERNAL_ID",
)
credentials = assumed_role_object['Credentials']
secret_dict = ast.literal_eval(get_secret(credentials, secret_name, sm_region))
execute_query(secret_dict, sql_query, s3_bucket_path, chained_s3_write_role, s3_bucket_region, redshift_db)
return {
'statusCode': 200
}
def get_secret(credentials, secret_name, sm_region):| App developer | | Run the unload command. | To unload the data to the S3 bucket, use the following example code.
# Create a Secrets Manager client
session = boto3.session.Session()
sm_client = session.client(
service_name='secretsmanager',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
region_name=sm_region
)
try:
get_secret_value_response = sm_client.get_secret_value(
SecretId=secret_name
)
except ClientError as e:
print(e)
raise e
else:
if 'SecretString' in get_secret_value_response:
return get_secret_value_response['SecretString']
else:
return base64.b64decode(get_secret_value_response['SecretBinary'])
def execute_query(secret_dict, sql_query, s3_bucket_path, chained_s3_write_role, s3_bucket_region, redshift_db):| App developer | ### Clean up | Task | Description | Skills required | | --- | --- | --- | | Delete the Lambda function. | To avoid incurring unplanned costs, remove the resources and the connection between the DEV and PROD accounts.To remove the Lambda function, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Remove the IAM roles and policies. | Remove the IAM roles and policies from the DEV and PROD accounts.In the DEV account, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html)In the PROD account, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Delete the secret in Secrets Manager. | To delete the secret, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Remove VPC peering and security group rules. | To remove VPC peering and security group rules, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Remove data from the S3 bucket. | To remove the data from Amazon S3, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Clean up AWS KMS keys. | If you created any custom AWS KMS keys for encryption, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | | Review and delete Amazon CloudWatch logs. | To delete the CloudWatch logs, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.html) | DevOps engineer | ## Related resources + [Amazon CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) + [IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) + [Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) + [Amazon Redshift documentation](https://docs.aws.amazon.com/redshift/latest/gsg/new-user-serverless.html) + [Amazon S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) + [AWS Secrets Manager documentation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) + [AWS security principles](https://docs.aws.amazon.com/en_us/wellarchitected/2022-03-31/framework/sec-design.html) ## Additional information After you unload the data from Amazon Redshift to Amazon S3, you can analyze it by using Amazon Athena. [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html) is a big data query service that's beneficial when you need to access large volumes of data. You can use Athena without having to provision servers or databases. Athena supports complex queries, and you can run it on different objects. As with most AWS services, the main benefit to using Athena is that it provides great flexibility in how you run queries without the added complexity. When you use Athena, you can query different data types, such as CSV and JSON, in Amazon S3 without changing the data type. You can query data from various sources, including outside AWS. Athena reduces complexity because you don't have to manage servers. Athena reads data directly from Amazon S3 without loading or changing the data before you run the query. # Database migration patterns by workload **Topics** + [IBM](databases-database-migration-patterns-by-workload-ibm-pattern-list.md) + [Microsoft](databases-database-migration-patterns-by-workload-microsoft-pattern-list.md) + [N/A](databases-database-migration-patterns-by-workload-notapplicable-pattern-list.md) + [Open source](databases-database-migration-patterns-by-workload-open-source-pattern-list.md) + [Oracle](databases-database-migration-patterns-by-workload-oracle-pattern-list.md) + [SAP](databases-database-migration-patterns-by-workload-sap-pattern-list.md) # IBM **Topics** + [Migrate a Db2 database from Amazon EC2 to Aurora MySQL-Compatible by using AWS DMS](migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.md) + [Migrate Db2 for LUW to Amazon EC2 by using log shipping to reduce outage time](migrate-db2-for-luw-to-amazon-ec2-by-using-log-shipping-to-reduce-outage-time.md) + [Migrate Db2 for LUW to Amazon EC2 with high availability disaster recovery](migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery.md) + [Migrate from IBM Db2 on Amazon EC2 to Aurora PostgreSQL-Compatible using AWS DMS and AWS SCT](migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.md) + [Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2](migrate-from-ibm-websphere-application-server-to-apache-tomcat-on-amazon-ec2.md) + [Stream data from IBM Db2, SAP, Sybase, and other databases to MongoDB Atlas on AWS](stream-data-from-ibm-db2-to-mongodb-atlas.md) # Microsoft **Topics** + [Accelerate the discovery and migration of Microsoft workloads to AWS](accelerate-the-discovery-and-migration-of-microsoft-workloads-to-aws.md) + [Access on-premises Microsoft SQL Server tables from Microsoft SQL Server on Amazon EC2 using linked servers](access-on-premises-microsoft-sql-server-tables-from-microsoft-sql-server-on-amazon-ec2-using-linked-servers.md) + [Assess query performance for migrating SQL Server databases to MongoDB Atlas on AWS](assess-query-performance-for-migrating-sql-server-databases-to-mongodb-atlas-on-aws.md) + [Automate database tasks in SQL Server Express on Amazon EC2 by using AWS Lambda and Task Scheduler](automate-database-tasks-in-sql-server-express-edition-running-on-amazon-ec2.md) + [Change Python and Perl applications to support database migration from Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible Edition](change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.md) + [Configure read-only routing in Always On availability groups in SQL Server on AWS](configure-read-only-routing-in-an-always-on-availability-group-in-sql-server-on-aws.md) + [Configure Windows authentication for Amazon RDS for Microsoft SQL Server using AWS Managed Microsoft AD](configure-windows-authentication-for-amazon-rds-using-microsoft-ad.md) + [Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel and Python](create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel-and-python.md) + [Deploy SQL Server failover cluster instances on Amazon EC2 and Amazon FSx by using Terraform](deploy-sql-server-failover-cluster-instances-on-amazon-ec2-and-amazon-fsx.md) + [Export a Microsoft SQL Server database to Amazon S3 by using AWS DMS](export-a-microsoft-sql-server-database-to-amazon-s3-by-using-aws-dms.md) + [Export Amazon RDS for SQL Server tables to an S3 bucket by using AWS DMS](export-amazon-rds-for-sql-server-tables-to-an-s3-bucket-by-using-aws-dms.md) + [Implement SHA1 hashing for PII data when migrating from SQL Server to PostgreSQL](implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.md) + [Ingest and migrate EC2 Windows instances into an AWS Managed Services account](ingest-and-migrate-ec2-windows-instances-into-an-aws-managed-services-account.md) + [Set up a Microsoft SQL Server failover cluster on Amazon EC2 by using FSx for Windows File Server](microsoft-sql-failover-cluster-on-amazon-ec2.md) + [Migrate a messaging queue from Microsoft Azure Service Bus to Amazon SQS](migrate-a-messaging-queue-from-microsoft-azure-service-bus-to-amazon-sqs.md) + [Migrate a Microsoft SQL Server database from Amazon EC2 to Amazon DocumentDB by using AWS DMS](migrate-a-microsoft-sql-server-database-from-amazon-ec2-to-amazon-documentdb-by-using-aws-dms.md) + [Migrate a Microsoft SQL Server database to Aurora MySQL by using AWS DMS and AWS SCT](migrate-a-microsoft-sql-server-database-to-aurora-mysql-by-using-aws-dms-and-aws-sct.md) + [Migrate a .NET application from Microsoft Azure App Service to AWS Elastic Beanstalk](migrate-a-net-application-from-microsoft-azure-app-service-to-aws-elastic-beanstalk.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon EC2](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-ec2.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using linked servers](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-linked-servers.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using native backup and restore methods](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-native-backup-and-restore-methods.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS DMS](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-dms.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS SCT data extraction agents](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-sct-data-extraction-agents.md) + [Migrate an on-premises Microsoft SQL Server database to Microsoft SQL Server on Amazon EC2 running Linux](migrate-an-on-premises-microsoft-sql-server-database-to-microsoft-sql-server-on-amazon-ec2-running-linux.md) + [Migrate data from Microsoft Azure Blob to Amazon S3 by using Rclone](migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone.md) + [Migrate IIS-hosted applications to Amazon EC2 by using appcmd.exe](migrate-iis-hosted-applications-to-amazon-ec2-by-using-appcmd.md) + [Migrate Microsoft SQL Server Always On availability group using AWS Application Migration Service](migrate-microsoft-sql-server-always-on-group-using-mgn.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon EC2 using Application Migration Service](migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.md) + [Migrate a relational database to MongoDB Atlas on AWS](migrate-relational-database-to-mongodb-atlas.md) + [Migrate SQL Server to AWS using distributed availability groups](migrate-sql-server-to-aws-using-distributed-availability-groups.md) + [Migrate Windows SSL certificates to an Application Load Balancer using ACM](migrate-windows-ssl-certificates-to-an-application-load-balancer-using-acm.md) + [Rehost on-premises workloads in the AWS Cloud: migration checklist](rehost-on-premises-workloads-in-the-aws-cloud-migration-checklist.md) + [Resolve connection errors after migrating Microsoft SQL Server to the AWS Cloud](resolve-connection-errors-after-migrating-microsoft-sql-server-to-the-aws-cloud.md) + [Send notifications for an Amazon RDS for SQL Server database instance by using an on-premises SMTP server and Database Mail](send-notifications-for-an-amazon-rds-for-sql-server-database-instance-by-using-an-on-premises-smtp-server-and-database-mail.md) + [Set up a CI/CD pipeline for database migration by using Terraform](set-up-ci-cd-pipeline-for-db-migration-with-terraform.md) + [Set up Multi-AZ infrastructure for a SQL Server Always On FCI by using Amazon FSx](set-up-multi-az-infrastructure-for-a-sql-server-always-on-fci-by-using-amazon-fsx.md) # N/A **Topics** + [Create an approval process for firewall requests during a rehost migration to AWS](create-an-approval-process-for-firewall-requests-during-a-rehost-migration-to-aws.md) + [Encrypt an existing Amazon RDS for PostgreSQL DB instance](encrypt-an-existing-amazon-rds-for-postgresql-db-instance.md) + [Estimate storage costs for an Amazon DynamoDB table](estimate-storage-costs-for-an-amazon-dynamodb-table.md) + [Implement cross-Region disaster recovery with AWS DMS and Amazon Aurora](implement-cross-region-disaster-recovery-with-aws-dms-and-amazon-aurora.md) # Open source **Topics** + [Automatically generate a PynamoDB model and CRUD functions for Amazon DynamoDB by using a Python application](automatically-generate-a-pynamodb-model-and-crud-functions-for-amazon-dynamodb-by-using-a-python-application.md) + [Connect by using an SSH tunnel in pgAdmin](connect-by-using-an-ssh-tunnel-in-pgadmin.md) + [Create application users and roles in Aurora PostgreSQL-Compatible](create-application-users-and-roles-in-aurora-postgresql-compatible.md) + [Enable encrypted connections for PostgreSQL DB instances in Amazon RDS](enable-encrypted-connections-for-postgresql-db-instances-in-amazon-rds.md) + [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL with AWS SCT and AWS DMS using AWS CLI and CloudFormation](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.md) + [Migrate an on-premises MariaDB database to Amazon RDS for MariaDB using native tools](migrate-an-on-premises-mariadb-database-to-amazon-rds-for-mariadb-using-native-tools.md) + [Migrate an on-premises MySQL database to Amazon EC2](migrate-an-on-premises-mysql-database-to-amazon-ec2.md) + [Migrate an on-premises MySQL database to Amazon RDS for MySQL](migrate-an-on-premises-mysql-database-to-amazon-rds-for-mysql.md) + [Migrate an on-premises MySQL database to Aurora MySQL](migrate-an-on-premises-mysql-database-to-aurora-mysql.md) + [Migrate an on-premises PostgreSQL database to Aurora PostgreSQL](migrate-an-on-premises-postgresql-database-to-aurora-postgresql.md) + [Migrate a Couchbase Server database to Amazon EC2](migrate-couchbase-server-ec2.md) + [Migrate from IBM WebSphere Application Server to Apache Tomcat on Amazon EC2 with Auto Scaling](migrate-from-ibm-websphere-application-server-to-apache-tomcat-on-amazon-ec2-with-auto-scaling.md) + [Migrate from Oracle 8i or 9i to Amazon RDS for Oracle using SharePlex and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-oracle-using-shareplex-and-aws-dms.md) + [Migrate from PostgreSQL on Amazon EC2 to Amazon RDS for PostgreSQL using pglogical](migrate-from-postgresql-on-amazon-ec2-to-amazon-rds-for-postgresql-using-pglogical.md) + [Migrate on-premises Java applications to AWS using AWS App2Container](migrate-on-premises-java-applications-to-aws-using-aws-app2container.md) + [Migrate on-premises MySQL databases to Aurora MySQL using Percona XtraBackup, Amazon EFS, and Amazon S3](migrate-on-premises-mysql-databases-to-aurora-mysql-using-percona-xtrabackup-amazon-efs-and-amazon-s3.md) + [Migrate Oracle external tables to Amazon Aurora PostgreSQL-Compatible](migrate-oracle-external-tables-to-amazon-aurora-postgresql-compatible.md) + [Migrate Oracle functions and procedures that have more than 100 arguments to PostgreSQL](migrate-oracle-functions-and-procedures-that-have-more-than-100-arguments-to-postgresql.md) + [Migrate Redis workloads to Redis Enterprise Cloud on AWS](migrate-redis-workloads-to-redis-enterprise-cloud-on-aws.md) + [Monitor Amazon Aurora for instances without encryption](monitor-amazon-aurora-for-instances-without-encryption.md) + [Restart the AWS Replication Agent automatically without disabling SELinux after rebooting a RHEL source server](restart-the-aws-replication-agent-automatically-without-disabling-selinux-after-rebooting-a-rhel-source-server.md) + [Schedule jobs for Amazon RDS for PostgreSQL and Aurora PostgreSQL by using Lambda and Secrets Manager](schedule-jobs-for-amazon-rds-for-postgresql-and-aurora-postgresql-by-using-lambda-and-secrets-manager.md) + [Transport PostgreSQL databases between two Amazon RDS DB instances using pg\$1transport](transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.md) + [Unload data from an Amazon Redshift cluster across accounts to Amazon S3](unload-data-from-amazon-redshift-cross-accounts-to-amazon-s3.md) # Oracle **Topics** + [Add HA to Oracle PeopleSoft on Amazon RDS Custom by using a read replica](add-ha-to-oracle-peoplesoft-on-amazon-rds-custom-by-using-a-read-replica.md) + [Convert JSON Oracle queries into PostgreSQL database SQL](convert-json-oracle-queries-into-postgresql-database-sql.md) + [Convert VARCHAR2(1) data type for Oracle to Boolean data type for Amazon Aurora PostgreSQL](convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.md) + [Emulate Oracle DR by using a PostgreSQL-compatible Aurora global database](emulate-oracle-dr-by-using-a-postgresql-compatible-aurora-global-database.md) + [Emulate Oracle PL/SQL associative arrays in Amazon Aurora PostgreSQL and Amazon RDS for PostgreSQL](emulate-oracle-plsql-associative-arrays-in-aurora-and-rds-postgresql.md) + [Estimate the Amazon RDS engine size for an Oracle database by using AWR reports](estimate-the-amazon-rds-engine-size-for-an-oracle-database-by-using-awr-reports.md) + [Handle anonymous blocks in Dynamic SQL statements in Aurora PostgreSQL](handle-anonymous-blocks-in-dynamic-sql-statements-in-aurora-postgresql.md) + [Incrementally migrate from Amazon RDS for Oracle to Amazon RDS for PostgreSQL using Oracle SQL Developer and AWS SCT](incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.md) + [Load BLOB files into TEXT by using file encoding in Aurora PostgreSQL-Compatible](load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.md) + [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL in SSL mode by using AWS DMS](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.md) + [Migrate an Amazon RDS for Oracle database to another AWS account and AWS Region using AWS DMS for ongoing replication](migrate-an-amazon-rds-for-oracle-database-to-another-aws-account-and-aws-region-using-aws-dms-for-ongoing-replication.md) + [Migrate an on-premises Oracle database to Amazon EC2 by using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.md) + [Migrate an on-premises Oracle database to Amazon OpenSearch Service using Logstash](migrate-an-on-premises-oracle-database-to-amazon-opensearch-service-using-logstash.md) + [Migrate an on-premises Oracle database to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-an-on-premises-oracle-database-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md) + [Migrate an on-premises Oracle database to Amazon RDS for Oracle](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.md) + [Migrate an Oracle database to Amazon RDS using direct Oracle Data Pump Import over a database link](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-by-using-direct-oracle-data-pump-import-over-a-database-link.md) + [Migrate an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-using-oracle-data-pump.md) + [Migrate an on-premises Oracle database to Amazon RDS for PostgreSQL by using an Oracle bystander and AWS DMS](migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.md) + [Migrate an on-premises Oracle database to Oracle on Amazon EC2](migrate-an-on-premises-oracle-database-to-oracle-on-amazon-ec2.md) + [Migrate an Oracle database from Amazon EC2 to Amazon RDS for MariaDB using AWS DMS and AWS SCT](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-mariadb-using-aws-dms-and-aws-sct.md) + [Migrate an Oracle database from Amazon EC2 to Amazon RDS for Oracle using AWS DMS](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-oracle-using-aws-dms.md) + [Migrate an Oracle database to Amazon DynamoDB using AWS DMS](migrate-an-oracle-database-to-amazon-dynamodb-using-aws-dms.md) + [Migrate an Oracle database to Amazon RDS for Oracle by using Oracle GoldenGate flat file adapters](migrate-an-oracle-database-to-amazon-rds-for-oracle-by-using-oracle-goldengate-flat-file-adapters.md) + [Migrate an Oracle Database to Amazon Redshift using AWS DMS and AWS SCT](migrate-an-oracle-database-to-amazon-redshift-using-aws-dms-and-aws-sct.md) + [Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT](migrate-an-oracle-database-to-aurora-postgresql-using-aws-dms-and-aws-sct.md) + [Migrate an Oracle JD Edwards EnterpriseOne database to AWS by using Oracle Data Pump and AWS DMS](migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.md) + [Migrate an Oracle partitioned table to PostgreSQL by using AWS DMS](migrate-an-oracle-partitioned-table-to-postgresql-by-using-aws-dms.md) + [Migrate an Oracle PeopleSoft database to AWS by using AWS DMS](migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.md) + [Migrate data from an on-premises Oracle database to Aurora PostgreSQL](migrate-data-from-an-on-premises-oracle-database-to-aurora-postgresql.md) + [Migrate from Amazon RDS for Oracle to Amazon RDS for MySQL](migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-mysql.md) + [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using materialized views and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-materialized-views-and-aws-dms.md) + [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using SharePlex and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-shareplex-and-aws-dms.md) + [Migrate from Oracle Database to Amazon RDS for PostgreSQL by using Oracle GoldenGate](migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.md) + [Migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-from-oracle-on-amazon-ec2-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md) + [Migrate from Oracle WebLogic to Apache Tomcat (TomEE) on Amazon ECS](migrate-from-oracle-weblogic-to-apache-tomcat-tomee-on-amazon-ecs.md) + [Migrate function-based indexes from Oracle to PostgreSQL](migrate-function-based-indexes-from-oracle-to-postgresql.md) + [Migrate legacy applications from Oracle Pro\$1C to ECPG](migrate-legacy-applications-from-oracle-pro-c-to-ecpg.md) + [Migrate Oracle CLOB values to individual rows in PostgreSQL on AWS](migrate-oracle-clob-values-to-individual-rows-in-postgresql-on-aws.md) + [Migrate Oracle Database error codes to an Amazon Aurora PostgreSQL-Compatible database](migrate-oracle-database-error-codes-to-an-amazon-aurora-postgresql-compatible-database.md) + [Migrate Oracle native functions to PostgreSQL using extensions](migrate-oracle-native-functions-to-postgresql-using-extensions.md) + [Migrate Oracle PeopleSoft to Amazon RDS Custom](migrate-oracle-peoplesoft-to-amazon-rds-custom.md) + [Migrate Oracle ROWID functionality to PostgreSQL on AWS](migrate-oracle-rowid-functionality-to-postgresql-on-aws.md) + [Migrate Oracle SERIALLY\$1REUSABLE pragma packages into PostgreSQL](migrate-oracle-serially-reusable-pragma-packages-into-postgresql.md) + [Migrate virtual generated columns from Oracle to PostgreSQL](migrate-virtual-generated-columns-from-oracle-to-postgresql.md) + [Monitor Oracle GoldenGate logs by using Amazon CloudWatch](monitor-oracle-goldengate-logs-by-using-amazon-cloudwatch.md) + [Analyze object dependencies for partial database migrations from Oracle to PostgreSQL](multilevel-object-analysis-for-database-migration-from-oracle-to-postgresql.md) + [Replatform Oracle Database Enterprise Edition to Standard Edition 2 on Amazon RDS for Oracle](replatform-oracle-database-enterprise-edition-to-standard-edition-2-on-amazon-rds-for-oracle.md) + [Set up an HA/DR architecture for Oracle E-Business Suite on Amazon RDS Custom with an active standby database](set-up-an-ha-dr-architecture-for-oracle-e-business-suite-on-amazon-rds-custom-with-an-active-standby-database.md) + [Set up Oracle UTL\$1FILE functionality on Aurora PostgreSQL-Compatible](set-up-oracle-utl_file-functionality-on-aurora-postgresql-compatible.md) + [Transition roles for an Oracle PeopleSoft application on Amazon RDS Custom for Oracle](transition-roles-for-an-oracle-peoplesoft-application-on-amazon-rds-custom-for-oracle.md) + [Validate database objects after migrating from Oracle to Amazon Aurora PostgreSQL](validate-database-objects-after-migrating-from-oracle-to-amazon-aurora-postgresql.md) # SAP **Topics** + [Automatically back up SAP HANA databases using Systems Manager and EventBridge](automatically-back-up-sap-hana-databases-using-systems-manager-and-eventbridge.md) + [Migrate from SAP ASE to Amazon RDS for SQL Server using AWS DMS](migrate-from-sap-ase-to-amazon-rds-for-sql-server-using-aws-dms.md) + [Migrate SAP ASE on Amazon EC2 to Amazon Aurora PostgreSQL-Compatible using AWS SCT and AWS DMS](migrate-sap-ase-on-amazon-ec2-to-amazon-aurora-postgresql-compatible-using-aws-sct-and-aws-dms.md) + [Migrate SAP HANA to AWS using SAP HSR with the same hostname](migrate-sap-hana-to-aws-using-sap-hsr-with-the-same-hostname.md) + [Set up disaster recovery for SAP on IBM Db2 on AWS](set-up-disaster-recovery-for-sap-on-ibm-db2-on-aws.md) # More patterns **Topics** + [Access an Amazon Neptune database from an Amazon EKS container](access-amazon-neptune-database-from-amazon-eks-container.md) + [Access, query, and join Amazon DynamoDB tables using Athena](access-query-and-join-amazon-dynamodb-tables-using-athena.md) + [Allow EC2 instances write access to S3 buckets in AMS accounts](allow-ec2-instances-write-access-to-s3-buckets-in-ams-accounts.md) + [Analyze and visualize nested JSON data with Amazon Athena and Amazon Quick Sight](analyze-and-visualize-nested-json-data-with-amazon-athena-and-amazon-quicksight.md) + [Automate backups for Amazon RDS for PostgreSQL DB instances by using AWS Batch](automate-backups-for-amazon-rds-for-postgresql-db-instances-by-using-aws-batch.md) + [Automatically archive items to Amazon S3 using DynamoDB TTL](automatically-archive-items-to-amazon-s3-using-dynamodb-ttl.md) + [Automatically remediate unencrypted Amazon RDS DB instances and clusters](automatically-remediate-unencrypted-amazon-rds-db-instances-and-clusters.md) + [Automatically stop and start an Amazon RDS DB instance using AWS Systems Manager Maintenance Windows](automatically-stop-and-start-an-amazon-rds-db-instance-using-aws-systems-manager-maintenance-windows.md) + [Build an AWS landing zone that includes MongoDB Atlas](build-aws-landing-zone-that-includes-mongodb-atlas.md) + [Build COBOL Db2 programs by using AWS Mainframe Modernization and AWS CodeBuild](build-cobol-db2-programs-mainframe-modernization-codebuild.md) + [Build an enterprise data mesh with Amazon DataZone, AWS CDK, and AWS CloudFormation](build-enterprise-data-mesh-amazon-data-zone.md) + [Change Python and Perl applications to support database migration from Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible Edition](change-python-and-perl-applications-to-support-database-migration-from-microsoft-sql-server-to-amazon-aurora-postgresql-compatible-edition.md) + [Convert and unpack EBCDIC data to ASCII on AWS by using Python](convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.md) + [Convert the Teradata NORMALIZE temporal feature to Amazon Redshift SQL](convert-the-teradata-normalize-temporal-feature-to-amazon-redshift-sql.md) + [Convert the Teradata RESET WHEN feature to Amazon Redshift SQL](convert-the-teradata-reset-when-feature-to-amazon-redshift-sql.md) + [Convert VARCHAR2(1) data type for Oracle to Boolean data type for Amazon Aurora PostgreSQL](convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.md) + [Create application users and roles in Aurora PostgreSQL-Compatible](create-application-users-and-roles-in-aurora-postgresql-compatible.md) + [Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel and Python](create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel-and-python.md) + [Deliver DynamoDB records to Amazon S3 using Kinesis Data Streams and Firehose with AWS CDK](deliver-dynamodb-records-to-amazon-s3-using-kinesis-data-streams-and-amazon-data-firehose-with-aws-cdk.md) + [Deploy a Cassandra cluster on Amazon EC2 with private static IPs to avoid rebalancing](deploy-a-cassandra-cluster-on-amazon-ec2-with-private-static-ips-to-avoid-rebalancing.md) + [Deploy a CockroachDB cluster in Amazon EKS by using Terraform](deploy-cockroachdb-on-eks-using-terraform.md) + [Develop advanced generative AI chat-based assistants by using RAG and ReAct prompting](develop-advanced-generative-ai-chat-based-assistants-by-using-rag-and-react-prompting.md) + [Emulate Oracle DR by using a PostgreSQL-compatible Aurora global database](emulate-oracle-dr-by-using-a-postgresql-compatible-aurora-global-database.md) + [Enable DB2 log archiving directly to Amazon S3 in an IBM Db2 database](enable-db2-logarchive-directly-to-amazon-s3-in-ibm-db2-database.md) + [Enable transparent data encryption in Amazon RDS for SQL Server](enable-transparent-data-encryption-in-amazon-rds-for-sql-server.md) + [Export a Microsoft SQL Server database to Amazon S3 by using AWS DMS](export-a-microsoft-sql-server-database-to-amazon-s3-by-using-aws-dms.md) + [Implement SHA1 hashing for PII data when migrating from SQL Server to PostgreSQL](implement-sha1-hashing-for-pii-data-when-migrating-from-sql-server-to-postgresql.md) + [Incrementally migrate from Amazon RDS for Oracle to Amazon RDS for PostgreSQL using Oracle SQL Developer and AWS SCT](incrementally-migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-using-oracle-sql-developer-and-aws-sct.md) + [Load BLOB files into TEXT by using file encoding in Aurora PostgreSQL-Compatible](load-blob-files-into-text-by-using-file-encoding-in-aurora-postgresql-compatible.md) + [Manage credentials using AWS Secrets Manager](manage-credentials-using-aws-secrets-manager.md) + [Migrate a Db2 database from Amazon EC2 to Aurora MySQL-Compatible by using AWS DMS](migrate-a-db2-database-from-amazon-ec2-to-aurora-mysql-compatible-by-using-aws-dms.md) + [Migrate a Microsoft SQL Server database from Amazon EC2 to Amazon DocumentDB by using AWS DMS](migrate-a-microsoft-sql-server-database-from-amazon-ec2-to-amazon-documentdb-by-using-aws-dms.md) + [Migrate a Microsoft SQL Server database to Aurora MySQL by using AWS DMS and AWS SCT](migrate-a-microsoft-sql-server-database-to-aurora-mysql-by-using-aws-dms-and-aws-sct.md) + [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL in SSL mode by using AWS DMS](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-in-ssl-mode-by-using-aws-dms.md) + [Migrate Amazon RDS for Oracle to Amazon RDS for PostgreSQL with AWS SCT and AWS DMS using AWS CLI and CloudFormation](migrate-amazon-rds-for-oracle-to-amazon-rds-for-postgresql-with-aws-sct-and-aws-dms-using-aws-cli-and-aws-cloudformation.md) + [Migrate an Amazon RDS DB instance to another VPC or account](migrate-an-amazon-rds-db-instance-to-another-vpc-or-account.md) + [Migrate an Amazon RDS for Oracle database to another AWS account and AWS Region using AWS DMS for ongoing replication](migrate-an-amazon-rds-for-oracle-database-to-another-aws-account-and-aws-region-using-aws-dms-for-ongoing-replication.md) + [Migrate an Amazon Redshift cluster to an AWS Region in China](migrate-an-amazon-redshift-cluster-to-an-aws-region-in-china.md) + [Migrate an on-premises MariaDB database to Amazon RDS for MariaDB using native tools](migrate-an-on-premises-mariadb-database-to-amazon-rds-for-mariadb-using-native-tools.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon EC2](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-ec2.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using linked servers](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-linked-servers.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon RDS for SQL Server using native backup and restore methods](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-rds-for-sql-server-using-native-backup-and-restore-methods.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS DMS](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-dms.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon Redshift using AWS SCT data extraction agents](migrate-an-on-premises-microsoft-sql-server-database-to-amazon-redshift-using-aws-sct-data-extraction-agents.md) + [Migrate an on-premises Microsoft SQL Server database to Microsoft SQL Server on Amazon EC2 running Linux](migrate-an-on-premises-microsoft-sql-server-database-to-microsoft-sql-server-on-amazon-ec2-running-linux.md) + [Migrate an on-premises MySQL database to Amazon RDS for MySQL](migrate-an-on-premises-mysql-database-to-amazon-rds-for-mysql.md) + [Migrate an on-premises MySQL database to Aurora MySQL](migrate-an-on-premises-mysql-database-to-aurora-mysql.md) + [Migrate an on-premises Oracle database to Amazon EC2 by using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-ec2-by-using-oracle-data-pump.md) + [Migrate an on-premises Oracle database to Amazon OpenSearch Service using Logstash](migrate-an-on-premises-oracle-database-to-amazon-opensearch-service-using-logstash.md) + [Migrate an on-premises Oracle database to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-an-on-premises-oracle-database-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md) + [Migrate an on-premises Oracle database to Amazon RDS for Oracle](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle.md) + [Migrate an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump](migrate-an-on-premises-oracle-database-to-amazon-rds-for-oracle-using-oracle-data-pump.md) + [Migrate an on-premises Oracle database to Amazon RDS for PostgreSQL by using an Oracle bystander and AWS DMS](migrate-an-on-premises-oracle-database-to-amazon-rds-for-postgresql-by-using-an-oracle-bystander-and-aws-dms.md) + [Migrate an on-premises Oracle database to Oracle on Amazon EC2](migrate-an-on-premises-oracle-database-to-oracle-on-amazon-ec2.md) + [Migrate an on-premises PostgreSQL database to Aurora PostgreSQL](migrate-an-on-premises-postgresql-database-to-aurora-postgresql.md) + [Migrate an on-premises ThoughtSpot Falcon database to Amazon Redshift](migrate-an-on-premises-thoughtspot-falcon-database-to-amazon-redshift.md) + [Migrate an Oracle database from Amazon EC2 to Amazon RDS for MariaDB using AWS DMS and AWS SCT](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-mariadb-using-aws-dms-and-aws-sct.md) + [Migrate an Oracle database from Amazon EC2 to Amazon RDS for Oracle using AWS DMS](migrate-an-oracle-database-from-amazon-ec2-to-amazon-rds-for-oracle-using-aws-dms.md) + [Migrate an Oracle database to Amazon RDS for Oracle by using Oracle GoldenGate flat file adapters](migrate-an-oracle-database-to-amazon-rds-for-oracle-by-using-oracle-goldengate-flat-file-adapters.md) + [Migrate an Oracle Database to Amazon Redshift using AWS DMS and AWS SCT](migrate-an-oracle-database-to-amazon-redshift-using-aws-dms-and-aws-sct.md) + [Migrate an Oracle database to Aurora PostgreSQL using AWS DMS and AWS SCT](migrate-an-oracle-database-to-aurora-postgresql-using-aws-dms-and-aws-sct.md) + [Migrate an Oracle JD Edwards EnterpriseOne database to AWS by using Oracle Data Pump and AWS DMS](migrate-an-oracle-jd-edwards-enterpriseone-database-to-aws-by-using-oracle-data-pump-and-aws-dms.md) + [Migrate an Oracle partitioned table to PostgreSQL by using AWS DMS](migrate-an-oracle-partitioned-table-to-postgresql-by-using-aws-dms.md) + [Migrate an Oracle PeopleSoft database to AWS by using AWS DMS](migrate-an-oracle-peoplesoft-database-to-aws-by-using-aws-dms.md) + [Migrate a Couchbase Server database to Amazon EC2](migrate-couchbase-server-ec2.md) + [Migrate data from an on-premises Oracle database to Aurora PostgreSQL](migrate-data-from-an-on-premises-oracle-database-to-aurora-postgresql.md) + [Migrate data to the AWS Cloud by using Starburst](migrate-data-to-the-aws-cloud-by-using-starburst.md) + [Migrate Db2 for LUW to Amazon EC2 by using log shipping to reduce outage time](migrate-db2-for-luw-to-amazon-ec2-by-using-log-shipping-to-reduce-outage-time.md) + [Migrate Db2 for LUW to Amazon EC2 with high availability disaster recovery](migrate-db2-for-luw-to-amazon-ec2-with-high-availability-disaster-recovery.md) + [Migrate from Amazon RDS for Oracle to Amazon RDS for MySQL](migrate-from-amazon-rds-for-oracle-to-amazon-rds-for-mysql.md) + [Migrate from Couchbase Server to Couchbase Capella on AWS](migrate-from-couchbase-server-to-couchbase-capella-on-aws.md) + [Migrate from IBM Db2 on Amazon EC2 to Aurora PostgreSQL-Compatible using AWS DMS and AWS SCT](migrate-from-ibm-db2-on-amazon-ec2-to-aurora-postgresql-compatible-using-aws-dms-and-aws-sct.md) + [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using materialized views and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-materialized-views-and-aws-dms.md) + [Migrate from Oracle 8i or 9i to Amazon RDS for PostgreSQL using SharePlex and AWS DMS](migrate-from-oracle-8i-or-9i-to-amazon-rds-for-postgresql-using-shareplex-and-aws-dms.md) + [Migrate from Oracle Database to Amazon RDS for PostgreSQL by using Oracle GoldenGate](migrate-from-oracle-database-to-amazon-rds-for-postgresql-by-using-oracle-goldengate.md) + [Migrate from Oracle on Amazon EC2 to Amazon RDS for MySQL using AWS DMS and AWS SCT](migrate-from-oracle-on-amazon-ec2-to-amazon-rds-for-mysql-using-aws-dms-and-aws-sct.md) + [Migrate from PostgreSQL on Amazon EC2 to Amazon RDS for PostgreSQL using pglogical](migrate-from-postgresql-on-amazon-ec2-to-amazon-rds-for-postgresql-using-pglogical.md) + [Migrate from SAP ASE to Amazon RDS for SQL Server using AWS DMS](migrate-from-sap-ase-to-amazon-rds-for-sql-server-using-aws-dms.md) + [Migrate function-based indexes from Oracle to PostgreSQL](migrate-function-based-indexes-from-oracle-to-postgresql.md) + [Migrate legacy applications from Oracle Pro\$1C to ECPG](migrate-legacy-applications-from-oracle-pro-c-to-ecpg.md) + [Migrate an on-premises Microsoft SQL Server database to Amazon EC2 using Application Migration Service](migrate-microsoft-sql-server-to-amazon-ec2-using-aws-mgn.md) + [Migrate on-premises Cloudera workloads to Cloudera Data Platform on AWS](migrate-on-premises-cloudera-workloads-to-cloudera-data-platform-on-aws.md) + [Migrate on-premises MySQL databases to Aurora MySQL using Percona XtraBackup, Amazon EFS, and Amazon S3](migrate-on-premises-mysql-databases-to-aurora-mysql-using-percona-xtrabackup-amazon-efs-and-amazon-s3.md) + [Migrate Oracle Business Intelligence 12c to the AWS Cloud from on-premises servers](migrate-oracle-business-intelligence-12c-to-the-aws-cloud-from-on-premises-servers.md) + [Migrate Oracle CLOB values to individual rows in PostgreSQL on AWS](migrate-oracle-clob-values-to-individual-rows-in-postgresql-on-aws.md) + [Migrate Oracle Database error codes to an Amazon Aurora PostgreSQL-Compatible database](migrate-oracle-database-error-codes-to-an-amazon-aurora-postgresql-compatible-database.md) + [Migrate Oracle external tables to Amazon Aurora PostgreSQL-Compatible](migrate-oracle-external-tables-to-amazon-aurora-postgresql-compatible.md) + [Migrate Oracle native functions to PostgreSQL using extensions](migrate-oracle-native-functions-to-postgresql-using-extensions.md) + [Migrate Oracle PeopleSoft to Amazon RDS Custom](migrate-oracle-peoplesoft-to-amazon-rds-custom.md) + [Migrate Oracle ROWID functionality to PostgreSQL on AWS](migrate-oracle-rowid-functionality-to-postgresql-on-aws.md) + [Migrate Oracle SERIALLY\$1REUSABLE pragma packages into PostgreSQL](migrate-oracle-serially-reusable-pragma-packages-into-postgresql.md) + [Migrate SAP ASE on Amazon EC2 to Amazon Aurora PostgreSQL-Compatible using AWS SCT and AWS DMS](migrate-sap-ase-on-amazon-ec2-to-amazon-aurora-postgresql-compatible-using-aws-sct-and-aws-dms.md) + [Migrate virtual generated columns from Oracle to PostgreSQL](migrate-virtual-generated-columns-from-oracle-to-postgresql.md) + [Monitor Amazon ElastiCache clusters for at-rest encryption](monitor-amazon-elasticache-clusters-for-at-rest-encryption.md) + [Query Amazon DynamoDB tables with SQL by using Amazon Athena](query-amazon-dynamodb-tables-sql-amazon-athena.md) + [Secure and streamline user access in a Db2 federation database on AWS by using trusted contexts](secure-and-streamline-user-access-in-a-db2-federation-database-on-aws-by-using-trusted-contexts.md) + [Set up a highly available PeopleSoft architecture on AWS](set-up-a-highly-available-peoplesoft-architecture-on-aws.md) + [Set up Oracle UTL\$1FILE functionality on Aurora PostgreSQL-Compatible](set-up-oracle-utl_file-functionality-on-aurora-postgresql-compatible.md) + [Streamline PostgreSQL deployments on Amazon EKS by using PGO](streamline-postgresql-deployments-amazon-eks-pgo.md) + [Transfer large-scale Db2 z/OS data to Amazon S3 in CSV files](transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.md) + [Transport PostgreSQL databases between two Amazon RDS DB instances using pg\$1transport](transport-postgresql-databases-between-two-amazon-rds-db-instances-using-pg-transport.md) + [Validate database objects after migrating from Oracle to Amazon Aurora PostgreSQL](validate-database-objects-after-migrating-from-oracle-to-amazon-aurora-postgresql.md)
conn_string = "dbname='%s' port='%s' user='%s' password='%s' host='%s'" \
% (redshift_db,
secret_dict["port"],
secret_dict["username"],
secret_dict["password"],
secret_dict["host"])
con = psycopg2.connect(conn_string)
unload_command = "UNLOAD ('{}') TO '{}' IAM_ROLE '{}' DELIMITER '|' REGION '{}';" \
.format(sql_query,
s3_bucket_path + str(datetime.datetime.now()) + ".csv",
chained_s3_write_role,
s3_bucket_region)
# Opening a cursor and run query
cur = con.cursor()
cur.execute(unload_command)
print(cur.fetchone())
cur.close()
con.close()