result : results) {
processComplexData(result);
}
```
*Using optimization strategies together*
These three strategies work synergistically: APIs provide the tools for efficient data access, caching reduces the need for repeated data retrieval, and business logic optimization ensures that these APIs are used in the most effective way possible. Regular monitoring and adjustment of these optimizations ensure continued performance improvements while maintaining the reliability and functionality of the modernized application. The key to success lies in understanding when and how to apply each strategy based on your application’s characteristics and performance goals.
## Tools
+ [JProfiler](https://www.ej-technologies.com/jprofiler) is a Java profiling tool that’s designed for developers and performance engineers. It analyzes Java applications and helps identify performance bottlenecks, memory leaks, and threading issues. JProfiler offers CPU, memory, and thread profiling as well as database and Java virtual machine (JVM) monitoring to provide insights into application behavior.
**Note**
As an alternative to JProfiler, you can use [Java VisualVM](https://visualvm.github.io/). This is a free, open source performance profiling and monitoring tool for Java applications that offers real-time monitoring of CPU usage, memory consumption, thread management, and garbage collection statistics. Because Java VisualVM is a built-in JDK tool, it is more cost-effective than JProfiler for basic profiling needs.
+ [pgAdmin](https://www.pgadmin.org/) is an open source administration and development tool for PostgreSQL. It provides a graphical interface that helps you create, maintain, and use database objects. You can use pgAdmin to perform a wide range of tasks, from writing simple SQL queries to developing complex databases. Its features include a syntax highlighting SQL editor, a server-side code editor, a scheduling agent for SQL, shell, and batch tasks, and support for all PostgreSQL features for both novice and experienced PostgreSQL users.
## Best practices
Identifying performance hotspots:
+ Document baseline performance metrics before you start optimizations.
+ Set clear performance improvement targets based on business requirements.
+ When benchmarking, disable verbose logging, because it can affect performance.
+ Set up a performance test suite and run it periodically.
+ Use the latest version of pgAdmin. (Older versions don't support the `EXPLAIN` query plan.)
+ For benchmarking, detach JProfiler after your optimizations are complete because it adds to latency.
+ For benchmarking, make sure to run the server in start mode instead of debug mode, because debug mode adds to latency.
Optimization strategies:
+ Configure **SetMaxResults** values in the `application.yaml` file, to specify right-sized batches according to your system specifications.
+ Configure **SetMaxResults** values based on data volume and memory constraints.
+ Change **SetOnGreatorOrEqual** to **SetOnEqual** only when subsequent calls are `.readNextEqual()`.
+ In batch write or update operations, handle the last batch separately, because it might be smaller than the configured batch size and could be missed by the write or update operation.
Caching:
+ Fields that are introduced for caching in `processImpl`, which mutate with each execution, should always be defined in the context of that `processImpl`. The fields should also be cleared by using the `doReset()` or `cleanUp()` method.
+ When you implement in-memory caching, right-size the cache. Very large caches that are stored in memory can take up all the resources, which might affect the overall performance of your application.
SQLExecutionBuilder:
+ For queries that you are planning to use in SQLExecutionBuilder, use key names such as `PROGRAMNAME_STATEMENTNUMBER`.
+ When you use SQLExecutionBuilder, always check for the `Sqlcod` field. This field contains a value that specifies whether the query was executed correctly or encountered any errors.
+ Use parameterized queries to prevent SQL injection.
Business logic optimization:
+ Maintain functional equivalence when restructuring code, and run regression testing and database comparison for the relevant subset of programs.
+ Maintain profiling snapshots for comparison.
## Epics
### Install JProfiler and pgAdmin
| Task | Description | Skills required |
| --- | --- | --- |
| Install and configure JProfiler. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
| Install and configure pgAdmin. | In this step, you install and configure a DB client to query your database. This pattern uses a PostgreSQL database and pgAdmin as a database client. If you are using another database engine, follow the documentation for the corresponding DB client.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
### Identify hotspots
| Task | Description | Skills required |
| --- | --- | --- |
| Enable SQL query logging in your AWS Blu Age application. | Enable the loggers for SQL query logging in the `application.properties` file of your AWS Blu Age application, as explained in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
| Generate and analyze query `EXPLAIN` plans to identify database performance hotspots. | For details, see the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
| Create a JProfiler snapshot to analyze a slow-performing test case. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
| Analyze the JProfiler snapshot to identify performance bottlenecks. | Follow these steps to analyze the JProfiler snapshot.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html)For more information about using JProfiler, see the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section and the [JProfiler documentation](https://www.ej-technologies.com/jprofiler/docs). | App developer |
### Establish a baseline
| Task | Description | Skills required |
| --- | --- | --- |
| Establish a performance baseline before you implement optimizations. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) | App developer |
### Apply optimization strategies
| Task | Description | Skills required |
| --- | --- | --- |
| Optimize read calls. | Optimize data retrieval by using the DAOManager **SetMaxResults** method. For more information about this approach, see the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer, DAOManager |
| Refactor the business logic to avoid multiple calls to the database. | Reduce database calls by using a SQL `JOIN` clause. For details and examples, see *Business logic optimization* in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer, SQLExecutionBuilder |
| Refactor the code to use caching to reduce the latency of read calls. | For information about this technique, see *Caching* in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
| Rewrite inefficient code that uses multiple DAOManager operations for simple update operations. | For more information about updating data directly in the database, see *Business logic optimization* in the [Architecture](#optimize-performance-aws-blu-age-modernized-application-architecture) section. | App developer |
### Test optimization strategies
| Task | Description | Skills required |
| --- | --- | --- |
| Validate each optimization change iteratively while maintaining functional equivalence. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html)Using baseline metrics as a reference ensures accurate measurement of each optimization's impact while maintaining system reliability. | App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| When you run the modern application, you see an exception with the error `Query_ID not found`. | To resolve this issue:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) |
| You’ve added indexes, but you don’t see any performance improvements. | Follow these steps to ensure that the query engine is using the index:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) |
| You encounter an out-of-memory exception. | Verify that the code releases the memory held by the data structure. |
| Batch write operations result in missing records in the table | Review the code to ensure that an additional write operation is performed when the batch count isn’t zero. |
| SQL logging doesn’t appear in application logs. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/optimize-performance-aws-blu-age-modernized-application.html) |
## Related resources
+ [Refactoring applications automatically with AWS Blu Age](https://docs.aws.amazon.com/m2/latest/userguide/refactoring-m2.html) (*AWS Mainframe Modernization User Guide*)
+ [pgAdmin documentation](https://www.pgadmin.org/docs/)
+ [JProfiler documentation](https://www.ej-technologies.com/jprofiler/docs)
# Secure and streamline user access in a Db2 federation database on AWS by using trusted contexts
*Sai Parthasaradhi, Amazon Web Services*
## Summary
Many companies are migrating their legacy mainframe workloads to Amazon Web Services (AWS). This migration includes shifting IBM Db2 for z/OS databases to Db2 for Linux, Unix, and Windows (LUW) on Amazon Elastic Compute Cloud (Amazon EC2). During a phased migration from on premises to AWS, users might need to access data in IBM Db2 z/OS and in Db2 LUW on Amazon EC2 until all applications and databases are fully migrated to Db2 LUW. In such remote data-access scenarios, user authentication can be challenging because different platforms use different authentication mechanisms.
This pattern covers how to set up a federation server on Db2 for LUW with Db2 for z/OS as a remote database. The pattern uses a trusted context to propagate a user’s identity from Db2 LUW to Db2 z/OS without re-authenticating on the remote database. For more information about trusted contexts, see the [Additional information](#secure-and-streamline-user-access-in-a-db2-federation-database-on-aws-by-using-trusted-contexts-additional) section.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account
+ A Db2 instance running on an Amazon EC2 instance
+ A remote Db2 for z/OS database running on premises
+ The on-premises network connected to AWS through [AWS Site-to-Site VPN](https://aws.amazon.com/vpn/) or [AWS Direct Connect](https://aws.amazon.com/directconnect/)
## Architecture
**Target architecture**
![\[On-premises mainframe connects through on-premises Db2 server and VPN to the Db2 DB on EC2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/9e04f0fe-bae2-412a-93ac-83da50222017/images/0a384695-7907-4fb8-bb7e-d170dcc114af.png)
## Tools
**AWS services**
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/ec2/) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [AWS Site-to-Site VPN](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html) helps you pass traffic between instances that you launch on AWS and your own remote network.
**Other tools**
+ [db2cli](https://www.ibm.com/docs/en/db2/11.5?topic=commands-db2cli-db2-interactive-cli) is the Db2 interactive command line interface (CLI) command.
## Epics
### Enable federation on the Db2 LUW database running on AWS
| Task | Description | Skills required |
| --- | --- | --- |
| Enable federation on the DB2 LUW DB. | To enable federation on DB2 LUW, run the following command.update dbm cfg using federated YES
| DBA |
| Restart the database. | To restart the database, run the following command.db2stop force;
db2start;
| DBA |
### Catalog the remote database
| Task | Description | Skills required |
| --- | --- | --- |
| Catalog the remote Db2 z/OS subsystem. | To catalog the remote Db2 z/OS database on Db2 LUW running on AWS, use the following example command.catalog TCPIP NODE tcpnode REMOTE mainframehost SERVER mainframeport
| DBA |
| Catalog the remote database. | To catalog the remote database, use the following example command.catalog db dbnam1 as ndbnam1 at node tcpnode
| DBA |
### Create the remote server definition
| Task | Description | Skills required |
| --- | --- | --- |
| Collect user credentials for the remote Db2 z/OS database. | Before proceeding with the next steps, gather the following information:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/secure-and-streamline-user-access-in-a-db2-federation-database-on-aws-by-using-trusted-contexts.html) | DBA |
| Create the DRDA wrapper. | To create the DRDA wrapper, run the following command.CREATE WRAPPER DRDA;
| DBA |
| Create the server definition. | To create the server definition, run the following example command.CREATE SERVER ndbserver
TYPE DB2/ZOS VERSION 12
WRAPPER DRDA
AUTHORIZATION "dbuser1" PASSWORD "dbpasswd" OPTIONS ( DBNAME 'ndbnam1',FED_PROXY_USER 'ZPROXY' );
In this definition, `FED_PROXY_USER` specifies the proxy user that will be used for establishing trusted connections to the Db2 z/OS database. The authorization user ID and password are required only for creating the remote server object in the Db2 LUW database. They won’t be used later during runtime. | DBA |
### Create user mappings
| Task | Description | Skills required |
| --- | --- | --- |
| Create a user mapping for the proxy user. | To create a user mapping for proxy user, run the following command.CREATE USER MAPPING FOR ZPROXY SERVER ndbserver OPTIONS (REMOTE_AUTHID 'ZPROXY', REMOTE_PASSWORD 'zproxy');
| DBA |
| Create user mappings for each user on Db2 LUW. | Create user mappings for all the users on the Db2 LUW database on AWS who need to access remote data through the proxy user. To create the user mappings, run the following command.CREATE USER MAPPING FOR PERSON1 SERVER ndbserver OPTIONS (REMOTE_AUTHID 'USERZID', USE_TRUSTED_CONTEXT 'Y');
The statement specifies that a user on Db2 LUW (`PERSON1`) can establish a trusted connection to the remote Db2 z/OS database (`USE_TRUSTED_CONTEXT 'Y'`). After the connection is established through the proxy user, the user can access the data by using the Db2 z/OS user ID (`REMOTE_AUTHID 'USERZID'`). | DBA |
### Create the trusted context object
| Task | Description | Skills required |
| --- | --- | --- |
| Create the trusted context object. | To create the trusted context object on the remote Db2 z/OS database, use the following example command.CREATE TRUSTED CONTEXT CTX_LUW_ZOS
BASED UPON CONNECTION USING SYSTEM AUTHID ZPROXY
ATTRIBUTES (
ADDRESS '10.10.10.10'
)
NO DEFAULT ROLE
ENABLE
WITH USE FOR PUBLIC WITHOUT AUTHENTICATION;
In this definition, `CTX_LUW_ZOS` is an arbitrary name for the trusted context object. The object contains the proxy user ID and the IP address of the server from which the trusted connection must originate. In this example, the server the Db2 LUW database on AWS. You can use the domain name instead of the IP address. The clause `WITH USE FOR PUBLIC WITHOUT AUTHENTICATION` indicates that switching the user ID on a trusted connection is allowed for every user ID. A password doesn't need to be provided. | DBA |
## Related resources
+ [IBM Resource Access Control Facility (RACF)](https://www.ibm.com/products/resource-access-control-facility)
+ [IBM Db2 LUW Federation](https://www.ibm.com/docs/en/db2/11.5?topic=federation)
+ [Trusted contexts](https://www.ibm.com/docs/en/db2-for-zos/13?topic=contexts-trusted)
## Additional information
**Db2 trusted contexts**
A trusted context is a Db2 database object that defines a trust relationship between a federated server and a remote database server. To define a trusted relationship, the trusted context specifies trust attributes. There are three types of trust attributes:
+ The system authorization ID that makes the initial database connection request
+ The IP address or domain name from which the connection is made
+ The encryption setting for data communications between the database server and the database client
A trusted connection is established when all attributes of a connection request match the attributes specified in any trusted context object defined on the server. There are two types of trusted connections: implicit and explicit. After an implicit trusted connection is established, a user inherits a role that isn't available to them outside the scope of that trusted connection definition. After an explicit trusted connection is established, users can be switched on the same physical connection, with or without authentication. In addition, Db2 users can be granted roles that specify privileges that are for use only within the trusted connection. This pattern uses an explicit trusted connection.
*Trusted context in this pattern*
After the pattern is complete, PERSON1 on Db2 LUW accesses remote data from Db2 z/OS by using a federated trusted context. The connection for PERSON1 is established through a proxy user if the connection originates from the IP address or domain name that is specified in the trusted context definition. After the connection is established, PERSON1's corresponding Db2 z/OS user ID is switched without re-authentication, and the user can access the data or objects based on the Db2 privileges set up for that user.
*Benefits of federated trusted contexts*
+ This approach maintains the principle of least privilege by eliminating the use of a common user ID or application ID that would need a superset of all the privileges required by all users.
+ The real identity of the user who performs the transaction on both the federated and remote database is always known and can be audited.
+ Performance improves because the physical connection is being reused across the users without the need for the federated server to re-authenticate.
# Transfer large-scale Db2 z/OS data to Amazon S3 in CSV files
*Bruno Sahinoglu, Abhijit Kshirsagar, and Ivan Schuster, Amazon Web Services*
## Summary
A mainframe is still a system of record in many enterprises, containing a massive amount of data including master data entities with records of current as well as the historical business transactions. It is often siloed and not easily accessed by the distributed systems within the same enterprise. With the emergence of cloud technology and big data democratization, enterprises are interested in using the insights hidden in the mainframe data to develop new business capabilities.
With that objective, enterprises are looking to open their mainframe Db2 data to their Amazon Web Services (AWS) Cloud environment. The business reasons are several and the transfer methods differ from case to case. You might prefer to connect your application directly to the mainframe, or you might prefer to replicate your data in near real time. If the use case is to feed a data warehouse or a data lake, having an up-to-date copy is no longer a concern, and the procedure described in this pattern might suffice, especially if you want to avoid any third-party product licensing costs. Another use case might be the mainframe data transfer for a migration project. In a migration scenario, data is required for performing the functional equivalence testing. The approach described in this post is a cost effective way to transfer the Db2 data to the AWS Cloud environment.
Because Amazon Simple Storage Service (Amazon S3) is one of the most integrated AWS services, you can access the data from there and gather insights directly by using other AWS services such as Amazon Athena, AWS Lambda functions, or Amazon QuickSight . You can also load the data to Amazon Aurora or Amazon DynamoDB by using AWS Glue or AWS Database Migration Service (AWS DMS). With that aim in mind, this describes how to unload Db2 data in CSV files in ASCII format on the mainframe and transfer the files to Amazon S3.
For this purpose, [mainframe scripts](https://github.com/aws-samples/unloaddb2-samples) have been developed to help to generate job control languages (JCLs) to unload and transfer as many Db2 tables as you need.
## Prerequisites and limitations
**Prerequisites**
+ An IBM z/OS operating system user with authorization to run Restructured Extended Executor (REXX) and JCL scripts.
+ Access to z/OS Unix System Services (USS) to generate SSH (Secure Shell) private and public keys.
+ A writable S3 bucket. For more information, see [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) in the Amazon S3 documentation.
+ An AWS Transfer Family SSH File Transfer Protocol (SFTP)-enabled server using **Service managed** as the identity provider and Amazon S3 as the AWS storage service. For more information, see [Create an SFTP-enabled server](https://docs.aws.amazon.com/transfer/latest/userguide/create-server-sftp.html) in the AWS Transfer Family documentation.
**Limitations**
+ This approach isn’t suitable for near real-time or real-time data synchronization.
+ Data can be moved only from Db2 z/OS to Amazon S3, not the other way around.
## Architecture
**Source technology stack**
+ Mainframe running Db2 on z/OS
**Target technology stack**
+ AWS Transfer Family
+ Amazon S3
+ Amazon Athena
+ Amazon QuickSight
+ AWS Glue
+ Amazon Relational Database Service (Amazon RDS)
+ Amazon Aurora
+ Amazon Redshift
**Source and target architecture**
The following diagram shows the process for generating, extracting, and transferring Db2 z/OS data in ASCII CSV format to an S3 bucket.
![\[Data flow from corporate data center to AWS Cloud, showing mainframe extraction and cloud processing steps.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66e6fa1a-1c7d-4b7a-8404-9ba85e433b24/images/87b13e0d-0be9-4462-bdbf-67342334416c.png)
1. A list of tables is selected for data migration from the Db2 catalog.
1. The list is used to drive the generation of unload jobs with the numeric and data columns in the external format.
1. The data is then transferred over to Amazon S3 by using AWS Transfer Family.
1. An AWS Glue extract, transform, and load (ETL) job can transform the data and load it to a processed bucket in the specified format, or AWS Glue can feed the data directly into the database.
1. Amazon Athena and Amazon QuickSight can be used to query and render the data to drive analytics.
The following diagram shows a logical flow of the entire process.
![\[Flowchart showing JCL process with TABNAME, REXXEXEC, and JCL decks steps, including inputs and outputs.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/66e6fa1a-1c7d-4b7a-8404-9ba85e433b24/images/d72f2572-10c9-43f9-b6c9-7e57c9a69d52.png)
1. The first JCL, called TABNAME, will use the Db2 utility DSNTIAUL to extract and generate the list of tables that you plan to unload from Db2. To choose your tables, you must manually adapt the SQL input to select and add filter criteria to include one or more Db2 schemas.
1. The second JCL, called REXXEXEC, will use the a JCL skeleton and the REXX program that is provided to process the Table list created by the JCL TABNAME and generate one JCL per table name. Each JCL will contain one step for unloading the table and another step for sending the file to the S3 bucket by using the SFTP protocol.
1. The last step consists of running the JCL to unload the table and transferring the file to AWS. The entire process can be automated using a scheduler on premises or on AWS.
## Tools
**AWS services**
+ [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is an interactive query service that helps you analyze data directly in Amazon Simple Storage Service (Amazon S3) by using standard SQL.
+ [Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) is a fully managed relational database engine that's built for the cloud and compatible with MySQL and PostgreSQL.
+ [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) is a fully managed extract, transform, and load (ETL) service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams.
+ [Amazon QuickSight](https://docs.aws.amazon.com/quicksight/latest/user/welcome.html) is a cloud-scale business intelligence (BI) service that helps you visualize, analyze, and report your data in a single dashboard.
+ [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html) is a managed petabyte-scale data warehouse service in the AWS Cloud.
+ [Amazon Relational Database Service (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) helps you set up, operate, and scale a relational database in the AWS Cloud.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [AWS Transfer Family](https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-family.html) is a secure transfer service that enables you to transfer files into and out of AWS storage services.
**Mainframe tools**
+ [SSH File Transfer Protocol (SFTP)](https://www.ssh.com/academy/ssh/sftp-ssh-file-transfer-protocol) is a secure file transfer protocol that allows remote login to and file transfer between servers. SSH provides security by encrypting all traffic.
+ [DSNTIAUL](https://www.ibm.com/docs/en/db2-for-zos/11?topic=dpasp-dsntiaul-sample-program) is a sample program provided by IBM for unloading data.
+ [DSNUTILB](https://www.ibm.com/docs/en/db2-for-zos/11?topic=sharing-recommendations-utilities-in-coexistence) is a utilities batch program provided by IBM for unloading data with different options from DSNTIAUL.
+ [z/OS OpenSSH](https://www.ibm.com/docs/en/zos/2.4.0?topic=zbed-zos-openssh) is a port of Open Source Software SSH running on the Unix System Service under the IBM operating system z/OS. SSH is a secure, encrypted connection program between two computers running on a TCP/IP network. It provides multiple utilities, including ssh-keygen.
+ [REXX (Restructured Extended Executor)](https://www.ibm.com/docs/en/zos/2.1.0?topic=guide-learning-rexx-language) script is used to automate JCL generation with the Db2 Unload and SFTP steps.
**Code**
The code for this pattern is available in the GitHub [unloaddb2](https://github.com/aws-samples/unloaddb2-samples) repository.
## Best practices
For the first unload, the generated JCLs should unload the entire table data.
After the first full unload, perform incremental unloads for better performance and cost savings. pdate the SQL query in the template JCL deck to accommodate any changes to the unload process.
You can convert the schema manually or by using a script on Lambda with the Db2 SYSPUNCH as an input. For an industrial process, [AWS Schema Conversion Tool (SCT)](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Source.DB2zOS.html) is the preferred option.
Finally, use a mainframe-based scheduler or a scheduler on AWS with an agent on the mainframe to help manage and automate the entire process.
## Epics
### Set up the S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create the S3 bucket. | For instructions, see [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). | General AWS |
### Set up the Transfer Family server
| Task | Description | Skills required |
| --- | --- | --- |
| Create an SFTP-enabled server. | To open and create an SFTP server on the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/), do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html) | General AWS |
| Create an IAM role for Transfer Family. | To create an AWS Identity and Access Management (IAM) role for Transfer Family to access Amazon S3, follow the instructions in [Create an IAM role and policy](https://docs.aws.amazon.com/transfer/latest/userguide/requirements-roles.html). | AWS administrator |
| Add an Amazon S3 service-managed user. | To add the Amazon S3 service-managed user, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/transfer/latest/userguide/service-managed-users.html#add-s3-user), and use your mainframe user ID. | General AWS |
### Secure the communication protocol
| Task | Description | Skills required |
| --- | --- | --- |
| Create the SSH key. | Under your mainframe USS environment, run the following command.ssh-keygen -t rsa
When prompted for a passphrase, keep it empty. | Mainframe developer |
| Give the right authorization levels to the SSH folder and key files. | By default, the public and private keys will be stored in the user directory `/u/home/username/.ssh`.You must give the authorization 644 to the key files and 700 to the folder.chmod 644 .ssh/id_rsa
chmod 700 .ssh
| Mainframe developer |
| Copy the public key content to your Amazon S3 service-managed user. | To copy the USS-generated public key content, open the [AWS Transfer Family console](https://console.aws.amazon.com/transfer/).[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html) | Mainframe developer |
### Generate the JCLs
| Task | Description | Skills required |
| --- | --- | --- |
| Generate the in-scope Db2 table list. | Provide input SQL to create a list of the tables that are scoped for data migration. This step requires you to specify selection criteria quering the Db2 catalog table SYSIBM.SYSTABLES using a SQL where clause. Filters can be customized to include a specific schema or table names that start with a particular prefix or based on a timestamp for incremental unload. Output is captured in a physical sequential (PS) dataset on the mainframe. This dataset will act as input for the next phase of JCL generation.Before you use the JCL TABNAME (You can rename it if necessary), make the following changes:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**Db2 table list extraction job**
//*
//* UNLOAD ALL THE TABLE NAMES FOR A PARTICULAR SCHEMA
//*
//STEP01 EXEC PGM=IEFBR14
//*
//DD1 DD DISP=(MOD,DELETE,DELETE),
// UNIT=SYSDA,
// SPACE=(1000,(1,1)),
// DSN=.DSN81210.TABLIST
//*
//DD2 DD DISP=(MOD,DELETE,DELETE),
// UNIT=SYSDA,
// SPACE=(1000,(1,1)),
// DSN=.DSN81210.SYSPUNCH
//*
//UNLOAD EXEC PGM=IKJEFT01,DYNAMNBR=20
//SYSTSPRT DD SYSOUT=*
//STEPLIB DD DISP=SHR,DSN=DSNC10.DBCG.SDSNEXIT
// DD DISP=SHR,DSN=DSNC10.SDSNLOAD
// DD DISP=SHR,DSN=CEE.SCEERUN
// DD DISP=SHR,DSN=DSNC10.DBCG.RUNLIB.LOAD
//SYSTSIN DD *
DSN SYSTEM(DBCG)
RUN PROGRAM(DSNTIAUL) PLAN(DSNTIB12) PARMS('SQL') -
LIB('DSNC10.DBCG.RUNLIB.LOAD')
END
//SYSPRINT DD SYSOUT=*
//*
//SYSUDUMP DD SYSOUT=*
//*
//SYSREC00 DD DISP=(NEW,CATLG,DELETE),
// UNIT=SYSDA,SPACE=(32760,(1000,500)),
// DSN=.DSN81210.TABLIST
//*
//SYSPUNCH DD DISP=(NEW,CATLG,DELETE),
// UNIT=SYSDA,SPACE=(32760,(1000,500)),
// VOL=SER=SCR03,RECFM=FB,LRECL=120,BLKSIZE=12
// DSN=.DSN81210.SYSPUNCH
//*
//SYSIN DD *
SELECT CHAR(CREATOR), CHAR(NAME)
FROM SYSIBM.SYSTABLES
WHERE OWNER = ''
AND NAME LIKE '%'
AND TYPE = 'T';
/*
| Mainframe developer |
| Modify the JCL templates. | The JCL templates that are provided with this pattern contain a generic job card and library names. However, most mainframe sites will have their own naming standards for dataset names, library names, and job cards. For example, a specific job class might be required to run Db2 jobs. The Job Entry Subsytem implementations JES2 and JES3 can impose additional changes. Standard load libraries might have a different first qualifier than `SYS1`, which is IBM default. Therefore, customize the templates to account for your site-specific standards before you run them.Make the following changes in the skeleton JCL UNLDSKEL:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**Unload and SFTP JCL skeleton**//&USRPFX.U JOB (DB2UNLOAD),'JOB',CLASS=A,MSGCLASS=A,
// TIME=1440,NOTIFY=&USRPFX
//* DELETE DATASETS
//STEP01 EXEC PGM=IEFBR14
//DD01 DD DISP=(MOD,DELETE,DELETE),
// UNIT=SYSDA,
// SPACE=(TRK,(1,1)),
// DSN=&USRPFX..DB2.PUNCH.&JOBNAME
//DD02 DD DISP=(MOD,DELETE,DELETE),
// UNIT=SYSDA,
// SPACE=(TRK,(1,1)),
// DSN=&USRPFX..DB2.UNLOAD.&JOBNAME
//*
//* RUNNING DB2 EXTRACTION BATCH JOB FOR AWS DEMO
//*
//UNLD01 EXEC PGM=DSNUTILB,REGION=0M,
// PARM=',UNLOAD'
//STEPLIB DD DISP=SHR,DSN=DSNC10.DBCG.SDSNEXIT
// DD DISP=SHR,DSN=DSNC10.SDSNLOAD
//SYSPRINT DD SYSOUT=*
//UTPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSPUN01 DD DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(1,1),RLSE),
// DSN=&USRPFX..DB2.PUNCH.&JOBNAME
//SYSREC01 DD DISP=(NEW,CATLG,DELETE),
// SPACE=(CYL,(10,50),RLSE),
// DSN=&USRPFX..DB2.UNLOAD.&JOBNAME
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
UNLOAD
DELIMITED COLDEL ','
FROM TABLE &TABNAME
UNLDDN SYSREC01
PUNCHDDN SYSPUN01
SHRLEVEL CHANGE ISOLATION UR;
/*
//*
//* FTP TO AMAZON S3 BACKED FTP SERVER IF UNLOAD WAS SUCCESSFUL
//*
//SFTP EXEC PGM=BPXBATCH,COND=(4,LE),REGION=0M
//STDPARM DD *
SH cp "//'&USRPFX..DB2.UNLOAD.&JOBNAME'"
&TABNAME..csv;
echo "ascii " >> uplcmd;
echo "PUT &TABNAME..csv " >>>> uplcmd;
sftp -b uplcmd -i .ssh/id_rsa &FTPUSER.@&FTPSITE;
rm &TABNAME..csv;
//SYSPRINT DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDENV DD *
//STDERR DD SYSOUT=*
| Mainframe developer |
| Generate the Mass Unload JCL. | This step involves running a REXX script under an ISPF environment by using JCL. Provide the list of in-scope tables created on the first step as input for mass JCL generation against the `TABLIST DD` name. The JCL will generate one new JCL per table name in a user-specified partitioned dataset specified against the `ISPFILE DD` name. Allocate this library beforehand. Each new JCL will have two steps: one step to unload the Db2 table into a file, and one step to send the file to the S3 bucket.Make the following changes in the JCL REXXEXEC (you can change the name):[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**Mass JCL generation job**//RUNREXX JOB (CREATEJCL),'RUNS ISPF TABLIST',CLASS=A,MSGCLASS=A,
// TIME=1440,NOTIFY=&SYSUID
//* Most of the values required can be updated to your site specific
//* values using the command 'TSO ISRDDN' in your ISPF session.
//* Update all the lines tagged with //update marker to desired
//* site specific values.
//ISPF EXEC PGM=IKJEFT01,REGION=2048K,DYNAMNBR=25
//SYSPROC DD DISP=SHR,DSN=USER.Z23D.CLIST
//SYSEXEC DD DISP=SHR,DSN=.TEST.REXXLIB
//ISPPLIB DD DISP=SHR,DSN=ISP.SISPPENU
//ISPSLIB DD DISP=SHR,DSN=ISP.SISPSENU
// DD DISP=SHR,DSN=.TEST.ISPSLIB
//ISPMLIB DD DSN=ISP.SISPMENU,DISP=SHR
//ISPTLIB DD DDNAME=ISPTABL
// DD DSN=ISP.SISPTENU,DISP=SHR
//ISPTABL DD LIKE=ISP.SISPTENU,UNIT=VIO
//ISPPROF DD LIKE=ISP.SISPTENU,UNIT=VIO
//ISPLOG DD SYSOUT=*,RECFM=VA,LRECL=125
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSDBOUT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSDBOUT DD SYSOUT=*
//SYSHELP DD DSN=SYS1.HELP,DISP=SHR
//SYSOUT DD SYSOUT=*
//* Input list of tablenames
//TABLIST DD DISP=SHR,DSN=.DSN81210.TABLIST
//* Output pds
//ISPFILE DD DISP=SHR,DSN=.TEST.JOBGEN
//SYSTSIN DD *
ISPSTART CMD(ZSTEPS )
/*
Before you use the REXX script, make the following changes:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transfer-large-scale-db2-z-os-data-to-amazon-s3-in-csv-files.html)**ZSTEPS REXX script**/*REXX - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
/* 10/27/2021 - added new parms to accommodate ftp */
Trace "o"
parse arg usrpfx ftpuser ftpsite
Say "Start"
Say "Ftpuser: " ftpuser "Ftpsite:" ftpsite
Say "Reading table name list"
"EXECIO * DISKR TABLIST (STEM LINE. FINIS"
DO I = 1 TO LINE.0
Say I
suffix = I
Say LINE.i
Parse var LINE.i schema table rest
tabname = schema !! "." !! table
Say tabname
tempjob= "LOD" !! RIGHT("0000" !! i, 5)
jobname=tempjob
Say tempjob
ADDRESS ISPEXEC "FTOPEN "
ADDRESS ISPEXEC "FTINCL UNLDSKEL"
/* member will be saved in ISPDSN library allocated in JCL */
ADDRESS ISPEXEC "FTCLOSE NAME("tempjob")"
END
ADDRESS TSO "FREE F(TABLIST) "
ADDRESS TSO "FREE F(ISPFILE) "
exit 0
| Mainframe developer |
### Run the JCLs
| Task | Description | Skills required |
| --- | --- | --- |
| Perform the Db2 Unload step. | After the JCL generation, you will have as many JCLs as you have tables that need to be unloaded.This story uses a JCL-generated example to explain the structure and the most important steps.No action is required on your part. The following information is for reference only. If your intention is to submit the JCLs that you have generated in the previous step, skip to the *Submit the LODnnnnn JCLs* task.When unloading Db2 data using a JCL with the IBM provided DSNUTILB Db2 utility, you must make sure that the unloaded data does not contain compressed numeric data. To accomplish this, use the DSNUTILB `DELIMITED` parameter.The `DELIMITED` parameter supports unloading the data in CSV format by adding a character as the delimiter and double quotation marks for the text field, removing the padding in the VARCHAR column, and converting all the numeric fields into EXTERNAL FORMAT, including the DATE fields.The following example shows what the unload step in the generated JCL looks like, using the comma character as a delimiter.
UNLOAD
DELIMITED COLDEL ','
FROM TABLE SCHEMA_NAME.TBNAME
UNLDDN SYSREC01
PUNCHDDN SYSPUN01
SHRLEVEL CHANGE ISOLATION UR;
| Mainframe developer, System engineer |
| Perform the SFTP step. | To use the SFTP protocol from a JCL, use the BPXBATCH utility. The SFTP utility can’t access the MVS datasets directly. You can use the copy command (`cp`) to copy the sequential file `&USRPFX..DB2.UNLOAD.&JOBNAME` to the USS directory, where it becomes `&TABNAME..csv`.Run the `sftp` command using the private key (`id_rsa`) and using the RACF user ID as the user name to connect to the AWS Transfer Family IP address.SH cp "//'&USRPFX..DB2.UNLOAD.&JOBNAME'"
&TABNAME..csv;
echo "ascii " >> uplcmd;
echo "PUT &TABNAME..csv " >>>> uplcmd;
sftp -b uplcmd -i .ssh/id_rsa &FTPUSER.@&FTP_TF_SITE;
rm &TABNAME..csv;
| Mainframe developer, System engineer |
| Submit the LODnnnnn JCLs. | The prior JCL has generated all LODnnnnn JCL tables that need to be unloaded, transformed into CSV, and transferred to the S3 bucket.Run the `submit` command on all the JCLs that have been generated. | Mainframe developer, System engineer |
## Related resources
For more information about the different tools and solutions used in this document, see the following:
+ [z/OS OpenSSH User’s Guide](https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc276806/$file/foto100_v2r4.pdf)
+ [Db2 z/OS – Sample UNLOAD control statements](https://www.ibm.com/docs/en/db2-for-zos/11?topic=unload-sample-control-statements)
+ [Db2 z/OS – Unloading delimited files](https://www.ibm.com/docs/en/db2-for-zos/11?topic=unload-unloading-delimited-files)
+ [Transfer Family – Create an SFTP-enabled server](https://docs.aws.amazon.com/transfer/latest/userguide/create-server-sftp.html)
+ [Transfer Family – Working with service-managed users](https://docs.aws.amazon.com/transfer/latest/userguide/service-managed-users.html)
## Additional information
After you have your Db2 data on Amazon S3, you have many ways to develop new insights. Because Amazon S3 integrates with AWS data analytics services, you can freely consume or expose this data on the distributed side. For example, you can do the following:
+ Build a [data lake on Amazon S3](https://aws.amazon.com/products/storage/data-lake-storage/), and extract valuable insights by using query-in-place, analytics, and machine learning tools without moving the data.
+ Initiate a [Lambda function](https://aws.amazon.com/lambda/) by setting up a post-upload processing workflow that is integrated with AWS Transfer Family.
+ Develop new microservices for accessing the data in Amazon S3 or in [fully managed database](https://aws.amazon.com/free/database/?trk=ps_a134p000007CdNEAA0&trkCampaign=acq_paid_search_brand&sc_channel=PS&sc_campaign=acquisition_FR&sc_publisher=Google&sc_category=Database&sc_country=FR&sc_geo=EMEA&sc_outcome=acq&sc_detail=amazon%20relational%20database%20service&sc_content=Relational%20Database_e&sc_matchtype=e&sc_segment=548727697660&sc_medium=ACQ-P|PS-GO|Brand|Desktop|SU|Database|Solution|FR|EN|Text&s_kwcid=AL!4422!3!548727697660!e!!g!!amazon%20relational%20database%20service&ef_id=CjwKCAjwzt6LBhBeEiwAbPGOgcGbQIl1-QsbHfWTgMZSSHEXzSG377R9ZyK3tCcbnHuT45L230FufxoCeEkQAvD_BwE:G:s&s_kwcid=AL!4422!3!548727697660!e!!g!!amazon%20relational%20database%20service) by using [AWS Glue](https://aws.amazon.com/glue/), which is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development.
In a migration use case, because you can transfer any data from the mainframe to S3, you can do the following:
+ Retire physical infrastructure, and create a cost-effective data archival strategy with Amazon S3 Glacier and S3 Glacier Deep Archive.
+ Build scalable, durable, and secure backup and restore solutions with Amazon S3 and other AWS services, such as S3 Glacier and Amazon Elastic File System (Amazon EFS), to augment or replace existing on-premises capabilities.
# Transform Easytrieve to modern languages by using AWS Transform custom
*Shubham Roy, Subramanyam Malisetty, and Harshitha Shashidhar, Amazon Web Services*
## Summary
This pattern provides prescriptive guidance for faster and lower-risk transformation of mainframe Broadcom [Easytrieve Report Generator](https://techdocs.broadcom.com/us/en/ca-mainframe-software/devops/ca-easytrieve-report-generator/11-6.html) (EZT) workloads using [AWS Transform custom](https://aws.amazon.com/transform/custom/) language-to-language transformation. It addresses the challenges of modernizing niche and proprietary mainframe EZT workloads that are commonly used for batch data processing and report generation. The pattern replaces expensive, lengthy, and error-prone migration approaches that rely on proprietary tooling and rare mainframe expertise with an agentic AI automated solution you create on AWS Transform.
This pattern provides a ready-to -use custom transformation definition for EZT transformation. The definition uses multiple transformation inputs:
+ EZT business rules extracted using [AWS Transform for mainframe](https://aws.amazon.com/transform/mainframe/)
+ EZT programming reference documentation
+ EZT source code
+ Mainframe input and output datasets
AWS Transform custom uses these inputs to generate functionally equivalent applications in modern target languages, such as Java or Python.
The transformation process uses intelligent test execution, automated debugging, and iterative fix capabilities to validate functional equivalence against expected outputs. It also supports continual learning, enabling the custom transformation definition to improve accuracy and consistency across successive transformations. Using this pattern, organizations can reduce migration effort and risk, address niche mainframe technical debt, and modernize EZT workloads on AWS to improve agility, reliability, security, and innovation.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ A mainframe EZT workload with input and output data
**Limitations**
*Scope limitations *
+ **Language support** – Only EZT-to-Java transformation is supported for this specific transformation pattern.
+ **Out of scope** – Transformation of other mainframe programming languages requires a new custom transformation definition in AWS Transform custom.
*Process limitations *
+ **Validation dependency** – Without baseline output data the transformation cannot be validated.
+ **Proprietary logic** – Highly specific, custom-developed utilities require additional user documentation and reference materials in order to be correctly interpreted by the AI agent.
*Technical limitations *
+ **Service limits** – For AWS Transform custom service limits and quotas see [AWS Transform User Guide - Quotas](https://docs.aws.amazon.com/transform/latest/userguide/transform-limits.html) and the [AWS General Reference - Transform Quotas](https://docs.aws.amazon.com/general/latest/gr/aws-transform.html).
**Product versions**
+ AWS Transform CLI – Latest version
+ Node.js – version 20 or later
+ Git – Latest version
+ Target environment
+ Java – version 17 or later
+ Spring Boot – version 3.x is the primary target for refactored applications
+ Maven – version 3.6 or later
## Architecture
**Source technology stack**
+ **Operating system** – IBM z/OS
+ **Programming language** – Easytrieve, Job control language (JCL)
+ **Database** – IBM DB2 for z/OS, Virtual Storage Access Method (VSAM), Mainframe flat files
**Target technology stack**
+ **Operating system** – Amazon Linux
+ **Compute** – Amazon Elastic Compute Cloud (Amazon EC2)
+ **Programming language** – Java
+ **Database** Amazon Relational Database Service (Amazon RDS)
**Target architecture**
![\[target architecture diagram for using AWS Transform custom to transform EZT to modern code.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/71f15422-42cb-4c7e-94fa-051a4f130445/images/eb89eed0-dd55-485c-a433-9869162eaad9.png)
**Workflow**
This solution uses an AWS Transform custom language-to-language migration transformation pattern to modernize mainframe Easytrieve (EZT) applications to Java through a four-step automated workflow.
*Step 1 – Provide your legacy code to AWS Transform for Mainframe, which:*
+ Analyzes the code
+ Extracts the high-level business logic
+ Extracts the detailed business logic.
*Step 2 – Create a folder with the required inputs:*
+ EZT business rules extracted using AWS Transform for mainframe
+ EZT programming reference documentation
+ EZT source code
+ Mainframe input and output datasets
*Step 3 – Create and run a custom transformation definition*
1. Use the AWS Transform CLI to describe transformation objectives in natural language. AWS Transform custom analyzes the BRE, source code, and EZT programming guides to generate a custom transformation definition for developer review and approval.
1. Then, invoke the AWS Transform CLI with the project source code. AWS Transform custom creates transformation plans, converts EZT to Java upon approval, generates supporting files, builds the executable JAR, and validates exit criteria.
1. Use the validation agent to test the functional equivalence against mainframe output. The Self-Debugger Agent autonomously fixes issues. Final deliverables include validated Java code and HTML validation reports.
**Automation and scale**
+ Agentic AI multi-mode execution architecture – AWS Transform custom leverages agentic AI with 3 execution modes (conversational, interactive, full automation) to automate complex transformation tasks including code analysis, refactoring, transformation planning and testing.
+ Adaptive learning feedback system – The platform implements continuous learning mechanisms through code sample analysis, documentation parsing, and developer feedback integration with versioned transformation definitions.
+ Concurrent application processing architecture – The system enables distributed parallel execution of multiple application transformation operations simultaneously across scalable infrastructure.
## Tools
**AWS services **
+ [AWS Transform custom](https://docs.aws.amazon.com/transform/latest/userguide/custom.html) is an agentic AI service is used to transform legacy EZT applications into modern programming languages.
+ [AWS Transform](https://docs.aws.amazon.com/transform/latest/userguide/what-is-service.html) uses agentic AI to help you accelerate the modernization of legacy workloads, such as .NET, mainframe, and VMware workloads.
+ [AWS Transform for mainframe ](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe.html)is used to analyze legacy EZT applications to extract embedded business logic and generate comprehensive business rule documentation, including logic summaries, acronym definitions, and structured knowledge bases. These serve as input data for AWS Transform custom.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. Amazon S3 serves as the primary storage service for AWS Transform custom for storing transformation definitions, code repositories, and processing results.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. IAM provides the security framework for AWS Transform custom, managing permissions and access control for transformation operations.
**Other tools**
+ [AWS Transform CLI](https://docs.aws.amazon.com/transform/latest/userguide/custom-command-reference.html) is the command-line interface for AWS Transform custom, enabling developers to define, execute, and manage custom code transformations through natural language conversations and automated execution modes. AWS Transform custom supports both interactive sessions (atx custom def exec) and autonomous transformations for scalable modernization of codebases.
+ [Git](https://git-scm.com/doc) version control system used for branch protection, change tracking, and rollback capabilities during automated fix application.
+ [Java](https://www.java.com/en/) is the programming language and development environment used in this pattern.
**Code repository**
The code for this pattern is available in [Easytrieve to Modern Languages Transformation with AWS Transform Custom](https://github.com/aws-samples/sample-mainframe-easytrieve-transform?tab=readme-ov-file#easytrieve-to-modern-languages-transformation-with-aws-transform-custom) on GitHub.
## Best practices
+ Establish standardized project structure – Create a four-folder structure (source-code, bre-doc, input-data, output-data), validate completeness, and document contents before transformation.
+ Use baseline files for validation – Use production baseline input files, perform byte-by-byte comparison with baseline output, accept zero tolerance for deviations.
+ Use all available reference documents – To increase accuracy of transformation provide all available reference documents such as business requirements and coding checklists.
+ Provide input for quality improvement – AWS Transform custom automatically extracts learnings from transformation executions (developer feedback, code issues) and creates knowledge items for them. after each successful transformation review knowledge items and approve the one that you would like to be used in future executions. This improves the quality of future transformations.
## Epics
### Generate a business rule extract (BRE)
| Task | Description | Skills required |
| --- | --- | --- |
| Configure AWS Transform for mainframe. | Set up the environment and required AWS Identity and Access Management (IAM) permissions to support mainframe modernization workflows. For more information see [Transformation of mainframe applications](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html) in AWS documentation. | App developer |
| Generate Business Rule Extract (BRE) documentation. | Extract business logic from source EZT or COBOL code to generate functional documentation. For instructions on how to initiate the extraction process and review the output, see [Extract business logic](https://docs.aws.amazon.com/transform/latest/userguide/transform-app-mainframe-workflow.html#transform-app-mainframe-workflow-extract-business-logic) in the AWS Transform documentation. | App developer |
### Set up AWS Transform custom
| Task | Description | Skills required |
| --- | --- | --- |
| Provision the infrastructure for AWS Transform custom. | Deploy the production-ready infrastructure required to host a secure transformation environment. This includes a private Amazon EC2 instance configured with the necessary tools, IAM permissions, and network settings for converting Easytrieve code. To provision the environment using infrastructure as code (IaC), follow the deployment instructions in the [Easytrieve to Modern Languages Transformation with AWS Transform Custom](https://github.com/aws-samples/sample-mainframe-easytrieve-transform) GitHub repository. | App developer, AWS administrator |
| Prepare input materials for transformation. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
### Create a custom transformation definition
| Task | Description | Skills required |
| --- | --- | --- |
| Create transformation definition. | Follow these steps to create the custom transformation definition for EZT to Java transformation with functional validation.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
| Publish transformation definition. | After review and validation of the transformation definition you can publish it to the AWS Transform custom registry with a natural language prompt, providing a definition name such as *Easytrieve-to-Java-Migration*. | App developer |
### Prepare baseline data for validation.
| Task | Description | Skills required |
| --- | --- | --- |
| Review the transformation validation summary. | Before executing the AWS Transform custom transformation, validate that the `input-data` folder contains the required data files captured before execution of the mainframe batch job. After the mainframe batch job execution, ensure that the `output-data` folder captures the resulting files. All files are in Sequential/Text/DB2 format using EBCDIC encoding based on execution requirements.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
| Run the custom transformation job. | Execute the AWS Transform CLI command, choosing the non-interactive or the interactive option::# Non-interactive execution (fully autonomous):
atx custom def exec \
--transformation-name "Easytrieve-to-Java-Migration" \
--code-repository-path ~/root/transform-workspace/mainframe-source/source-code \
--build-command "mvn clean install" \
--non-interactive \
--trust-all-tools \
# Interactive execution (with human oversight):
atx custom def exec \
-n "Easytrieve-to-Java-Migration" \
-p ~/root/transform-workspace/mainframe-source/source-code \
-c "mvn clean install"
# Resume interrupted execution:
atx -resume
# OR
atx --conversation-id
AWS Transform automatically validates through build/test commands during transformation execution. | App developer |
### Validate and deliver tested code
| Task | Description | Skills required |
| --- | --- | --- |
| Review the transformation validation summary. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) | App developer |
| Access validation reports. | Enter these commands to review the detailed validation artifacts:# Full validation report
cat ~/.aws/atx/custom/$LATEST_SESSION/artifacts/validation_report.html
# Generated code location
ls ~/.aws/atx/custom/$LATEST_SESSION/generated/
# Execution logs
cat ~/.aws/atx/custom/$LATEST_SESSION/logs/execution.log
| App developer |
| Enable knowledge items for continuous learning. | Improve future transformation accuracy by promoting suggested knowledge items to your persistent configuration. After a transformation, the agent stores identified patterns and mapping rules in your local session directory. To review and apply these learned items, run these commands on your Amazon EC2 instance:# List all knowledge items for a specific transformation definition
atx custom def list-ki -n
# Retrieve the details of a specific knowledge item
atx custom def get-ki -n --id
# Update the status of a knowledge item (ENABLED or DISABLED)
atx custom def update-ki-status -n --id --status ENABLED
# Update the knowledge item configuration to enable auto-approval
atx custom def update-ki-config -n --auto-enabled TRUE
| App developer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| *Input and output path configuration*Input files are not being read, or output files are not being written correctly. | Specify the complete directory path where input files are stored and clearly indicate the location where output should be written. Ensure proper access permissions are configured for these directories. Best practices include using absolute paths rather than relative paths to avoid ambiguity and verifying that all specified paths exist with appropriate read/write permissions. |
| *Resuming interrupted executions*Execution was interrupted or needs to be continued from where stopped | You can resume execution from where you left off by providing the conversation ID in the CLI command.Find the conversation ID in the logs of your previous execution attempt. |
| *Resolving memory constraints*Out of memory error occurs during execution. | You can ask AWS Transform to share the current in-memory JVM size and then increase the memory allocation based on this information. This adjustment helps accommodate larger processing requirements.Consider breaking large jobs into smaller batches if memory constraints persist after adjustments. |
| *Addressing output file discrepancies*Output files don't match expectations, and AWS Transform indicates no further changes are possible. | Provide specific feedback and technical reasons explaining why the current output is incorrect. Include additional technical or business documentation to support your requirements. This detailed context helps AWS Transform correct the code to generate the proper output files. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/transform-easytrieve-modern-languages.html) |
## Related resources
+ [AWS Transform custom documentation](https://docs.aws.amazon.com/transform/latest/userguide/custom.html)
+ [Easytrieve Report Generator 11.6](https://techdocs.broadcom.com/us/en/ca-mainframe-software/devops/ca-easytrieve-report-generator/11-6/getting-started.html)
# More patterns
**Topics**
+ [Deploy the Security Automations for AWS WAF solution by using Terraform](deploy-the-security-automations-for-aws-waf-solution-by-using-terraform.md)
+ [Replicate mainframe databases to AWS by using Precisely Connect](replicate-mainframe-databases-to-aws-by-using-precisely-connect.md)