Mainframe modernization: DevOps on AWS with Micro Focus

Created by Kevin Yung (AWS)

Summary

Customer challenges

Organizations that run core applications on mainframe hardware usually encounter a few challenges when the hardware needs to scale up to meet the demands of digital innovations. These challenges include the following constraints. 

Pattern outcomes

To address these challenges, Amazon Web Services (AWS) and Micro Focus, an AWS Partner Network (APN) Partner, have collaborated to create this pattern. The solution is designed to help you achieve the following outcomes.

Technologies summary

In this pattern, the target stack contains the following components.

Prerequisites and limitations

Prerequisites 

Limitations 

Product versions

Architecture

Mainframe environment

In conventional mainframe development, the developers need to use mainframe hardware to develop and test programs. They face capacity limitations, for example restricted million instructions per second (MIPS) for the dev/test environment, and they must rely on the tools that are available on the mainframe computers.

In many organizations, mainframe development follows the waterfall development methodology, with teams relying on long cycles to release changes. These release cycles are usually longer than digital product development.   

The following diagram shows multiple mainframe projects sharing mainframe hardware for their development. In mainframe hardware, it is expensive to scale out a development and test environment for more projects.

AWS architecture  

This pattern extends mainframe development to the AWS Cloud. First, it uses Micro Focus AccuRev SCM to host the mainframe source code on AWS. Then it makes Micro Focus Enterprise Developer and Micro Focus Enterprise Test Server available for building and testing the mainframe code on AWS. 

The following sections describe the pattern's three major components.

1. SCM

In AWS, the pattern uses Micro Focus AccuRev to create a set of SCM workspaces and version control for the mainframe source code. Its stream-based architecture enables parallel mainframe development for multiple teams. To merge a change, AccuRev uses the promote concept. To add that change to other workspaces, AccuRev uses the update concept.

At the project level, each team can create one or more streams in AccuRev to track project level changes. These are called project streams. These project streams are inherited from the same parent stream. The parent stream is used to merge the changes from different project streams.

Each project stream can promote code to AccuRev, and a promote post trigger is set up to initiate the AWS CI/CD pipeline. The successful build for a project stream change can be promoted to its parent stream for more regression tests.  

Usually, the parent stream is called the system integration stream. When there is a promotion from a project stream to a system integration stream, a post promotion trigger initiates another CI/CD pipeline to run regression tests.

In addition to mainframe code, this pattern includes AWS CloudFormation templates, Systems Manager Automation documents, and scripts. Following infrastructure-as-code best practices, they are version-controlled in AWS CodeCommit. 

If you need to synchronize mainframe code back to a mainframe environment for deployment, Micro Focus provides the Enterprise Sync solution, which synchronizes code from the AccuRev SCM back to the mainframe SCM.

2. Developer and test environments

In a large organization, scaling more than a hundred or even more than a thousand mainframe developers is challenging. To address this constraint, the pattern uses Amazon EC2 Windows instances for development. On the instances, Micro Focus Enterprise Developer for Eclipse tools are installed. The developer can perform all mainframe code test and debugging locally on the instance. 

AWS Systems Manager State Manager and Automation documents are used to automate the developer instance provisioning. The average time to create a developer instance is within 15 minutes. The following software and configurations are prepared.

In the development cycle, developers use the EC2 instance to develop and test mainframe code locally. When the local changes are tested successfully, developers promote the change into the AccuRev server.  

3. CI/CD pipelines

In the pattern, CI/CD pipelines are used for integration tests and regression tests before deployment to the production environment. 

As explained in the SCM section, AccuRev uses two types of streams: a project stream and an integration stream. Each stream is hooked up with CI/CD pipelines. To perform the integration between the AccuRev server and AWS CodePipeline, the pattern uses AccuRev post promotion script to create an event to initiate CI/CD.

For example, when a developer promotes a change to a project stream in AccuRev, it initiates a post promotion script to run in AccuRev Server. Then the script uploads the metadata of the change into an Amazon Simple Storage Service (Amazon S3) bucket to create an Amazon S3 event. This event will initiate a CodePipeline configured pipeline to run. 

The same event-initiating mechanism is used for the integration stream and its associated pipelines. 

In the CI/CD pipeline, CodePipeline uses CodeBuild with the Micro Focus AccuRev Linux client container to check out the latest code from the AccuRev streams. Then the pipeline starts CodeBuild to use the Micro Focus Enterprise Developer Windows container to compile the source code, and to use the Micro Focus Enterprise Test Server Windows container in CodeBuild to test mainframe applications.

The CI/CD pipelines are built using AWS CloudFormation templates, and the blueprint will be used for new projects. By using the templates, it takes less than an hour for a project to create a new CI/CD pipeline in AWS.

To scale your mainframe test capability on AWS, the pattern builds out the Micro Focus DevOps test suite, Micro Focus Verastream and Micro Focus UFT server. By using the modern DevOps tools, you can run as many tests on AWS as you need.

An example mainframe development environment with Micro Focus on AWS is shown in the following diagram.

Target technology stack 

This section provides a closer look at the architecture of each component in the pattern.

1. Source code repository – AccuRev SCM 

Micro Focus AccuRev SCM is set up to manage mainframe source code versions. For high availability, AccuRev supports primary and replica modes. Operators can fail over to the replica when performing maintenance on the primary node. 

To speed up the response of the CI/CD pipeline, the pattern uses Amazon CloudWatch Events to detect source code changes and initiate the start of the pipeline.

When the pipeline runs, it kicks off a CodeBuild project to use an AccuRev Linux client container to check out the latest mainframe code from an associated AccuRev stream.   

The following diagram shows an AccuRev Server setup.

2. Enterprise Developer template 

The pattern uses Amazon EC2 templates to simplify creation of the developer instance. By using State Manager, it can apply software and license settings to EC2 instances consistently.

The Amazon EC2 template builds in its VPC context settings and default instance settings, and it follows enterprise tagging requirements. By using a template, a team can create their own new development instances. 

When a developer instance starts, by associating with tags, Systems Manager uses State Manager to apply automation. The automation includes the following general steps.

The following diagram shows an Enterprise developer instance created by the Amazon EC2 template, with software and configuration applied to the instance by State Manager. Enterprise developer instances connect to Micro Focus License Manager to activate their license.

3.  CI/CD pipelines

As explained in AWS architecture section, in the pattern, there are project-level CI/CD pipelines and system integration pipelines. Each mainframe project team creates a pipeline or multiple CI/CD pipelines for building the programs that they are developing in a project. These project CI/CD pipelines check out source code from an associated AccuRev stream. 

In a project team, developers promote their code in the associated AccuRev stream. Then the promotion initiates the project pipeline to build the code and run and integration tests. 

Each project CI/CD pipeline uses CodeBuild projects with the Micro Focus Enterprise Developer tool Amazon ECR image and Micro Focus Enterprise Test Server tool Amazon ECR image. 

CodePipeline and CodeBuild are used to create the CI/CDs pipelines. Because CodeBuild and CodePipeline have no upfront fees or commitments, you pay only for what you use. Compared to mainframe hardware, the AWS solution greatly reduces hardware provisioning lead time and lowers the cost of your testing environment.

In modern development, multiple test methodologies are used. For example, test-driven development (TDD), BDD, and Robot Framework. With this pattern, developers can use these modern tools for mainframe testing. For example, by using x3270, py3270 and the Behave python test tool, you can define an online application's behavior. You can also use build mainframe 3270 robot framework in these CI/CD pipelines.

The following diagram shows the team stream CI/CD pipeline. 

The following diagram shows the project CI/CD test report produced by CodePipeline in Mainframe3270 Robot Framework.

The following diagram shows the project CI/CD test report produced by CodePipeline in Py3270 and Behave BDD.

After project level tests are passed successfully, the tested code is manually promoted to the integration stream in AccuRev SCM. You can automate this step after the teams have a confidence on the tests coverage of their project pipeline.

When code is promoted, the system integration CI/CD pipeline checks out the merged code and performs regression tests. The merged code is promoted from all parallel project streams.

Depending on how fine grain the test environment are required, customers can have more system integration CI/CD pipelines in different environment, for example UAT, Pre-Production. 

In the pattern, the tools used in the system integration pipeline are Micro Focus Enterprise Test Server, Micro Focus UFT Server, and Micro Focus Verastream. All these tools can be deployed into the Docker container and used with CodeBuild.

After successfully testing of the mainframe programs, the artifact is stored, with version control, in an S3 bucket. 

The following diagram shows a system integration CI/CD pipeline.

After the artifact has been successfully tested in the system integration CI/CD pipelines, it can be promoted for production deployment. 

If you need to deploy source code back to the mainframe, Micro Focus offers the Enterprise Sync solution to synchronize source code from AccuRev back to Mainframe Endeavour.

The following diagram shows a production CI/CD pipeline deploying the artifact into Micro Focus Enterprise Servers. In this example, CodeDeploy orchestrates the deployment of the tested mainframe artifact into Micro Focus Enterprise Server.

In addition to the architecture walkthough of the CI/CD pipeline, you can also read the AWS DevOps blog post Automate thousands of mainframe tests on AWS with the Micro Focus Enterprise Suite for more information on testing mainframe applications in CodeBuild and CodePipeline. Refer to the blog post for the best practices and details of doing mainframe tests on AWS.

Tools

Tools

AWS automation tools

Micro Focus tools

Other tools

Epics

Related resources

References

AWS Marketplace

AWS Quick Start