Convert mainframe files from EBCDIC format to character-delimited ASCII format in Amazon S3 using AWS Lambda

Create the environment variables.

Copy the following environment variables to a text editor, and then replace the <placeholder> values in the following example with your resource values:

bucket=<your_bucket_name>
account=<your_account_number>
region=<your_region_code>

Note: You will create references to your S3 bucket, AWS account, and AWS Region later.

To define environment variables, open the CloudShell console, and then copy and paste your updated environment variables onto the command line.

Note: You must repeat this step every time the CloudShell session restarts.

Create a working folder.

To simplify the resource clean-up process later on, create a working folder in CloudShell by running the following command:

mkdir workdir; cd workdir

Note: You must change the directory to the working directory (workdir) every time you lose a connection to your CloudShell session.

Create a trust policy for the Lambda function.

The EBCDIC converter runs in a Lambda function. The function must have an IAM role. Before you create the IAM role, you must define a trust policy document that enables resources to assume that policy.

From the CloudShell working folder, create a policy document by running the following command:

E2ATrustPol=$(cat <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF
)
printf "$E2ATrustPol" > E2ATrustPol.json

Create the IAM role for Lambda conversion.

To create an IAM role, run the following AWS CLI command from the CloudShell working folder:

aws iam create-role --role-name E2AConvLambdaRole --assume-role-policy-document file://E2ATrustPol.json

Create the IAM policy document for the Lambda function.

The Lambda function must have read-write access to the S3 bucket and write permissions for Amazon CloudWatch Logs.

To create an IAM policy, run the following command from the CloudShell working folder:

E2APolicy=$(cat <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Logs",
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogStream",
                "logs:CreateLogGroup"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:*",
                "arn:aws:logs:*:*:log-group:*:log-stream:*"
            ]
        },
        {
            "Sid": "S3",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::%s/*",
                "arn:aws:s3:::%s"
            ]
        }
    ]
}
EOF
)
printf "$E2APolicy" "$bucket" "$bucket" > E2AConvLambdaPolicy.json

Attach the IAM policy document to the IAM role.

To attach the IAM policy to the IAM role, run the following command from your CloudShell working folder:

aws iam put-role-policy --role-name E2AConvLambdaRole --policy-name E2AConvLambdaPolicy --policy-document file://E2AConvLambdaPolicy.json

Download the EBCDIC conversion source code.

From the CloudShell working folder, run the following command to download the mainframe-data-utilities source code from GitHub:

git clone https://github.com/aws-samples/mainframe-data-utilities.git mdu

Create the ZIP package.

From the CloudShell working folder, run the following command to create the ZIP package that creates the Lambda function for EBCDIC conversion:

cd mdu; zip ../mdu.zip *.py; cd ..

Create the Lambda function.

From the CloudShell working folder, run the following command to create the Lambda function for EBCDIC conversion:

aws lambda create-function \
--function-name E2A \
--runtime python3.9 \
--zip-file fileb://mdu.zip \
--handler extract_ebcdic_to_ascii.lambda_handler \
--role arn:aws:iam::$account:role/E2AConvLambdaRole \
--timeout 10 \
--environment "Variables={layout=$bucket/layout/}"

Note: The environment variable layout tells the Lambda function where the JSON metadata resides.

Create the resource-based policy for the Lambda function.

From the CloudShell working folder, run the following command to allow your Amazon S3 event notification to trigger the Lambda function for EBCDIC conversion:

aws lambda add-permission \
--function-name E2A \
--action lambda:InvokeFunction \
--principal s3.amazonaws.com \
--source-arn arn:aws:s3:::$bucket \
--source-account $account \
--statement-id 1

Create the configuration document for the Amazon S3 event notification.

The Amazon S3 event notification initiates the EBCDIC conversion Lambda function when files are placed in the input folder.

From the CloudShell working folder, run the following command to create the JSON document for the Amazon S3 event notification:

{
"LambdaFunctionConfigurations": [
    {
      "Id": "E2A",
      "LambdaFunctionArn": "arn:aws:lambda:%s:%s:function:E2A",
      "Events": [ "s3:ObjectCreated:Put" ],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "input/"
            }
          ]
        }
      }
    }
  ]
}
EOF
)
printf "$S3E2AEvent" "$region" "$account" > S3E2AEvent.json

Create the Amazon S3 event notification.

From the CloudShell working folder, run the following command to create the Amazon S3 event notification:

aws s3api put-bucket-notification-configuration --bucket $bucket --notification-configuration file://S3E2AEvent.json

Parse the COBOL copybook.

From the CloudShell working folder, run the following command to parse a sample COBOL copybook into a JSON file (which defines how to read and slice the data file properly):

python3       mdu/parse_copybook_to_json.py \
-copybook     mdu/LegacyReference/COBKS05.cpy \
-output       CLIENT.json \
-output-s3key CLIENT.ASCII.txt \
-output-s3bkt $bucket \
-output-type  s3 \
-print        25

Add the transformation rule.

The sample data file and its corresponding COBOL copybook is a multi-layout file. This means that the conversion must slice data based on certain rules. In this case, bytes on position 3 and 4 in each row define the layout.

From the CloudShell working folder, edit the CLIENT.json file and change the contents from "transf-rule": [], to the following:

"transf-rule": [
{
"offset": 4,
"size": 2,
"hex": "0002",
"transf": "transf1"
},
{
"offset": 4,
"size": 2,
"hex": "0000",
"transf": "transf2"
}
],

Upload the JSON metadata to the S3 bucket.

From the CloudShell working folder, run the following AWS CLI command to upload the JSON metadata to your S3 bucket:

aws s3 cp CLIENT.json s3://$bucket/layout/CLIENT.json

Send the EBCDIC file to the S3 bucket.

From the CloudShell working folder, run the following command to send the EBCDIC file to the S3 bucket:

aws s3 cp mdu/sample-data/CLIENT.EBCDIC.txt s3://$bucket/input/

Note: We recommend that you set different folders for input (EBCDIC) and output (ASCII) files to avoid calling the Lambda conversion function again when the ASCII file is uploaded to the S3 bucket.

Check the output.

From the CloudShell working folder, run the following command to check if the ASCII file is generated in your S3 bucket:

awss3 ls s3://$bucket/

Note: The data conversion can take several seconds to happen. We recommend that you check for the ASCII file a few times.

After the ASCII file is available, run the following command to download the file from the S3 bucket to the current folder:

aws s3 cp s3://$bucket/CLIENT.ASCII.txt .

Check the ASCII file content:

head CLIENT.ASCII.txt

(Optional) Prepare the variables and folder.

If you lose connection with CloudShell, reconnect and then run the following command to change the directory to the working folder:

cd workdir

Ensure that the environment variables are defined:

bucket=<your_bucket_name>
account=<your_account_number>
region=<your_region_code>

Remove the notification configuration for the bucket.

From the CloudShell working folder, run the following command to remove the Amazon S3 event notification configuration:

aws s3api put-bucket-notification-configuration \
--bucket=$bucket \
--notification-configuration="{}"

Delete the Lambda function.

From the CloudShell working folder, run the following command to delete the Lambda function for the EBCDIC converter:

awslambdadelete-function--function-nameE2A

Delete the IAM role and policy.

From the CloudShell working folder, run the following command to remove the EBCDIC converter role and policy:

aws iam delete-role-policy --role-name
E2AConvLambdaRole --policy-name
E2AConvLambdaPolicy

aws iam delete-role --role-name
E2AConvLambdaRole

Delete the files generated in the S3 bucket.

From the CloudShell working folder, run the following command to delete the files generated in the S3 bucket:

aws s3 rm s3://$bucket/layout --recursive
aws s3 rm s3://$bucket/input --recursive
aws s3 rm s3://$bucket/CLIENT.ASCII.txt

Delete the working folder.

From the CloudShell working folder, run the following command to remove workdir and its contents:

cd ..; rm -Rf workdir