AWS Blu Age Blusam

On mainframe systems (referred to in the following topic as "legacy"), business data is often stored using VSAM (Virtual Storage Access Method). The data is stored in "records" (byte arrays), belonging to a "data set".

There are four data set organizations:

When modernizing legacy applications, using AWS Blu Age refactoring approach, modernized applications are no longer intended to access VSAM stored data, while preserving the data access logic. The Blusam component is the answer: it allows importing data from legacy VSAM data sets exports, provides an API for the modernized application to manipulate them along with a dedicated administration web application. See AWS Blu Age Blusam Administration Console.

The Blusam API makes it possible to preserve data access logic (sequential, random, and relative reads; insert, update, and delete records), whereas the components architecture, relying on a mix of caching strategies and RDBMS-based storage, permits high throughput I/O operations with limited resources.

Blusam infrastructure

Blusam relies on PostgreSQL RDBMS for data sets storage, both for raw records data and keys indexes (when applicable). The favorite option is to use the Amazon Aurora PostgreSQL compatible engine. The examples and illustrations in this topic are based on this engine.

Caching

In addition to the storage itself, Blusam operates faster when coupled to a cache implementation.

Two cache engines are currently supported, EhCache and Redis, each having its own use-case:

Locking

To deal with concurrent access to data sets and records, Blusam relies on a configurable locking system. Locking can be applied to both levels: data sets and records:

Configuring the Blusam locking system should be done accordingly to the cache configuration:

Blusam intrinsics and data migration from legacy

Storing data sets: records and indexes

Each legacy data set, when imported to Blusam, will be stored to a dedicated table; each row of the table represents a record, using two columns:

See for example the content of a KSDS data set used in the CardDemo application:

Regarding the data set metadata and keys indexes: each data set is associated with two rows in the table named metadata. This is the default naming convention. To learn how to customize it, see Blusam configuration.

Additionally, large/very large data sets indexes (if applicable) are stored using a pagination mechanism; index pages binary serializations are stored as rows of a dedicated table (one table per data set key). Each page of indexes is stored in a row, having the following columns:

The table name is a concatenation of the data set name and the key internal name, which contains information about the key, such as the key offset, whether the key accepts duplicates (set to true to allow duplicates), and the key length. For example, consider a data set named "AWS_LARGE_KSDS" that has the following two defined keys:

In this case, the following tables store the indexes related to the two keys.

Optimizing I/O throughput using write-behind mechanism

To optimize insert / update / delete operations performances, the Blusam engine relies on a configurable write-behind mechanism. The mechanism is built upon a pool of dedicated threads that deal with persistence operations using bulk update queries, to maximize I/O throughput towards the Blusam storage.

The Blusam engine collects all update operations done on records by applications and build records lots that are being dispatched for treatment to the dedicated threads. The lots are then being persisted to the Blusam storage, using bulk update queries, avoiding the usage of atomic persistence operations, ensuring the best possible usage of network bandwidth.

The mechanism uses a configurable delay (defaults to one second) and a configurable lot size (defaults to 10000 records). The build persistence queries are executed as soon as the first of the two following conditions is met:

To learn how to configure the write-behind mechanism, see Optional properties.

Picking up the proper storage scheme

As shown in the previous section, the way data sets are being stored depends on their "weight". But what is considered as small, medium or large for a data set? When to pick the paginated storage strategy rather than the regular one?

The answer to that question depends on the following.

When using non-paginated indexes storage scheme, the full key indexes and records sizes collections will be loaded into the server memory at data set opening time, for each data set. In addition, if caching is involved, all data set records might be pre-loaded into cache with the regular approach, which might lead to memory resource exhaustion on the cache infrastructure side.

Depending on the number of defined keys, the length of the key values, the number of records and the number of data sets opened at the same time, the amount of consumed memory can be roughly evaluated for the given known use-cases.

To learn more, see Estimating the memory footprint for a given data set.

Blusam migration

Once the proper storage scheme has been selected for a given data set, the blusam storage must be populated by migrating legacy data sets.

To achieve this, one has to use raw binary exports of the legacy data sets, without any charset conversion being used during the export process. When transferring data set exports from the legacy system, make sure not to corrupt the binary format. For example, enforce binary mode when using FTP.

The raw binary exports contain only the records. The import mechanism does not need the keys/indexes exports as all keys/indexes are being re-computed on the fly by the import mechanism.

Once a data set binary export is available, several options to migrate it to Blusam exist:

On AWS Mainframe Modernization managed environment:

or

or

On AWS Blu Age Runtime on Amazon EC2:

or

Import data sets using Groovy scripts

This section will help you writing groovy scripts to import legacy data sets into Blusam.

It starts with some mandatory imports:

import com.netfective.bluage.gapwalk.bluesam.BluesamManager
import com.netfective.bluage.gapwalk.bluesam.metadata.Key;
import com.netfective.bluage.gapwalk.rt.provider.ServiceRegistry
import java.util.ArrayList; //used for alternate keys if any

After that, for each data set to import, the code is built upon the given pattern:

There are 5 service implementations that can be retrieved from the service registry, using the following identifiers:

Whether to pick the regular vs large version of service for KSDS/ESDS depends on the size of the data sets and the storage strategy you want to apply for it. To learn how to pick the proper storage strategy, see Picking up the proper storage scheme.

To be able to successfully import the data set into Blusam, the proper properties must be provided to the loading service.

Common properties:

Data set kind dependant properties:

Key constructor calls:

The following Groovy samples illustrate various loading scenarios.

Loading a large KSDS, with two alternate keys:

import com.netfective.bluage.gapwalk.bluesam.BluesamManager
import com.netfective.bluage.gapwalk.bluesam.metadata.Key;
import com.netfective.bluage.gapwalk.rt.provider.ServiceRegistry
import java.util.ArrayList;

// Loading a large KSDS into Blusam
def map = [:]
map.put("bluesamManager", applicationContext.getBean(BluesamManager.class));
map.put("datasetName", "largeKsdsSample");
map.put("inFilePath", "/work/samples/largeKsdsSampleExport");
map.put("recordLength", 49);
map.put("primaryKey", new Key(0, 18));
ArrayList altKeys = [new Key(true, 10, 8), new Key(false, 0, 9)]
map.put("alternateKeys", altKeys);
map.put("indexingPageSizeInMb", 25);
def service = ServiceRegistry.getService("BluesamLargeKSDSFileLoader");
service.runService(map);

Loading a variable record length ESDS, with no alternate keys:

import com.netfective.bluage.gapwalk.bluesam.BluesamManager
import com.netfective.bluage.gapwalk.bluesam.metadata.Key;
import com.netfective.bluage.gapwalk.rt.provider.ServiceRegistry

// Loading an ESDS into Blusam
def map = [:]
map.put("bluesamManager", applicationContext.getBean(BluesamManager.class));
map.put("datasetName", "esdsSample");
map.put("inFilePath", "/work/samples/esdsSampleExport");
map.put("recordLength", 0);
def service = ServiceRegistry.getService("BluesamESDSFileLoader");
service.runService(map);

Variable record length data sets exports will contain the mandatory Record Decriptor Word (RDW) information to allow records splits at reading time.

Loading a fixed record length RRDS:

import com.netfective.bluage.gapwalk.bluesam.BluesamManager
import com.netfective.bluage.gapwalk.bluesam.metadata.Key;
import com.netfective.bluage.gapwalk.rt.provider.ServiceRegistry

// Loading a RRDS into Blusam
def map = [:]
map.put("bluesamManager", applicationContext.getBean(BluesamManager.class));
map.put("datasetName", "rrdsSample");
map.put("inFilePath", "/work/samples/rrdsSampleExport");
map.put("recordLength", 180);
def service = ServiceRegistry.getService("BluesamRRDSFileLoader");
service.runService(map);

Loading data sets in Multi-schema mode:

Multi-schema mode: In some legacy systems, VSAM files are organized into file sets, allowing programs to access, and modify data within specified partitions. Modern systems treat each file set as a schema, enabling similar data partitioning and access control.

To enable Multi-schema mode in the application-main.yml file refer to Blusam configuration. In this mode, data sets can be loaded into a specific schema using a Shared Context which is an in-memory registry for runtime information. To load a data set into a specific schema, prefix the data set name with the relevant schema name.

Loading a KSDS file into a specific schema for Multi-schema mode:

import com.netfective.bluage.gapwalk.bluesam.BluesamManager
import com.netfective.bluage.gapwalk.bluesam.metadata.Key;
import com.netfective.bluage.gapwalk.rt.provider.ServiceRegistry
import java.util.ArrayList;
import com.netfective.bluage.gapwalk.rt.shared.SharedContext;

// Loading a KSDS into Blusam
def map = [:]
String schema = "schema1";
String datasetName = schema+"|"+"ksdsSample";
SharedContext.get().setCurrentBlusamSchema(schema);
schema = SharedContext.get().getCurrentBlusamSchema();
map.put("bluesamManager", applicationContext.getBean(BluesamManager.class));
map.put("datasetName", datasetName);
map.put("inFilePath", "/work/samples/ksdsSampleExport");
map.put("recordLength", 49);
map.put("primaryKey", new Key(0, 18));
map.put("indexingPageSizeInMb", 25);
def service = ServiceRegistry.getService("BluesamKSDSFileLoader");
service.runService(map);

Loading a Large KSDS file into a specific schema for Multi-schema mode:

import com.netfective.bluage.gapwalk.bluesam.BluesamManager
import com.netfective.bluage.gapwalk.bluesam.metadata.Key;
import com.netfective.bluage.gapwalk.rt.provider.ServiceRegistry
import java.util.ArrayList;
import com.netfective.bluage.gapwalk.rt.shared.SharedContext;

// Loading a Large KSDS into Blusam
def map = [:]
String schema = "schema1";
String datasetName = schema+"|"+"largeKsdsSample";
SharedContext.get().setCurrentBlusamSchema(schema);
schema = SharedContext.get().getCurrentBlusamSchema();
map.put("bluesamManager", applicationContext.getBean(BluesamManager.class));
map.put("datasetName", datasetName);
map.put("inFilePath", "/work/samples/LargeKsdsSampleExport");
map.put("recordLength", 49);
map.put("primaryKey", new Key(0, 18));
map.put("indexingPageSizeInMb", 25);
def service = ServiceRegistry.getService("BluesamLargeKSDSFileLoader");
service.runService(map);

In addition, a configuration entry (to be set in the application-main.yml configuration file) can be used to fine tune the import process:

Blusam configuration

Configuring Blusam happens in the application-main.yml configuration file (or in the application-bac.yml configuration file for the stand-alone deployment of the Blusam Administration Console -- BAC -- application).

Blusam has to be configured on two aspects:

Blusam storage and caches access configuration

For information about how to configure access to Blusam storage and caches using either secrets managers or datasources, see Set up configuration for AWS Blu Age Runtime.

Blusam engine configuration

Disabling Blusam support

First, let's mention that it is possible do completely disable Blusam support, by setting the bluesam.disabled property to true. An information message will be displayed in the server logs at application startup to remind Blusam disabling:

BLUESAM is disabled. No operations allowed.

No further configuration about Blusam is required in that case and any attempt to use Blusam related features (either programmatically or through REST calls) will raise an UnsupportedOperationException in the Java code execution, with a relevant explanation message about Blusam being disabled.

Blusam engine properties

The Blusam engine configuration properties are regrouped under the bluesam key prefix:

Mandatory properties

Optional properties

Blusam Locks: the properties are prefixed with locks

Write-behind mechanism: the properties are prefixed with write-behind key:

Embedded EhCache fine-tuning: the properties are prefixed with ehcache key:

Multi-schema configuration properties

Sample configuration snippet:

dataSource:
  bluesamDs:
    driver-class-name: org.postgresql.Driver
    ...
    ...
bluesam:
  locks:
    lockTimeOut: 700
  cache: ehcache
  persistence: pgsql
  ehcache:
    resource-pool:
      size: 8GB
  write-behind:
    enabled: true
    threads: 8
    batchsize: 5000  
  pgsql:
    dataSource : bluesamDs 

Sample configuration snippet (with Multi-schema mode enabled for Blusam):

dataSource:
  bluesamDs:
    driver-class-name: org.postgresql.Driver
    ...
    ...
bluesam:
  locks:
    lockTimeOut: 700
  cache: ehcache
  persistence: pgsql
  ehcache:
    resource-pool:
      size: 8GB
  write-behind:
    enabled: true
    threads: 8
    batchsize: 5000 
  multiSchema: true 
  pgsql:
    dataSource : bluesamDs
    schemas: 
      - "schema1"
      - "schema2" 
      - "schema3"
    fallbackSchema: schema3

Blusam Administration Console

The Blusam Administration Console (BAC) is a web-application, used to administrate the Blusam storage. For information about the BAC, see AWS Blu Age Blusam Administration Console.

Appendix

General data set metadata attributes

General data set metadata serialization attributes list:

Estimating the memory footprint for a given data set

For small to medium sized data sets, the metadata (sizes and indexes for various keys) will be fully loaded into memory. Allocating proper resources for the machine hosting the server used to run modernized applications requires to figure out the memory consumption induced by the Blusam data sets, in particular regarding metadata. This section give practical answers to concerned operators.

The given formulas only apply to Blusam small to medium data sets, not using the "Large" storage strategy.

Blusam data set metadata

For a Blusam data set, metadata are split into two parts:

Calculating Internal Metadata footprint

Records sizes map

First, the internal metadata stores a map to hold the size of every record (as an integer) given its RBA (relative byte address — stored as a long number).

The memory footprint of that data structure is, in bytes: 80 * number of records

This applies to all data set kinds.

Indexes

Regarding the indexes for either the primary key of KSDS or alternate keys on both ESDS and KSDS, the calculation of the footprint depends on two factors:

The graphic below shows the size of the key index per record (y-axis) based on the size of the key (x-axis).

The corresponding formula for evaluating the footprint for a given key index of a data set is:

index footprint = number of records * ( 83 + 8 (key length / 8))

where ' / ' stands for the integer division.

Examples:

The total footprint for a given data set is the sum of all the footprints for all keys indexes and the footprint for the records sizes map.

For instance, taking the example data set 2, that has only a single key, the global footprint is: