AWS Transform for mainframe refactor

Setup the AWS DevOps Pipeline requirements

Setup the AWS Blu Age DevOps Pipelines requirements

Download the AWS Transform for mainframe Runtime

  1. Download the runtime
    The AWS Transform for mainframe Runtime is accessible from AWS Transform for mainframe refactor Toolbox.
    You can download the desired version.

  2. In Linux environment use “unzip” command to decompress the AWS Transform for mainframe Runtime archive. Under Windows, use 7-zip to unzip the runtime compressed archive for example.

    unzip gapwalk-<GAPWALK_VERSION>.zip

Setup your own AWS Transform for mainframe Runtime S3 bucket

Every client account has its own S3 containing the binary files named s3-<AWS_REGION>-<PROJECT_NAME>-velocity-<AWS_ACCOUNT_ID>. This is the one used for the pipeline for creating the Tomcat image and as a developer you will have to maintain it.

  1. For all the commands below make sure you are in the velocity folder (from the runtime decompressed archive) and make sure S3 bucket is accessible from your environment.
  2. Push the shared folder in the runtime folder of your S3 bucket
aws s3 sync shared s3://s3-<AWS_REGION>-<PROJECT_NAME>-velocity-<AWS_ACCOUNT_ID>/<BRANCH_NAME>/velocity/runtime/

Note:
Your default existing branch name is master
.pom and .parent files must be pushed as well. Otherwise, the pipeline won't work.

WARNING:
The sync command only adds new files but does not delete them if they don’t exist anymore. If the runtime folder is not empty, I recommend deleting and using the command sync again on the s3 folder. You need to be sure you are not mixing up 2 different gapwalk versions.

  • Push the webapps folder in the webapps folder of your S3 bucket
aws s3 sync webapps s3://s3-<AWS_REGION>-<PROJECT_NAME>-velocity-<AWS_ACCOUNT_ID>/<BRANCH_NAME>/velocity/webapps/

Note:
*web.war and *service.war must not appear in the velocity/webapps directory because the building pipeline builds them

  • If you need webapps-extra, push the ones you want in velocity/webapps (e.g cl-command, gs21 ...):
aws s3 cp webapps-extra/gapwalk-gs21-<version>.war s3://s3-<AWS_REGION>-<PROJECT_NAME>-velocity-<AWS_ACCOUNT_ID>/<BRANCH_NAME>/velocity/webapps/
  • For Japan: IBM930, you will need to add some .jar files in the classpath (e.g icu4j.jar):
aws s3 cp icu4j.jar s3://s3-<AWS_REGION>-<PROJECT_NAME>-velocity-<AWS_ACCOUNT_ID>/<BRANCH_NAME>/velocity/classpath/

Note:
All .jar files added in velocity/classpath will be added automatically in the CLASSPATH variable in setenv.sh

How to configure the application?

Web pom.xml file

The latest versions of the CI do not have access to internet. You will have to update the section build of your web pom.xml with the following:

<build>  
<plugins>  
<plugin>  
<groupId>org.codehaus.mojo</groupId>  
<artifactId>exec-maven-plugin</artifactId>  
<version>3.0.0</version>  

<executions>  
<!-- Execution for npm install -->  
<execution>  
<id>npm-install</id>  
<phase>initialize</phase>  
<goals>  
<goal>exec</goal>  
</goals>  
<configuration>  
<executable>npm</executable>  
<workingDirectory>${project.basedir}/angular</workingDirectory>  
<arguments>  
<argument>install</argument>  
</arguments>  
</configuration>  
</execution>  

<!-- Execution for npm run build -->  
<execution>  
<id>npm-run-build</id>  
<phase>compile</phase>  
<goals>  
<goal>exec</goal>  
</goals>  
<configuration>  
<executable>npm</executable>  
<workingDirectory>${project.basedir}/angular</workingDirectory>  
<arguments>  
<argument>run</argument>  
<argument>buildEcs</argument>  
</arguments>  
</configuration>  
</execution>  
</executions>  
</plugin>  

<!-- Package into war -->  
<plugin>  
<groupId>org.apache.maven.plugins</groupId>  
<artifactId>maven-war-plugin</artifactId>  
<version>3.2.0</version>  
<configuration>  
<webResources>  
<resource>  
<directory>angular/dist</directory>  
</resource>  
</webResources>  
</configuration>  
</plugin>  
</plugins>  
</build>

Note: The com.github.eirslett groupId should not appear anymore

Note2: To use org.codehaus.mojo, nodejs and npm should already be installed on your EC2, which is the case by default if you used the windows template to create your EC2

Tomcat configuration

The config files used by AWS Blu Age DevOps Pipeline for Tomcat will be the ones in gitlab-<PROJECT_NAME>-server, that is, the ones used by developers. However, a couple of modifications are needed to be able to have the same config files between Developers and DevOps:

  • In conf/server.xml, the variable related to the DB DNS will be APP_DB_DNS. See the example below:
<Resource auth="Container" driverClassName="org.postgresql.Driver" initialSize="0" maxIdle="10" maxOpenPreparedStatements="-1" maxTotal="20" maxWaitMillis="-1" name="jdbc/primary" password="<HARDCODED_PWD>" poolPreparedStatements="true" testOnBorrow="true" type="javax.sql.DataSource" url="jdbc:postgresql://${APP_DB_DNS}:5432/<HARDCODED_NAME>" username="<HARDCODED_USERNAME>"/>

Note: HARDCODED variables can stay hardcoded in the server.xml file Note2: Modify the Jics and Bluesam URLs with APP_DB_DNS as well

  • In conf/server.xml, run the application on the port 8080. See example below.
<Connector connectionTimeout="20000" port="8080" protocol="HTTP/1.1" redirectPort="8443" />
  • In conf/server.xml, make sure </Host> is on a separate line than the lines starting with the <Context> tag . Here is an example of the Host section:
<Host appBase="webapps" autoDeploy="true" name="localhost" unpackWARs="true">  
<Valve [...] />  
<Context [...] />  
</Host>

Note: <Context> lines will be deleted by the CI so if </Host> is at the end of a <Context> line, it will be removed and will cause issues

  • In config/application-main.yml, you will have to add the following line:
filesDirectory: ${FILES_DIR}

Of course, you will have to modify your Eclipse configuration by adding these 2 variables, that is to say, ${APP_DB_DNS} and ${FILES_DIR}

IMPORTANT: On the DevOps side, the conf/server.xml file is modified by deleting the path to the .war webapps because we don’t need to specify the path on Linux and Devs use Windows paths. So <Context> lines are taken out.

IMPORTANT2: The common.loader line in the conf/catalina.properties file is also modified. This line is modified in the modify_app_config.sh script located in the gitlab-devops/codebuild/codebuild-app/scripts

Modify the application web

Initially, the web part can only run on a specific IP (e.g localhost). This is an issue because for testing we deploy the application multiple times on different IPs. So we need the application to run on any IP.

To run the application on any IP:

  • Position yourself at the root of your web project
  • Add a “BuildEcs” script to the package.json file (the building pipeline runs that script)
"buildEcs": "ng build --configuration ecs"
  • Add & modify the following environments file:
  • Create /angular/src/environments/environment.ecs.ts
export const environment = {  
production: false,  
ecs: true  
};
  1. Modify /angular/src/environments/environment.prod.ts
export const environment = {  
production: true,  
ecs: false  
};
  1. Modify /angular/src/environments/environment.ts
export const environment = {  
production: false,  
ecs: false  
};

  • Add a configuration for the created ecs environment file in the angular.json file, within the “architect” → “build” section:

    "configurations": {  
"ecs": {  
"optimization": true,  
"outputHashing": "all",  
"sourceMap": false,  
"namedChunks": false,  
"aot": true,  
"extractLicenses": true,  
"vendorChunk": false,  
"buildOptimizer": true,  
"fileReplacements": [  
{  
"replace": "src/environments/environment.ts",  
"with": "src/environments/environment.ecs.ts"  
}  
]  
}
  • Finally, replace the following condition block at line 69 in the src/app/config-service.ts file’s configuration() method, to provide the backendUrl:
if (!environment.production && !environment.ecs) {  
// Port 4200 is used by the local development angular server  
// configuration.useModernLegacyStyle = true;  
configuration.backendURL = 'http://localhost:4200/gapwalk-application';  
configuration.verbose = true;  
}  
if (environment.ecs) {  
const firstPathSegment = window.location.pathname.split('/')[1]  
const contextPath = firstPathSegment ? `/${firstPathSegment}` : '';  
configuration.backendURL = window.location.origin + contextPath + '/gapwalk-application';  
}

If there is a compilation issue on config-service.ts you can remove those unused imports lines:

import { Observable, of } from 'rxjs';  
import { env } from 'process';

Create the logback for the application

Below is a simple configuration for logging. Sifter is active this will create a log file per job/batch execution.

logback.xml (workingDir/conf/logback/)

<?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE XML>  
<configuration scan="true">  

<include file="./config/logback/logback-definitions.xml" />  

<logger name="com.netfective.bluage.gapwalk.runtime.statements.DynamicFileBuilder" level="info"/>  
<logger name="com.netfective.bluage.gapwalk.io.infrastructure.basic.SystemOutFile" level="info"/>  

<root level="info">  
<appender-ref ref="CONSOLE" />  
<appender-ref ref="FILE" />  
<appender-ref ref="SIFTER" />  
</root>  

</configuration>

logback-definitions.xml (workingDir/conf/logback/)

<?xml version="1.0" encoding="UTF-8"?>  
<included>  

<!-- Hide logback configuration logging. -->  
<statusListener class="ch.qos.logback.core.status.NopStatusListener" />  

<property name="DEFAULT_LOG_PATTERN" value="%date{yyyy-MM-dd HH:mm:ss.SSS,Asia/Tokyo} %-15.15contextName %-5.5level ${PID:- } --- [%thread] %-40.40logger{15} : %msg%n" />  
<property name="DEFAULT_CHARSET" value="utf8" />  
<property name="LOG_OUTPUT_PATH" value="./logs" />  

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">  
<encoder name="defaultEncoder">  
<pattern>${DEFAULT_LOG_PATTERN}</pattern>  
<charset>${DEFAULT_CHARSET}</charset>  
</encoder>  
</appender>  

<appender name="SIFTER"  
class="ch.qos.logback.classic.sift.SiftingAppender">  
<discriminator>  
<key>jobDiscriminator</key>  
<defaultValue>default_main</defaultValue>  
</discriminator>  
<sift>  
<appender name="JOB${jobDiscriminator}"  
class="ch.qos.logback.core.FileAppender">  
<file>${LOG_OUTPUT_PATH}/${jobDiscriminator}.log</file>  
<layout class="ch.qos.logback.classic.PatternLayout">  
<pattern>${DEFAULT_LOG_PATTERN}</pattern>  
<charset>${DEFAULT_CHARSET}</charset>  
</layout>  
</appender>  
</sift>  
</appender>  

<appender name="SIFTER2"  
class="ch.qos.logback.classic.sift.SiftingAppender">  
<discriminator>  
<key>progDiscriminator</key>  
<defaultValue>default_prog</defaultValue>  
</discriminator>  
<sift>  
<appender name="JOB${progDiscriminator}"  
class="ch.qos.logback.core.FileAppender">  
<file>${LOG_OUTPUT_PATH}/${progDiscriminator}.log</file>  
<layout class="ch.qos.logback.classic.PatternLayout">  
<pattern>%msg%n</pattern>  
<charset>${DEFAULT_CHARSET}</charset>  
</layout>  
</appender>  
</sift>  
</appender>  

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">  
<prudent>true</prudent>  
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">  
<fileNamePattern>  
${LOG_OUTPUT_PATH}\log_%d{yyyy-MM-dd}_%i.log  
</fileNamePattern>  
<maxFileSize>100MB</maxFileSize>  
<maxHistory>100</maxHistory>  
<totalSizeCap>10GB</totalSizeCap>  
</rollingPolicy>  
<encoder>  
<pattern>${DEFAULT_LOG_PATTERN}</pattern>  
<charset>${DEFAULT_CHARSET}</charset>  
</encoder>  
</appender>  

<logger name="com.uwv.functions" additivity="false" level="INFO">  
<appender-ref ref="CONSOLE" />  
<appender-ref ref="FILE" />  
<appender-ref ref="SIFTER2" />  
</logger>  

</included>