How To Deploy Maintenance Template v1.

2 - QA & PRD

WORK INSTRUCTION

Doc Title: WEBMETHODS COMMAND CENTRAL - HOW TO DEPLOY MAINTENANCE TEMPLATE v1.2 - QA & PRD

Doc ID: N/A

Version: 1.0

Process Category: Work Instruction

OBJECTIVE

The purpose of this document is to describe steps required to deploy Command Central Maintenance Template v1.2 on an existing webMethods instance.

Scope

This procedure applies to the JJTS Global Operational Support Team that need to provide operational support and apply maintenance template on an
existing webMethods instance using Command Central.

Part of Maintenance Template v1.2 following capabilities will be used :

1) Set,Resume IS Maintenance mode (Maintenance Mode).

2) Introduction of latest Adapters Drivers.
3) Fix installation functionality, bring all products to latest fix level.
4) Custom packages installation (Framework packages to latest versions).
5) Force Package enabled/Disabled/Delete mode.
6) Database Configurator update (DB Schemas components update to latest component level).
7) External Script Call.
8) Migrate IS, MWS, CTP, SPM & TCA Oracle JVM to Azul Zulu JDK.
9) IS/MWS configuration file.

Preparation :

Below activities must be handled part of the template rollout :

1) Check/Update latest CCE operational NAS directories.

2) Execute automated backup script (MWS jetty.xml, IS Adapters drivers, /opr/webMethods Operational Scripts, Automated DB Schemas backup, File system
backup).
3) Create CCE Maintenance properties file.
4) Apply CCE Maintenance Template on Server 1
5) Apply CCE Maintenance Template on Server 2
6) Execute GWY server Java JDK to Azul JDK migration.
5) Run Post Functionalities Verification.

How To Deploy Maintenance Template v1.2 - DEV.

Step Tasks

Johnson & Johnson Information Technology - Confidential Page 1 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

1. Check/Update latest CCE operational NAS directories | This step is handled by L3 Team.

Make sure you using latest CCE operational NAS directories.

Connect to your regional CCE Linux Server host – see Confluence - CCE Architecture link for server details list
You must be logged as webm7 user to execute all commands.

/opt/webMethods/depot/CCE/v912install/*.* -- Make sure this top level NAS folder/sub-folders are in sync with EMEA Master server - itsbebel00455.jnj.com

Verify the package folders exist and are in sync ** compare with EMEA**
/opt/webMethods/depot/CCE/v912install/v9common/packages_egwy ** ri server**
/opt/webMethods/depot/CCE/v912install/v9common/packages_internal **internal server**
/opt/webMethods/depot/CCE/v912install/v9common/packages **internal server**
/opt/webMethods/depot/CCE/v912install/v9common/packages_zip **internal server**

Verify the supporting package folders are in sync ** compare with EMEA**
/opt/webMethods/depot/CCE/v912install/v9common/supportingPackage/CCE_MaintenenceUtility

Verify the backup script is in sync ** compare with EMEA**

/opt/webMethods/depot/CCE/v912install/backupwm.ksh

2. Update CCE Server custom_wrapper.conf disabling TCA & Broker server restart

Connect to to your respective CCE Linux Server(as webm7 user) - Use this link for CCE Regional Server Overview

Execute below commands :

# cd /opt/softwareag1/profiles/CCE/configuration
# vi custom_wrapper.conf

Make sure to add below lines if not yet found the bottom of the file

wrapper.java.additional.535=-Dcom.softwareag.platform.management.client.template.composite.skip.restart.for=TES-default,Broker-Server-6550,Brok

# wq!

Restart CCE Server so change becomes effective

# cd /opt/webMethods
# ./cce.sh stop default
# ./cce.sh start default

Johnson & Johnson Information Technology - Confidential Page 2 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

2. Execute backup script (File system) - Automated

Prior you continue with below step do make sure to disable all respective Inframon instance Alias (MWS/IS/GWY/Broker/WebRessource/etc...).
Since we updating DB components level to latest state part of this bundle both servers will need to be stopped while executing the maintenance.

Backup script will handle the following activities :

1) Backup of MWS jetty.xml, IS Adapters drivers, /opr/webMethods Operational Scripts, Upgrade of Oracle JDK to Azul JDK.
2) Stop all running components (SPM, IS, MWS, TCA, Broker, CTP).
3) Automated DB Schemas backup
4) Backup complete File system - /opt/softwareag1 and create a compressed tarball.
5) Restart all stopped components (SPM, IS, MWS, TCA, Broker, CTP).

Since this is a cluster environment type logon to your target Server 1 & 2 and launch Backup Script.

Start backup script on Server 1 - itsbebel00396.jnj.com server.

# script -a /opt/softwareag1/JnJ_Maintenance_9_12_v1_2-Evidence.txt
# cd /opt/webMethods/depot/CCE/v912install
# ./backupwm.ksh tar

Follow Q&A Screen Menu

SHUTDOWN TERRACOTTA y/[n]? n

BACKUP DB [y]/n? y
START TERRACOTTA [y]/n? n
START BROKER [y]/n? n
START GWY_BROKER [y]/n? n
START SPM [y]/n? y
START MWS [y]/n? y
START IS [y]/n? y
START CTP [y]/n? y

# exit

Start backup script on Server 2 - itsbebel00398.jnj.com server .

# script -a /opt/softwareag1/JnJ_Maintenance_9_12_v1_2-Evidence.txt
# cd /opt/webMethods/depot/CCE/v912install
# ./backupwm.ksh tar

Follow Q&A Screen Menu

SHUTDOWN TERRACOTTA y/[n]? n

BACKUP DB [y]/n? n
START TERRACOTTA [y]/n? n
START BROKER [y]/n? y
START GWY_BROKER [y]/n? y
START SPM [y]/n? y
START MWS [y]/n? n
START IS [y]/n? y
START CTP [y]/n? y

# exit

3. Create CCE Maintenance properties file

Create/Configure CCE Maintenance Properties file to reflect your environment config.

Connect to to your respective CCE Linux Server (as webm7 user) - Use this link for CCE Regional Server Overview

# cd /opt/webMethods/depot/CCE/Custom_Config_10.4

Update CCE_wm912_configs configuration with your environment specifics details

# vi CCE_wm912_configs
Go to Internal Server Maintenance - IS/MWS/BKR/SPM Section and update in accordance

Do update/check config file for both clustered servers configuration details – Make sure to respect proper format as it is case sensitive

# Syntax:
<SEQID> <TEMP_Name> <region> <env> <sector> <env_type(internal/egw/internal_egw)> <internal_spmport> <internal_host> <installdir> <bkr_syn
<Update_jetty.xml(true/false)> <serverlog_level(Off/Fatal/Error/Warn/Info/Debug/Trace)> <jdbc_migration(true/false)> <dbname> <dbpwd> <dbhost>
<dbtn> <dbarch> <dbproc> <dbaudit> <dbmws> <gwy_spmport> <gwy_host> <gwy_installdir> <gwy_quiesce_port> <mws_instance_name>

Johnson & Johnson Information Technology - Confidential Page 3 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

Server 1 :
CCE_MNTCNOBKR_BPM_ITSBEBEL00396 JnJ_Maintenance_9_12_v1_2 EMEA QA BPM internal 8504 itsbebel00396 /opt/softwareag1 false 55
itsbebelsp02784 1521 _DATA01 _INDX01 BPM_EU_QA_LOGS1 BPM_EU_QA_TN1 BPM_EU_QA_ARCH1 BPM_EU_QA_LOGS1 BPM_EU_QA_LO
itxxxxxx /opt/webMethods9/EMEA/BPM/softwareag1 3xx9 default

Server 2 :
CCE_MNTCNOMWS_BPM_ITSBEBEL00398 JnJ_Maintenance_9_12_v1_2 EMEA QA BPM internal 8504 itsbebel00398 /opt/softwareag1 false 5
itsbebelsp02784 1521 _DATA01 _INDX01 BPM_EU_QA_LOGS1 BPM_EU_QA_TN1 BPM_EU_QA_ARCH1 BPM_EU_QA_LOGS1 BPM_EU_QA_LO
itxxxxxx /opt/webMethods9/EMEA/BPM/softwareag1 3xx9 default

# wq! - Exit the script

Remark : when installing patch on Internal IS + GWY server -- > select internal_egw
when installing patch on Internal IS only -- > select internal
when installing patch on GWY server only -- > select egw
Make sure to update the MWS instance name if it is not “default”

Run the script against Server 1 - itsbebel00396.jnj.com

# ./create_propfile.ksh CCE_MNTCNOBKR_BPM_ITSBEBEL00396

Johnson & Johnson Information Technology - Confidential Page 4 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

This will create the following output :

Extracting values from Master Config file

-----------------------------------------
SEQUENCE ID : CCE_MNTCNOBKR_BPM_ITSBEBEL00396
Template Name : JnJ_Maintenance_9_12_v1_2
REGION : EMEA
ENVIRONMENT : QA
SECTOR : BPM
ENV TYPE : internal
INT Target Server: itsbebel00396
INT INSTALLDIR : /opt/softwareag1
INT SPM PORT : 8504
INT QUIESCE PORT : 5509
INT UPDATE JETTY : false
INT LOG LEVEL : Info
INT BKR SYNC : false
INT JDBC MIG : false
DB NAME : JBEQ0193
DB HOST : itsbebelsp02784
DB PORT : 1521
TableSpace Data : _DATA01
TableSpace Index : _INDX01
IS SCHEMA : BPM_EU_QA_LOGS1
TN SCHEMA : BPM_EU_QA_TN1
ARCH SCHEMA : BPM_EU_QA_ARCH1
PROC SCHEMA : BPM_EU_QA_LOGS1
AUDIT SCHEMA : BPM_EU_QA_LOGS1
MWS SCHEMA : BPM_EU_QA_MWS1
MWS Instance : default
-----------------------------------------

Created below Properties file under required CCE template path..

- /opt/softwareag1/profiles/CCE/data/templates/composite/JnJ_Maintenance_9_12_v1_2/ITSBEBEL00396_BPM_standalone_maintenance_nobkr.pro

Run the script against Server 2- itsbebel00398.jnj.com

# ./create_propfile.ksh CCE_MNTCNOMWS_BPM_ITSBEBEL00398

This will create the following output :

Extracting values from Master Config file

-----------------------------------------
SEQUENCE ID : CCE_MNTCNOMWS_BPM_ITSBEBEL00398
Template Name : JnJ_Maintenance_9_12_v1_2
REGION : EMEA
ENVIRONMENT : QA
SECTOR : BPM
ENV TYPE : internal
INT Target Server: itsbebel00398
INT INSTALLDIR : /opt/softwareag1
INT SPM PORT : 8504
INT QUIESCE PORT : 5509
INT UPDATE JETTY : false
INT LOG LEVEL : Info
INT BKR SYNC : false
INT JDBC MIG : false
DB NAME : JBEQ0193
DB HOST : itsbebelsp02784
DB PORT : 1521
TableSpace Data : _DATA01
TableSpace Index : _INDX01
IS SCHEMA : BPM_EU_QA_LOGS1
TN SCHEMA : BPM_EU_QA_TN1
ARCH SCHEMA : BPM_EU_QA_ARCH1
PROC SCHEMA : BPM_EU_QA_LOGS1
AUDIT SCHEMA : BPM_EU_QA_LOGS1
MWS SCHEMA : BPM_EU_QA_MWS1
MWS Instance : default
-----------------------------------------

Created below Properties file under required CCE template path..

- /opt/softwareag1/profiles/CCE/data/templates/composite/JnJ_Maintenance_9_12_v1_2/ITSBEBEL00398_BPM_standalone_maintenance_nomws.p

Johnson & Johnson Information Technology - Confidential Page 5 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

Validate both properties file content

# cd /opt/softwareag1/profiles/CCE/data/templates/composite/JnJ_Maintenance_9_12_v1_2
# vi ITSBEBEL00396_BPM_standalone_maintenance_nobkr.properties
# vi ITSBEBEL00398_BPM_standalone_maintenance_nomws.properties
# wq! - Exit the script

4. Apply CCE Maintenance Template on Server 1 - itsbebel00396.jnj.com

Prior continuing the maintenance template do make sure all components on itsbebel00396.jnj.com server are up & running.
This is applicable for both Internal and GWY server side.

# Logon to CCE Admin UI - https://itsbebelsp02332.jnj.com:8091/cce/web

# Click on Home > Installations Tab > Click on your respective instance > Instances > Start all components
In case SPM/CTP server are stopped (Red) do start them in accordance.

Remark :
In case you can’t connect to server through CCE do connect to itsbebel00396.jnj.com and check/restart related SPM server
# cd /opr/webMethods | ./spm.sh start default

Expected screen should be looking as follow where Status state are in green for each components :

Launch CCE Maintenance template provisioning on Server 1 - itsbebel00396.jnj.com

# Connect to your CCE Linux Server - <CCE_Server_Hostname> (as webm7 user)

# cd /opt/softwareag1/profiles/CCE/data/templates/composite/JnJ_Maintenance_9_12_v1_2
# sagcc exec templates composite apply JnJ_Maintenance_9_12_v1_2 -i ITSBEBEL00396_BPM_standalone_maintenance_nobkr.properties

Expected result should be as follow :

ID Duration Status Progress Description Progress Description Status Description Custom Status Custom Status Description
167 0 SCHEDULED 0 Applying template JnJ_Maintenance_9_12_v1_2 SCHEDULED

Logon to CCE UI https://<CCE_Server_Hostname>:8091/cce/web/?entry=stacks#jobs:

Johnson & Johnson Information Technology - Confidential Page 6 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

# Check the job started successfully:

# Click Applying template JnJ_Maintenance_v1_2 to view logs.

preMaintenance.sh, checkQuiescePort.sh and setMaintenanceMode.sh should be executed

# Following screen should be displayed upon successful run :

# Prior moving to next step do make sure all components have been restarted and are available :

# Logon to CCE Admin UI - https://<CCE_Server_Hostname>:8091/cce/web

# Click on Home > Installations Tab > Click on your respective instance > Instances > Expand tree for IS/MWS

5. Apply CCE Maintenance Template on Server 2 - itsbebel00398.jnj.com

Prior continuing the maintenance template do make sure all components on itsbebel00396.jnj.com server are up & running.
This is applicable for both Internal and GWY server side.

# Logon to CCE Admin UI - https://itsbebelsp02332.jnj.com:8091/cce/web

# Click on Home > Installations Tab > Click on your respective instance > Instances > Start all components
In case SPM/CTP server are stopped (Red) do start them in accordance.

Remark :
In case you can’t connect to server through CCE do connect to itsbebel00396.jnj.com and check/restart related SPM server
# cd /opr/webMethods | ./spm.sh start default

Expected screen should be looking as follow where Status state are in green for each components :

Johnson & Johnson Information Technology - Confidential Page 7 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

Launch CCE Maintenance template provisioning on Server 2 - itsbebel00398.jnj.com

# Connect to your CCE Linux Server - <CCE_Server_Hostname> (as webm7 user)

# cd /opt/softwareag1/profiles/CCE/data/templates/composite/JnJ_Maintenance_9_12_v1_2
# sagcc exec templates composite apply JnJ_Maintenance_9_12_v1_2 -i ITSBEBEL00398_BPM_standalone_maintenance_nomws.propertie

Expected result should be as follow :

ID Duration Status Progress Description Progress Description Status Description Custom Status Custom Status Description
170 0 SCHEDULED 0 Applying template JnJ_Maintenance_9_12_v1_2 SCHEDULED

Logon to CCE UI https://<CCE_Server_Hostname>:8091/cce/web/?entry=stacks#jobs:

# Check the job started successfully:

# Click Applying template JnJ_Maintenance_v1_2 to view logs.

preMaintenance.sh, checkQuiescePort.sh and setMaintenanceMode.sh should be executed

# Following screen should be displayed upon successful run :

Johnson & Johnson Information Technology - Confidential Page 8 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

# Prior moving to next step do make sure all components have been restarted and are available :

# Logon to CCE Admin UI - https://<CCE_Server_Hostname>:8091/cce/web

# Click on Home > Installations Tab > Click on your respective instance > Instances > Expand tree for IS/MWS

6. If your landscape do include GWY server do migrate GWY Server Java JDK to Azul JDK using below steps.

Logon to your target GWY server and stop both SPM/IS servers instances (in this case MDD)

# cd /opr/webMethods
# ./wM.sh stop MDD
# ./spm.sh stop MDD

Run below migration script

# cd /opt/webMethods/depot/CCE/v912install/v9common/Scripts

# ./CCE_JnJ_Java_Maintenance_GWY.ksh <IS_HOME_DIR>
# ./CCE_JnJ_Java_Maintenance_GWY.ksh /opt/webMethods9/EMEA/MDD/softwareag1

Once executed do verify corresponding log file for proper script execution.

# cd /opt/webMethods/depot/CCE/v912install/v9common/Scripts/logs
# cat CCE_JnJ_Java_Maintenance_GWY_190927_1828.log

Restart both SPM/IS servers instances (in this case MDD)

# cd /opr/webMethods
# ./wM.sh start MDD
# ./spm.sh start MDD

7. Prior you continue with below step :

Do make sure to re-enable Inframon instance Alias.

Do update webMethods v9.12 CCE Maintenance Template v1.2 Overview.xlsx sheet on Sharepoint as you move along and complete with your instance pa

Maintenance Template Functionalities Verification

Step Tasks

1. Check/Update latest changes being implemented

Internal IS Admin and GWY Server (if applicable) should have below added extended settings :

watt.server.rg.internalregistration.timeout=90 (Only Internal IS)

watt.art.connection.byPassConnValidation=true (Only Internal IS)

Johnson & Johnson Information Technology - Confidential Page 9 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

2. Check all products are running with /appjava/zulu_jdk8.0_212

Logon to both target server and check all products are running with zulu_jdk8.0_212.

# ps -fHu webm7 >> make sure all components are running with zulu_jdk8.0_212

3. Check webMethods Database component upgrade have ran successfully

Logon to one of your target server and navigate to "dbConfigurator.sh" script

# cd /opt/softwareag1/common/db/logs
# ls -ltr

View all last latest log file

# vi dcc_201909191514*

Check and confirm database components migration are successful and matching below DB components version :

Johnson & Johnson Information Technology - Confidential Page 10 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

3. Execute SG001_ITS_ServerQualification report to validate proper IS state on both servers.

Logon to your target IS server

Prior running the report do make sure to enable IS Logging > Security logger feature.
Navigate to Settings > Logging > Click on Security Logger if not enabled

Navigate to Packages -> Management

Select Home option for package "SG001_ITS_ServerQualification"
Verify all Validation Test Results for the instances are successful

Once report generated do revert back IS Logging > Security logger feature to disabled state,
Navigate to Settings > Logging > Click on Security Logger to disable the same.

4. Package versioning should comply to below table :

6. Re-enable Inframon disabled instance Alias.

Maintenance Troubleshooting Guide

Step Tasks

Step Tasks

Johnson & Johnson Information Technology - Confidential Page 11 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

1.
Issue Description :

Test 33 & 34 failing after running SG001_ITS_ServerQualification

Remediation Steps :
Logon to your target IS server
Navigate to Security > Port > Click on HTTPS Port - 5503
Click on Edit HTTPS Port Configuration
When prompted Would you like to disable the port?
Click on OK > Click on Enable > Click on Save Changes
Navigate to Settings > Logging > Click on Security Logger if not enabled
Do make sure HTTPS port is in enabled state

Prior running the report do make sure to enable IS Logging > Security logger feature.

Navigate to Packages -> Management

Select Home option for package "SG001_ITS_ServerQualification"
Verify all Validation Test Results for the instances are successful
Do make sure to disable IS Logging > Security logger feature (DEV-QA only)

2
Issue Description :

After executing the sagcc command in Step 4 in the above table, Job appears to be struck at IS quiesce mode switching and nothing
executed beyond even after half an hour.
Observation : SPM State update jobs for other servers were long running and appears to be in hung state.

Remediation Steps :

Restarted CCE and SPM one after the other using the below commands from putty.

/opt/softwareag1/profiles/CCE/bin/shutdown.sh
/opt/softwareag1/profiles/CCE/bin/startup.sh
/opt/softwareag1/profiles/SPM/bin/shutdown.sh
/opt/softwareag1/profiles/SPM/bin/startup.sh

After successful restart of CCE and SPM , rerun the template execution command mentioned in Step 4 in the above table.

3.
Issue Description :

Unable to establish connectivity to Gateway from Internal server. test it by invoking "SG001_JNJ_Admin.services.testing.loadBalancing:
wMISRunning" from outside JNJ network in the server where Maintenance patch was completed.
expected logs in RI and Internal server below

Johnson & Johnson Information Technology - Confidential Page 12 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

Remediation Steps :
go to Security > Ports in Internal server Disable all the Registration port (http and https), click on the disabled port and click ‘Edit HTTP Port
Configuration’.
Clear existing Administrator credentials in ‘Registration Credential’ and Save Changes. Once done update the credentials Again and Save
changes and Enable the registration port

Check again by invoking the service from outside JNJ and see if issue is resolved.

Johnson & Johnson Information Technology - Confidential Page 13 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

4 Issue Description :

TCA component is not started automatically after Template execution

Remediation Steps :

Login to the CCE , in Installations click of the server under maintenance and click instances, click on corresponding TC and start

Johnson & Johnson Information Technology - Confidential Page 14 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

5 Issue Description :

Below screen when running the SG001_ITS_ServerQualification report in Gateway server

Remediation Steps :

in Gateway server go to Security>ports and click on Gateway external port for HTTPS protocol , and click on 'Edit Enterprise Gateway Server
Configuration' Enable and save the
changes and execute the Qualification report again.

Training Requirements

Employees and contractors who will have the role of Integration services T1 Team and Integration services T2 Team needs to be trained on this Work Instruction.

Trainees are required to verify that they read and understood this Work Instruction and will use this process when performing the work.

Document History

Superseded Document(s)

Doc ID: – N/A Title: – N/A Effective Date: – N/A

Revision History

Version DD-MMM-YYYY Author Change Summary Major/MinorChange

1.0 10-OCT-2019 David Legrand Initial Initial

Johnson & Johnson Information Technology - Confidential Page 15 of 16

How To Deploy Maintenance Template v1.2 - QA & PRD

Johnson & Johnson Information Technology - Confidential Page 16 of 16