ROP Mainframe Data Services

ITON Approach and Capabilities for ROP Mainframe
Integration and Migration Services

A Technical Report

June 2011
08
Fall
ROP Mainframe Data Services

I T O N S o f t w a r e L L C . # 0 2 0 4 1 , K n o w l e d g e O a s i s M u s c a t , O m a n .

ITON Mainframe Data Services Confidential 2

Copyright ITON 2011 All Rights Reserved
ROP Mainframe Data Services

Ref No: ITON/WP-41/2011
No. of Pages: 30

Author:
ITON Software Group
Business Contact:
Siva Prasad
+968 99437934
[email protected]
ITON Corporation
PO Box 54667, Dubai
Dubai Airport Free Zone
United Arab Emirates
Phone (+971) 4 204 5518
Fax (+968) 4 204 5519
URL www.itoncorp.com
Version 1.0 (04 June 2011)
No part of this document may be reproduced in any form or by any means without
written permission of:
ITON Corporation

NOTICE:
This document contains proprietary details of ITON and ROP and is derived in
whole or in part from materials developed by ITON and ROP. The information in
this document is subject to change without notice. ITON assumes no
responsibilities for any errors that may appear in this document.
ITON and the ITON logo are trademarks or registered trademarks of ITON
Corporation
Other company, product and service names may be trademarks or service marks
of others.

Annotations
[ITON] ITON Corporation
[ROP] Royal Oman Police

Document History
Version 1.0:
Related Documents

ROP Mainframe Data Services 1.0

In the event of any ambiguity or conflict this response takes precedence over the
above documents and any other communication either written or verbal.

TABLE OF CONTENTS:

1.0 INTRODUCTION ....................................................................................................... 4
2.0 ITON EXPERIENCE .................................................................................................. 5
3.0 ISSUES IN MIGRATING NATURAL/ADABAS ......................................................... 9
4.0 ITON APPROACH & METHODOLOGY:................................................................. 11
TRELATIONAL AND DPS FUNCTIONALITY: ...................................................................... 16
TRELATIONAL DATA ANALYSIS AND MODELING .............................................................. 17
TRELATIONAL FILE IMPLEMENTATION: ........................................................................... 18
TRELATIONAL ADABAS FILE ANALYSIS ........................................................................ 19
RDBMS SCHEMA AUTO-GENERATION.......................................................................... 21
TRELATIONALPC AUTO-GENERATION ........................................................................... 21
DPS ARCHITECTURE ................................................................................................... 23
DPS MATERIALIZATION (ETL) ...................................................................................... 23
DPS MATERIALIZATION DATA ...................................................................................... 24
DPS PROPAGATION (CDC) ......................................................................................... 25
DPS PROPAGATION .................................................................................................... 26
5.0 APPENDIX .............................................................................................................. 29

1.0 INTRODUCTION

ROP Mainframe Environment:

ROP Business Applications are developed using Natural Development Tools. The
ADABAS database is the repository of defined pieces of data that make up records.
These records are stored using ADABAS management System. ADABAS is a DBMS
that uses inverted lists with relational capabilities developed by Software AG for
Mainframe. It comprises of ADABAS data files, which are collections of logically related
data, organized by data fields and logical records where Indexes are not embedded into
the Data.

ROP uses EntireX V7 as middleware for Application, Data Connectivity and Integration,
which ITON team has worked with ROP for various other Projects that are delivered with
in ROP like ROP Immigration & ROP Prison Management. EntireX V7 provides data and
application connectivity. The EntireX suite is comprised of an EntireX Kernel plus
EntireX Broker and Security Components that support RPC and Messaging
Communication among Client-Server applications. EntireX is a Message orinented
Middleware (MOM) that offers guarantee delivery of messages along with Push-pull
services.

Data Volume: 17 Million Travel Documents to be considered to be moved out of
Mainframe.

MOFA Lotus DOMINO Architecture:

Lotus Domino Application architecture at MOFA is designed by ITON to serve all MOFA
missions using Lotus Databases with Replication facilities running on Domino Clusters.
ITON has built and designed the Entire MOFA Domino application Infrastructure and is
well versed the architecture and data models that are applied with in the whole solution.
ITON has custom tools and processes ready for MOFA migration and will be mapped to
the new eVISA System and Data Architecture.

2.0 ITON Experience

ITON has been working with Royal Oman Police for over 10 years and has successfully
integrated with Mainframe for various ROP Business Applications. The key challenge is
always integration with ROP Mainframe for various business applications like ROP
Prison System, ROP Immigration DMS and ROP FIU. ITON has successfully handled
the Integration and Migration Services with ROP using EntireX as a middleware platform
provided by ROP. ITON can utilize Software AG tools like Natural Engineer and SQL
Gateway, if available with ROP which is not mandatory.

ITON has partnered with various international experts from organizations like TreeHouse
Inc, USA who are specialized in dealing with Mainframe Migrations especially ADABAS
to handle ROP Mainframe Migration and Integration Requirements. We at ITON
understand how Important and Complicated is the Mainframe Data Migration for eVisa
Project. ITON will be putting in all the best resources for this entire activitiy. As part of
this Program, ITON will have Mr. Moosa Al Lawati, who is ex-ROP and has worked in
DGIT for over 27 years in various projects, planning, development and managing
Mainframe based Applications. Mr. Moosa, having years of hands on Expereince, he
clearly understands the core mainframe design and the migration requirements. Many
Expert teams and Consultants are made available for this Data Migration Project.

ITON has worked extensively with EntireX .Net for integration services. ITON has
experience working with .Net Wrapper, .Net Client or .Net Server DLL. The EntireX .Net
wrapper provides acces to the EntireX RPC servers for .Net client application and .Net
servers for any EntireX client. The .Net Wrapper takes as input an EntireX IDL file, that
describes the interface for the RPC, and generates C# classes that implement the
methods and data types of the interface.


The EntireX .NET Wrapper works as follows:
C# code is generated from the EntireX IDL file. Using C# is a natural
choice when full fledged .NET programming is required, since C# was
designed for the .NET platform.
The .NET Wrapper runtime implements functionality that is not specific to
a given IDL file (e.g., marshalling and unmarshalling of data). The C#
code makes use of the .NET wrapper runtime functionality. The customer
interface and the .NET Wrapper runtime is "managed" .NET code (C#)
and makes use of advanced .NET features such as Attributes,
VersionInfo, etc.
The .NET Wrapper runtime makes use of the functionality of the
"unmanaged" EntireX runtime ("dllimport" in C#). "Managed" .NET code
and "unmanaged" DLL code can be combined safely.
The EntireX IDL compiler and an appropriate template are used for the C# code
generation.

The .Net Client Application for a given IL file, the For a given IDL file, the EntireX IDL
compiler (erxidl.exe) and a C# code generation template csharp_client.tpl are used to
generate a client stub. The source code generated by the .NET Wrapper can be
compiled into a .NET assembly with the Visual Studio .NET C# compiler. Application
developers can use the generated client stub assembly to write .NET applications that
access EntireX RPC servers. They are not limited to C# as programming language. Any
.NET programming language based on the Common Language Runtime (CLR) can
make use of the client stub assembly. Choices are C#, VisualBasic.NET or managed
C++.


The .Net Server DLL model

The EntireX IDL compiler and a C# code generation template csharp-server.tpl are used
to generate a C# code frame for a specific IDL. Application developers can use the
generated frame to write their own server code for each program in the IDL. The source
code can be compiled into a .NET assembly (DLL) with the Visual Studio.NET C#
compiler.

Following are some of ITON Implementations implemented using SoftwareAG EntireX
Components and Services:

ROP ePrison System:

ITON has worked closely with ROP team to build EntireX components to provide
mainframe data services for all Prisoner related information which was used to connect
from Lotus Domino to extract all Prisoner details from various files in mainframe and
store in the ePrison Domino system. The data would be updated on a periodic basis into
Domino System by communicating with EntireX Gateway.

ROP Immigration EDMS:

ITON again has worked closely in integrating Passport EDMS system ROP Mainframe
again using EntireX components for searching Omani Citizen information for indexing
purposes at the Passport Imaging and Indexing facility in ROP and store into RDBMS
System.

ROP FIU System:

ITON has successfully worked with ROP DGIT and ROP DGCS to connect to NRS Web
Services to extract various personal data from the system and save into the Local
RDBMS System and maintain the updated information.

3.0 Issues in Migrating Natural/ADABAS

There are a variety of problems that must be handled in migrating Natural and
ADABAS-based COBOL code:
Database Conversion
o Conversion of the ADABAS data model to a Relational model. The
ADABAS data model is not relational and definitely not normalized.
The data model must be converted into an efficient, maintainable
Relational model, using data types supported by the Relational
model as opposed to the legacy ADABAS data types. A key
consequence is that the placement and representation of the data in
the database will not be the same and therefore data accesses and
updates must change.
o Conversion of ADABAS data base content into Relational database
content. It is called "data processing" because ultimately these
applications are about the data! The existing ADABAS database
content must be extracted and moved into the new Relational form
without losing any information. If the new target platform is not the
mainframe, the data must moved off the mainframe.
COBOL code modification to change ADABAS calls to relational calls. If
COBOL programs will be retained on the mainframe, then ADABAS calls in the
COBOL programs must be modified to access the data from the new
Relational model.
Code translation of Natural or COBOL code with ADABAS calls to modern OO
languages. If the goal is to move from mainframe then
o The source code of the application must be converted to Java or C#
o Explicit ADABAS calls in COBOL code and implicit calls in Natural
must be replaced by database calls that understand the relational
model and the representational changes. Reads and writes of flat
files must be replaced by modern file reads and writes.
o External interfaces to other systems must be replaced by some
equivalent

o EntireX interfaces must be converted in a way that enables short-
term continued access through EntireX with eventual ability to
replace by equivalent remote access facilities such as Web services.
o Job Control and Scheduling must be replaced
o Security must revised to support the broader accessibility of the
application
Application Testing. This provides assurance to all parties that the result
preserves the business logic and functionality.
Maintenance change Integration.
Migrations take time.
o The legacy application will continue to serve the organization during
the migration, and business needs will force changes to the legacy
application to even as the migration occurs. These changes must be
moved into the migrated application.

4.0 ITON Approach & Methodology:

ITON works with various Adabas migration tools and services from Tree House Inc,
USA, which is a very specialized, company dealing primarily with Adabas Migration
Services. Tree House, primarily looks in to the 10,000 IBM mainframe sites worldwide,
with particular focus on the 2,500 organizations using ADABAS and NATURAL, 500 of
these being in North America.

Most mainframe users face critical data management challenges due to the complexity
and proprietary nature of ADABAS and other databases. TSI meets the needs of these
customers through a comprehensive and robust set of productswith supporting
servicesto enable standards-based access to, and replication and integration of,
mainframe data sources throughout the technology architecture. The companys
visionary leadership in this market has included pioneering ADABAS-to-RDBMS ETL
(extract-transform-load) and CDC (change data capture) with tRelational/DPS in the mid-
1990s, and launching of the era of real-time ADABAS-to-RDBMS replication with
DPSync in 2004. Today, TSI stands alone in its product maturity, capability and breadth
of product portfolio. Despite the rapid pace of change in the IT landscape, TSIs
customer base can be assured that there remains a strong commitment to providing
continued support and upgrades for the product suite.

The increasing demand for mainframe application modernization and migration
particularly in the Software AG customer arenahas enabled TSI to position itself as an
important contributor to such initiatives. TSIs long experience, deep knowledge and
wide-ranging capabilities in ADABAS/NATURAL and other mainframe technologies
make the company a valued partner for third-party solution providers and a trusted
advisor to customers.


Finally, for customers and service providers struggling with a shortage of mainframe-
skilled resources and rapidly changing IT environments, TSI offers staff augmentation
and outsourcing services for technical roles. TSI works with over 700 large Corporations
in the world. Some of the references:
Banking & Finance Asian Development Bank, Philippines
B-Source SA, Switzerland
Banco Real, Brazil
Bank Vontobel AG, Switzerland
Dexia Banque Internationale Luxembourg, Luxembourg
KAS BANK N.V., The Netherlands
Volkswagen Bank, Germany
Communication Verizon, Colorado
Telemar, Brazil
U.S. Sprint, California, Kansas, Texas
Education Penn State University, Pennsylvania
University of Texas, Austin (several campuses)
University of Toronto, Canada
Entertainment Club Med, France
Universal Music, International
Food Dansk Supermarked A/S, Denmark
Dole Fresh Vegetables, California
EDEKA ZENTRALE AG & Co. KG, Germany


Food Lion, North Carolina
Government (Federal) NASA, U.S.
Royal Oman Police, Oman
Department of Labor, United States
Government (Local) County of Sacramento, California
County of Los Angeles, California
Government (State) Informatikzentrum Landesverwaltung Baden-Wuerttemberg, Germany
State of California (Several Sites)
State of Minnesota
State of Utah
Services First American Real Estate, California, Florida, Texas
Insurance GMAC Insurance, Missouri
Berufsgenossenschaft Druck und Papierverarbeitung, Germany
Berufsgenossenschaft Handel und Warenwirtschaft, Germany
Berufsgenossenschaft Metall Nord Sd, Germany
Regence Blue Cross & Blue Shield of Oregon
SIGNAL IDUNA, Germany
Manufacturing Bradford Exchange, Illinois
EATON Corporation, Pennsylvania
Dow Chemical Company, Michigan
Newell Companies, Illinois
Nissan Motor Manufacturing Ltd., United Kingdom, United States
REHAU AG + Co, Germany
Yves Rocher, Germany
Outsourcing CSC, United States
EDS, United Kingdom, Canada
IBM Global Services, United States, Canada
Publishing Hachette Book Group, Massachusetts
Curtis Circulation Co., New Jersey
Transportation Federal Express, Tennessee
NMBS Holding, Belgium
Southwest Airlines, Texas
Utilities American Electric Power Service, Ohio
Tampa Electric Company (TECO), Florida

High Level Migration Process for Open Technical Platform:


ITON uses various TSI Products and Tools to perform Adabas Data Migration in
cooperation with TSI. Various product components that are used for ROP eVisa data
migration services include:

tRelational, An ADABAS-to-RDBMS modeling, mapping and data analysis tool
Data Propagation System (DPS), an ADABAS-to-RDBMS data migration and
propagation system for data distribution and warehousing
tRelationalPC, a windows-based client/server GUI data modeling and mapping
environment (included with tRelational)
Treehouse Remote Access (TRA), middleware that allows tRelationalPC to
communicate with tRelational on the Mainframe (included with tRelational)
tRelational Features:

Modeling and Mapping
- Native ADABAS/NATURAL application
- Predict metadata discovery and ADABAS file analysis
- Automated generation of normalized RDBMS schemata with explicit ADABAS
field to RDBMS column mapping
- Robust modeling and mapping normalize, denormalize, substring,
concatenate
- Single rule base and metadata repository
Code Generation
- RDBMS Data Definition Language (DDL) Create tables, columns and
contraints
- DPS Parameters Extract and Transformation parameters

tRelational and DPS Functionality:

- Captures Logical (PREDICT) and Physical (ADABAS FDT) file definitions and
resolves any discrepancies.
- The implemented file provides the basis for modeling and mapping to the
RDBMS table(s).
- Captures statistical analysis to provide or confirm the understanding of the
source data. The analysis provides for improved modeling and early
identification of problem data.
- Provides physical modeling and explicit ADABAS to RDBMS mapping. Auto
Generation provides intelligent and automatic modeling and mapping from
an Implemented File.
- tRelational generates all input parameters needed to begin Materialization
and Propagation.

tRelational Data Analysis and Modeling

- tRelational generates output for the creation of tables, columns, and
constraints for your target RDBMS.
- The Materialization process requires No Direct ADABAS Access
- Extracts from an ADABAS utility backup
- Extracted data is transformed into a relational form
- RDBMS tables are then populated by the native RDBMS loader utility
- The Propagation process requires No Direct ADABAS Access
- ADABAS transaction data is extracted from the ADABAS Protection Log Files
- The extracted data is transformed into SQL UPDATE, INSERT, and
DELETE statements

tRelational File Implementation:

Capture Logical (Predict) file, Userviews, and Physical (FDT) definitions.

Fields that are defined logically and physically are highlighted.

tRelational ADABAS File Analysis

One time capture of statistical analysis of repeating data (MUs and PEs),
Candidate variable text data, and descriptors for improved modeling.

Statistics of MUs and PEs for sizing of child tables and potential de-normalization
of tables to individual columns.


Statistics of alphanumeric fields for candidate variable character text columns

This Screen shows descriptor/superdescriptor usage statistics to determine
candidate Primary Keys and Indexed Columns.

RDBMS Schema Auto-Generation

Generates table(s), Columns, constraints, and mappings for a selected
implemented file.
tRelationalPC Auto-Generation

Auto Generation Example: 4 tables with Primary Key and Foreign Key
Constrains, and the added DPS PE Sequencer (PE Occurrence).

Output Generated from Metadata

- RDBMS Data Definition Language (DDL)
- DPS specifications (parameters) for ETL and CDC Processing
- Metadata Reports (Summary and Detail)
- tRelational API enables Metadata export to other tools and repositories
Generated RDBMS DDL from Sample:

DPS Architecture

- Written in Assembler for efficiency
- Executed as batch job
- No Calls to active ADBAS required
- No Impact on Production Environment
- External Transformation Routines (ETRs)
- A Call to a Linked Object
- Dozens of Built-ins
- Custom Transformation and Data Cleansing

DPS Materialization (ETL)

- Provides initial load of the RDBMS
- Extracts from ADASAV
- Intelligent Transformation based on Model/Mappings
- Generates rows for target tables and SQL Utility Load Control Statements
- Provide refresh of the RDBMS when required or desired


DPS Materialization Data

Contains all row images to be loaded into the RDBMS REPOSITORY. Each row
is prefixed with a Table ID and is formatted and delimited natively for the RDBMS
loader.

Native loader control statements are automatically generated with each DPS
Materialization run.

DPS Propagation (CDC)

DPS Propagation

- provides periodic synchronization of the RDBMS target with the source
ADABAS database
- Extracts from PLOG archives
- Intelligent transformation based on Update and model/mappings
- Generates SQL for Inserts, Updates, Deletes and Commits

DPS Propagation

Sample SQL resulting from an update to Personnel ID, mapped to a Primary Key
showing the Deletes and Inserts generated to maintain referential integrity.


Overall Data Migration and Integration Challenges:

DATA
MIGRATION
Challenges
Transform Non-
Relational data
Structures
Minimize
Performance
IMPACT to
production
systems
Minimize
System
Downtimes
Reduce
Learning Curve
Reduce Data
Discovery Time
Quality Control


Solution Architecture Data Migration:

ITON will be making expert Adbas/Natural Programmers to run the entire
process under the Guidance of Mr. Moosa Al Lawati. These expert Programmers
will be analysing the data models and use the right tools to deliver the data out of
the mainframe. In required the Programming team will be generating custom
scripts in Mainframe to handle the complex data migration scenarios.

ITON approach is mainly utilizing powerful TSI tools for ADABAS Migration and
Software AG EntireX, SOA Gateway for Mainframe Integration Services.

Availability of Specialized Adabas / Natural Developers in the team will give more
flexibility to generate required scripts for smooth Migration of Data.


5.0 APPENDIX

(Additional reference material and profiles)

ROP Mainframe Data Services

Uploaded by

Copyright:

Available Formats

ROP Mainframe Data Services

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ROP Mainframe Data Services

Uploaded by

Copyright:

Available Formats

What challenges are involved in migrating data from a non-relational database?

What is ITON's approach and methodology for migrating data from a non-relational database?

ITON Approach and Capabilities for ROP Mainframe

Integration and Migration Services

You might also like