Reference & Master Data

SLIDES IS ADAPTED FROM :

HOW TO IDENTIFY THE CORRECT MASTER DATA SUBJECT AREAS & TOOLING FOR YOUR
MDM INITIATIVE
CHRISTOPHER BRADLEY
Reference &
Master Data
Management:
Context (DMBoK)
Master Data Management, WHY?
› No organisation has just one system (unless the are tiny)
› Details about the same noun are found in multiple systems,
eg. Customer
The same customers may be defined
in:
› Problems • Finance systems
» Data may need to be rekeyed in each system • Marketing systems
• Line of business systems
» Systems may not be in synch (new records, updated records)
» Duplicate data: are “ABC Ltd” and “ABC Limited” the same thing?
» No single version of the truth
» Reporting / Analysis: difficult to combine data from multiple
systems
What is Event / Transaction Data?

Event data example:

› “Bob bought a Mars bar from Morrisons on Monday 3rd Jan at 4pm and paid using
cash.”

WHO WHAT WHERE WHEN HOW QUANTITY AMOUNT

Bob Bobson Mars bar Morrisons, Bath 16:00 Monday 3rd January 2011 Cash 1 £0.60

CUSTOMER PRODUCT VENDOR DATE PAYMENT QUANTITY AMOUNT

CODE CODE CODE METHOD
BB005 CONF101 WMBATH 2011-01-03 16:00 CASH 1 £0.60

Terminology
Field (or attribute) = column in a database table
Record = row in a database table
About Event Data

› AKA Transaction data Includes information identifying Does not include information
› Describes an action (a verb): the nouns that were involved in describing the nouns:
the event (the Who / What / » Bob Bobson is male, aged
» E.g. “buy” Where / When / How and 25 and works for British
› May include measurements maybe even the Why): Airways
about the action: » Bob Bobson » Monday 3rd Jan 2011 is a
» Quantity bought » Mars bar bank holiday
» Amount paid » Morrisons, Bath » The address of Morrisons
» 16:00 Monday 3rd Jan 2011 Bath is: York Place, London
Road, Bath, BA1 6AE.
» Cash
What is Reference Data…
REFERENCE DATA
REFERENCE DATA? MANAGEMENT ?
» Data that defines the set of permissible control over defined domain values
values to be used by other data fields.
(also known as vocabularies),
» Reference data often is defined including control over standardized
by standards organizations
(such as country codes as
terms, code values and other unique
defined in ISO 3166-1). identifiers, business definitions
for each value, business
» Example: country code,
province code, etc. relationships within and across
domain value lists, and the
consistent, shared use of accurate,
timely and relevant reference data
values to classify and categorize data..
[DAMA, the Data Management
Association]
What is Reference Data…
» Reference data is data used to classify or categorize other data
» Reference data values should conform to a set of allowable data values called
value domain
» Value domain:
 Internal standard:
 Order Status: New, In Progress, Closed, Cancelled, and so on
 External standard (government or industry standard)
 Two-letter United States Postal Service standard postal code abbreviations for U.S. states,
such as CA for California
» More than one set of reference data value domains may refer to the same
conceptual domain.
 An official name (―California‖).
 A legal name (―State of California‖).
 A standard postal code abbreviation (―CA‖).
 An International Standards Organization (ISO) standard code (―US-CA ‖).
 A United States Federal Information Processing Standards (FIPS) code (―06 ‖).
 What is Reference Data…

Reference data sets

with cross-reference
multiple code
values representing
the same things

Reference data sets with

business definitions for each
value
What is Reference Data…

Reference data sets define

a taxonomy of data
values, specifying the
hierarchical relationships
between data values using
the Universal Standard
Products and Services
Classification (UNSPSC)
What is Master Data…
MASTER DATA MANAGEMENT (MDM)?
MASTER DATA?  The ongoing reconciliation and maintenance
» Defines and describes the nouns (things) of master data.
of the business, e.g. Field, Well, Rig,
Product, Store, Theraputic Area, Adverse
 Control over master data values to enable
Event, etc. consistent, shared, contextual use across
systems, of the most accurate, timely, and
» Data about the “things” that will
participate in events. relevant version of truth about essential
business entities.
» Provides contextual information about
events / transactions. [DAMA, the Data Management
Association]
» Stored in many systems
» Packaged Systems
MASTER DATA MANAGEMENT (MDM)?
» Line of Business Systems
Comprises a set of processes and tools that
» Spreadsheets
consistently defines and manages the
» SharePoint Lists non- transactional data entities of an
organisation.
[Wikipedia]
 What is Master Data…
» Common organizational master data includes data about:
 (1) Parties include individuals, organizations, and their roles, such as customers, citizens,
patients, vendors, suppliers, business partners, competitors, employees, students, and so on.
 MDM for customer data is also called Customer Data Integration (CDI)
 Managing business party master data poses unique challenges due to:
 The complexity of roles and relationships played by individuals and organizations.
 Difficulties in unique identification.
 The high number of data sources.
 The business importance and potential impact of the data.

 (2) Products, both internal and external.

 It may include information about bill-of-materials component assemblies, part /
ingredient usage, versions, patch fixes, pricing, discount terms, auxiliary products,
manuals, design documents and images (CAD drawings), recipes (manufacturing
instructions), and standard operating procedures
 Product Lifecycle Management (PLM) focuses on managing the lifecycle of a product or
service from its conception (such as research), through its development, manufacturing,
sale / delivery, service, and disposal
What is Master Data…
» Common organizational master data includes data about:
 (3) Financial structures, such as general ledger accounts, cost
centers, profit centers, and so on.
 Typically, an Enterprise Resource Planning (ERP) system serves as
the central hub for financial master data (chart of accounts), with
project details and transactions created and maintained in one or
more spoke applications

 (4)Locations, such as addresses.

 Location reference data typically includes geopolitical data, such as
countries, states / provinces, counties, cities / towns, postal codes,
geographic regions, sales territories, and so on.
 Location master data includes business party addresses and business
party location, and geographic positioning coordinates, such as
latitude, longitude, and altitude.
Term
 A System of Record is an authoritative system where data is
created/captured, and/or maintained through a defined set of rules
and expectations (e.g., an ERP system may be the System of Record
for sell-to customers).
 A System of Reference is an authoritative system where data
consumers can obtain reliable data to support transactions and
analysis, even if the information did not originate in the system of
reference. MDM applications, Data Sharing Hubs, and Data
Warehouses often serve as systems of reference.
 A Trusted Source is recognized as the ‘best version of the truth’ based
on a combination of automated rules and manual stewardship of
data content. A trusted source may also be referred to as a Single
View, 360° View.
 Within a trusted source, records that represent the most accurate
data about entity instances can be referred to as Golden Records.
The challenges of MDM

1) to determine the most accurate, golden data values from among

potentially conflicting data values
2) to use the golden values instead of other less accurate data

◦ Master data management systems attempt to determine the

golden data values and then make that data available wherever
needed
Primary MDM focus areas

» Identification of duplicate records within and across data sources to

build and maintain global IDs and associated cross-references to
enable information integration.
» Reconciliation across data sources and providing the “golden
record” or the best version of the truth. These consolidated records
provide a merged view of the information across systems and seek
to address name and address inconsistencies.
» Provision of access to the golden data across applications, either
through direct reads, or by replication feeds to OLTP and DW / BI
databases.
Business Driver
Master data management:
 Meeting organizational data requirements
Managing data quality
 Managing the costs of data integration
 Reducing risk
Reference data management:
 Meet data requirements for multiple initiatives and
reduce the risks and costs of data integration
through use of consistent Reference Data
 Manage the quality of Reference Data
Reference and
Master Data
Management
Activities
MDM Activities (DMBoK 2)
① Define MDM drivers and requirements
② Evaluate and assess data sources
③ Define architectural approach
④ Model master data
⑤ Define stewardship and maintenance process
⑥ Establish governance policies to enforce use
of master data
RDM Activities (DMBoK 2)
① Define drivers and requirements
② Assess data sources
③ Define architectural approach
④ Model reference data sets
⑤ Define stewarship and maintenance process
⑥ Establish reference data governance policies
(1) Understand Reference and Master Data
Integration Needs

» Analyzing the root causes of a data quality problem

Data architecture?
Human habit?
etc
» Challenges for understanding single application vs across
applications
 It is harder to understand the needs of entire enterprise
(2) Identify Reference and Master Data
Sources and Contributors

» Identify the original and interim source databases, files,

applications, organizations, and even the individual roles that create
and maintain the data.
» Understand both the up-stream sources and the down-stream needs
to capture quality data at its source.
(3) Define and Maintain the Data
integration Architecture

» Data integration architecture controls the shared access, replication,

and flow of data to ensure data quality and consistency, particularly
for reference and master data.
 RDM “Hub and Spoke” Architecture
» The database of record serves as a reference data “hub”
supplying reference data to other “spoke” applications
and databases
» Data updates approaches:
Push data through a subscribe-and-publish approach in near-
real time (asynchronous updates)
for data updates
Pull data for data
updates
 General MDM “Hub and Spoke” Architecture

» The database of record serves as a master data “hub”

supplying master data to other “spoke” applications and
databases
» Data updates approaches:
Push data through
a subscribe-and-
publish approach
in near-real time
(asynchronous
updates) for data
updates
Pull data for data
updates
Master Data Environment

DATA OWNERS

To store data models

SYSTEMS OF ENTRY MASTER DATA ENVIRONMENT and meta CONSUMING SYSTEMS
data

SYSTEM META-DATA SYSTEM

CATALOGUE
DATA DATA
SOURCING DISTRIBUTION
SYSTEM SYSTEM
SYSTEM OF
RECORD
“Best
version”
SYSTEM SYSTEM
DATA QUALITY USER INTERFACE

Standardise, De- To highlight issues

duplicate, Merge, Enrich, and enable them to
etc be fixed
DATA STEWARDS
 Master Data Environment: methods of moving Data

DATA OWNERS

SYSTEMS OF ENTRY MASTER DATA ENVIRONMENT CONSUMING SYSTEMS

ACTIVE PUSH
SYSTEM Webservices
META-DATA ACTIVE PUSH SYSTEM
Webservices
CATALOGUE
BATCH SCHEDULE BATCH SCHEDULE
LTEETL job
ETL job
EII/DV
SYSTEM EII/DV cache refresh SYSTEM
PULL ON DEMAND
SYSTEM OF PULL ON DEMAND
Webservices
Webservices RECORD SQL queries
SQL queries

SYSTEM SYSTEM
DATA QUALITY USER INTERFACE

DATA STEWARDS
Three standard “Hub” architectures

1.Repository
*A key difference is the
2.Registry number of fields that are
stored centrally

3.Hybrid
Example: Customer

Customer First Last Date of birth Preferred Preferred Credit Occupation Car
code name name delivery address delivery address
line 1 post code rating

BB005 Bob Bobson 1985-12-25 Royal Crescent BA1 7LA A Information Audi R8
Architect

ALL FIELDS
REPOSITORY
IDENTIFIERS
CORE
FIELDS
HYBRID
CORE FIELDS

IDENTIFIERS
REGISTRY
ALL FIELDS
 DATA OWNERS

SYSTEMS OF ENTRY MASTER DATA ENVIRONMENT

CONSUMING SYSTEMS
SYSTEM META-DATA SYSTEM
CATALOGUE DATA
DATA
DATA
ACTIVE PUSH DISTRIBUTION SYSTEM
SYSTEM Webservices SRYESPTOESMITOORF PULL ON DEMAND
Webservices
(CUD)
Y RECORD (R)

SYSTEM SYSTEM
DATA QUALITY USER INTERFACE

DATA STEWARDS

REPOSITORY
Master Data Environment: › Repository serves as the single source of the master data.
› Repository contains the only version of the master data.
Repository Architecture › All applications use the data in the repository via services.
› No latency or synchronisation.
 DATA OWNERS

SYSTEMS OF ENTRY MASTER DATA ENVIRONMENT CONSUMING SYSTEMS

SYSTEM META-DATA SYSTEM

CATALOGUE ?
BATCH SCHEDULE
BATCH SCHEDULE
ETL job
ETL job
EII/DV REGISTRY
SYSTEM PULL ON DEMAND SYSTEM
SYSTEM
VIRTUAL
R
Webservices
SQL queries ?
REPOSITORY

SYSTEM SYSTEM
DATA QUALITY USER INTERFACE ?

DATA STEWARDS

REGISTRY VIRTUAL REPOSITORY

Master Data Environment: › Registry only contains identifying attributes. › A virtual database built using Enterprise
› Persistent values are still stored in the source systems.
Registry Architecture
Information Integration (EII) technology.
› Registry serves as an index to source data locations. › Survivorship rules are applied to return
› Some latency due to caching. the master record.
 DATA OWNERS

SYSTEMS OF ENTRY MASTER DATA ENVIRONMENT CONSUMING SYSTEMS

ACTIVE PUSH
Webservices META-DATA ACTIVE PUSH
SYSTEM
SYSTEM Webservices
CATALOGUE ?
BATCH SCHEDULE
ETL job BATCH SCHEDULE
DISTRIBUTION
ETL job
EII/DV
PULL ON DEMAND
Webservices SYSTEM
SYSTEM PULL ON DEMAND
SRYESPTOESMITO SQL queries
Webservices
SQL queries ORFY RECORD ?

SYSTEM SYSTEM
DATA QUALITY USER INTERFACE ?

DATA STEWARDS

Master Data Environment: › Repository contains identifying information and core

Hybrid Architecture
information.
› Application-specific data is retained only in the application
HYBRID database.
› Applications still manage the full set of data.
› Core information is published back to source systems.
› 2-way synchronisation – normally some latency.
Select your MDM
Architecture
and Toolset
carefully

A hub is not the only way...

MDM… A Hub is not the only way
OPERATIONAL
DATA-STORE
E.G. CUSTOMER
MASTER
MIDDLEWARE SYNCHRONISATION LAYER

EXTRACT-TRANSFORM-LOAD
SYNCHRONISATION LAYER

ERP CRM LEGACY

ERP CRM LEGACY

MASTER DATA CONTRIBUTORS

Hub Based Master

Synchronized Master  Operational Hub structure that overlaps
operational & analytical environments
 Multiple operational systems acting as
 Supports concept of an Enterprise Data
master data contributors
Warehouse
 Real-time information availability
 Multiple systems acting as data providers
 Well suited to enterprises where data is
 Appropriate for low data latency & velocity
stored across multiple source systems
operations
 Well suited to low data velocity
 Requires careful data quality management
operations
MDM… A Hub is not the only way
INDUSTRY
SPECIFIC
MASTER
SYNCHRONISATION LAYER

SYNCHRONISATION LAYER

ERP CRM LEGACY

SINGLE DATA ERP CRM LEGACY

SOURCE

Applic Master Overlay

ation
Speci
› Stand-alone, application-neutral master
fic
data
Mast
› Industry-specific data model
er
› Well suited to vertical industries in aligning front-
office & back-office systems in real time
› One
operational
system as
master data
provider
MDM… A Hub is not the only way

SAP 1 DSL LEGACY

MASTER
DATA

MESSAGING Non
SAP 2
SAP
DV (FEDERATED) MASTER DATA

SAP 3 DW CRM
ORACLE …..
….. …..

SAP ….. …..

REPORTS SIEBEL …..

PACKAGES DATABASES FILES XML

Real Time Data Movement Data Virtualization (Federated) Master Data

 Virtual MDM hub created via DV layer
 Data is moved between the various systems
using a messaging integration hub or ESB  DV layer informed from Logical Data Model
 Data movement is done in real time  Powerful data integration to access multiple disparate
 Movement of data can be one or two way as systems
required  Rapid time to solution
 Outstanding prototyping, proof of value approach
 Compliments other Data Integration approaches
 Data Integration Service Architecture
Data integration service includes:
» Change request processing, including
review and approval.
» Data quality checks on externally
acquired reference and master data.
» Consistent application of data quality
rules and matching rules.
» Consistent patterns of processing.
» Consistent meta-data about mappings,
transformations, programs and jobs.
» Consistent audit, error resolution and
performance monitoring data.
» Consistent approaches to replicating
data (including ―subscribe and
publish‖).
(4) Implement Reference and Master Data
Management Solutions
» The implementation needs several related projects and
phases, guided by their architecture, business priorities,
and an implementation program roadmap
» Vendor solution
customer data integration (CDI)
product data integration (PDI)
some other master data subject area, such as other
parties, locations, and financial structures
(5) Define and Maintain Match Rules
» Data from different sources may have different format and standard
 Need rules for matching, merging, and linking of data from multiple systems about the
same person
 Matching is particularly challenging for data about people
» Purpose:
 to remove redundancy;
 to improve data quality; and
 to provide information that is more comprehensive
» Support by data cleansing tools and MDM applications for inference engines to
match data
» How much the confidence level of your matching rule?
 If two records share the same last name, first name, birth date, and social security number,
but the street address differs, is it safe to assume they are about the same person who has
changed their mailing address?
 If two records share the same social security number, street address, and first name, but
the last name differs, is it safe to assume they are about the same person who has changed
their last name? Would the likelihood be increased or decreased based on gender and age?
 How do these examples change if the social security number is unknown for one record?
What other identifiers are useful to determine the likelihood of a match? How much
confidence is required for the organization to assert a match?
(5) Define and Maintain Match Rules
» Three primary scenarios to match rules:
 Duplicate identification match rules focus on a specific set of fields that
uniquely identify an entity and identify merge opportunities without
taking automatic action. Business data stewards can review these
occurrences and decide to take action on a case-by-case basis.
 Match-merge rules match records and merge the data from these records
into a single, unified, reconciled, and comprehensive record. If the rules
apply across data sources, create a single unique and comprehensive
record in each database. Minimally, use trusted data from one database
to supplement data in other databases, replacing missing values or
values thought to be inaccurate.
 Match-link rules identify and cross-reference records that appear to relate
to a master record without updating the content of the cross-referenced
record. Match-link rules are easier to implement and much easier to
» Match-merge rules are complex due to the need to identify so many
possible circumstances, with different levels of confidence and trust
placed on data values in different fields from different sources
(6) Establish Golden Records
» Vocabulary Management and Reference Data
 A vocabulary is a collection of terms / concepts and their relationships
 Vocabulary management is defining, sourcing, importing, and maintaining a vocabulary and its
associated reference data
 ANSI / NISO Z39.19-2005 Guidelines for the Construction, Format, and Management of
Monolingual Controlled Vocabularies
 Vocabulary management requires data governance, enabling data stewards to assess stakeholder
needs, and the impacts of proposed changes, before making collaborative and formally
approved decisions
» Defining Golden Master Data Values
 Golden data values are the data values thought to be the most accurate, current, and
relevant for shared, consistent use across applications.
 Organizations determine golden values by analyzing data quality, applying data quality
rules and matching rules, and incorporating data quality controls into the applications that
acquire, create, and update data
 Applications can enforce data quality rules, including:
 Incorporating simple edit checks against referenced data and key business rules.
 Ensuring new records, such as addresses, that are being entered do not already exist in the
system through applying data standardization and search-before-create automation.
 Creating prompts for the user if data does not meet accuracy (this address does not exist)
expectations, while providing a way to submit exceptions that can be audited in the future.
 (6) Establish Golden Records

» Defining Golden Master Data Values (cont’)

 Assess data quality through a combination of data profiling
activities and verification against adherence to business rules
(7) Define and Maintain Hierarchies and Affiliations
» Vocabularies hierarchy (relationships between the terms in the
vocabularies)
 general-to-specific classifications (―is a kind of‖ relationships)
 whole-part assemblies (―is a part of‖ relationships)
» There may also be non-hierarchical relationships between terms in
vocabularies
» Affiliation management is the establishment and maintenance of
relationships between master data records
 Examples:
 ownership affiliations (such as Company X is a subsidiary of
Company Y, a parent-child relationship)
 associations (such as Person XYZ works at Company X)
(8) Plan and Implement Integration of New
Data Sources
» Integrating new reference data sources involves (among other
tasks):
 Receiving and responding to new data acquisition requests from
different groups.
 Performing data quality assessment services using data
cleansing and data profiling tools.
 Assessing data integration complexity and cost.
 Piloting the acquisition of data and its impact on match rules.
 Determining who will be responsible for data quality.
 Finalizing data quality metrics.
(9) Replicate and Distribute Reference and
Master Data
» Reference and master data may be read directly from a database of
record, or may be replicated from the database of record to other
application databases for transaction processing, and data
warehouses for business intelligence.
» Replication mechanism ensures referential integrity
 only valid reference data codes and master data identifiers are
used as foreign key values in other tables
» Data integration procedures must ensure timely replication and
distribution of reference and master data to these application
databases
(10) Manage Changes to Reference and Master Data
» Assign specific individuals as business data steward
 to create, update, and retire reference data values, and to a lesser extent, in
some circumstances, master data values
» Business data stewards work with data professionals to ensure the highest
quality reference and master data
» Common change request process:
1. Create and receive a change request.
2. Identify the related stakeholders and understand their interest
3. Identify and evaluate the impacts of the proposed change.
4. Decide to accept or reject the change, or recommend a decision to
management or governance.
5. Review and approve or deny the recommendation, if needed.
6. Communicate the decision to stakeholders prior to making the change.
7. Update the data.
8. Inform stakeholders the change has been made.
» Any changes to reference data that was replicated elsewhere must also be
applied to the replicated data
(10) Manage Changes to Reference and
Master Data
» How about if terms and codes are retired?
 Code tables require effective date and expiration date columns,
 Application logic must refer to the currently valid codes when
establishing new foreign key relationships.
» Carefully assess the impact of reference data changes.
Reference and Master Data Governance
 The data sources to be integrated
 The data quality rules to be enforced
 The conditions of use rules to be followed
 The activities to be monitored and the frequency of
monitoring
 The priority and response levels of data stewardships
efforts
 How information is to be represented to meet
stakeholder needs
 Standard approval gates, expectations in RDM and
MDM deployment
Metrics
Data quality and compliance
Data change activity
Data ingestion and consumption
Service Level Agreements
Data Steward coverage
Total Cost of Ownership
Data sharing volume and usage
 Guiding Principles
» Shared reference and master data belongs to the organization, not to a
particular application or department.
» Reference and master data management is an on-going data quality
improvement program; its goals cannot be achieved by one project alone.
» Business data stewards are the authorities accountable for controlling
reference data values. Business data stewards work with data professionals to
improve the quality of reference and master data.
» Golden data values represent the organization‘s best efforts at determining
the most accurate, current, and relevant data values for contextual use. New
data may prove earlier assumptions to be false. Therefore, apply matching
rules with caution, and ensure that any changes that are made are reversible.
» Replicate master data values only from the database of record.
» Request, communicate, and, in some cases, approve of changes to reference
data values before implementation.