Data All Around

Lots of data is being collected

and warehoused
Web data, e-commerce
Financial transactions, bank/credit transactions
Online trading and purchasing
Social Network
 How Much Data Do We
have?
Google processes 20 PB a day (2008)
Facebook has 60 TB of daily logs
eBay has 6.5 PB of user data + 50 TB/day (5/2009)
1000 genomes project: 200 TB

Cost of 1 TB of disk: $35

Time to read 1 TB disk: 3 hrs
(100 MB/s)
 Big Data
Big Data is any data that is expensive to manage and hard to extract value from
Volume
The size of the data
Velocity
The latency of data processing relative to the growing demand for interactivity
Variety and Complexity
the diversity of sources, formats, quality, structures.
Big Data
 Types of data based on
Structure
• Structured -
• Unstructured
• Natural language
• Machine-generated
• Graph-based
• Audio, video, and images
• Streaming
M/c gent data – Web lserver log scr
shot
Graph data
 What To Do With These
Data?
Aggregation and Statistics
Data warehousing and OLAP
Indexing, Searching, and Querying
Keyword based search
Pattern matching (XML/RDF)
Knowledge discovery
Data Mining
Statistical Modeling
 What is Data Science?

An area that manages, manipulates, extracts, and

interprets knowledge from tremendous amount of
data.

Data science (DS) is a multidisciplinary field of

study with goal to address the challenges in big data.

Data science principles apply to all data – big and

small
 What is Data Science?

Theories and techniques from many fields and disciplines

are used to investigate and analyze a large amount of data to
help decision makers in many industries such as science,
engineering, economics, politics, finance, and education.

Computer Science
Pattern recognition, visualization, data warehousing, High performance
computing, Databases, AI
Mathematics
Mathematical Modeling
Statistics
Statistical and Stochastic modeling, Probability.
 Real Life Examples

Companies learn your secrets, shopping patterns,

and preferences
For example, can we know if a woman is pregnant, even if she doesn’t want us to
know? Target case study
Data Science and election (2008, 2012)
1 million people installed the Obama Facebook app that gave access to info on “friends”
 Data Scientists

Data Scientist
The Sexiest Job of the 21st Century

They find stories, extract knowledge. They are not

reporters
 Data Scientists

Data scientists are the key to realizing the opportunities presented by

big data. They bring structure to it, find compelling patterns in it, and
advise executives on the implications for products, processes, and
decisions.
 What do Data Scientists do?
National Security
Cyber Security
Business Analytics
Engineering
Healthcare
And more ….
 Concentration in Data Science
Mathematics and Applied Mathematics
Applied Statistics/Data Analysis
Solid Programming Skills (R, Python, Julia, SQL)
Data Mining
Data Base Storage and Management
Machine Learning and discovery
Introduction to Data Analytics &
Life Cycle
 Big Data Overview

• Service Oriented Market

• Data is created constantly, and at an ever-
increasing rate.
• Mobile phones, social media, imaging
technologies to determine a medical diagnosis-all
these and more create new data, and that must
be stored somewhere for some purpose.
• Devices and sensors automatically generate
diagnostic information that needs to be stored
and processed in real time
 Big Data Overview

• Merely keeping up with this huge influx of data is

difficult, but substantially more challenging is
analyzing vast amounts of it
• Especially when it does not confirm to traditional
notions of data structure, to identify meaningful
patterns and extract useful information
 Big Data Overview

• Several industries have led the way in developing

their ability to gather and exploit data
– Credit Card to increase fraudulent detection accurracy
– Mobile phone companies analyze subscribers' calling
patterns to determine, for example, whether a caller 's
frequent contacts are on a rival network.
– For companies such as Linked In and Facebook,
data itself is their primary product
Drivers of Big Data
 Drivers of Big Data
• The data now comes from multiple sources, such as these:
– Medical information, such as genomic sequencing and diagnostic imaging
– Photos and video footage uploaded to the World Wide Web
– Video surveillance, such as the thousands of video cameras spread across
a city
– Mobile devices, which provide geospatiallocation data of the users, as
well as metadata about text messages, phone calls, and application usage
on smart phones
– Smart devices, which provide sensor-based collection of information from
smart electric grids, smart buildings, and many other public and industry
infrastructures
– Nontraditional IT devices, including the use of radio-frequency
identification (RFID) readers, GPS navigation systems, and seismic processing
 Data Science Applications/
Examples
•Healthcare: Data science can identify and predict disease, and personalize
healthcare recommendations.

•Transportation: Data science can optimize shipping routes in real-time.

•Sports: Data science can accurately evaluate athletes’ performance.

•Government: Data science can prevent tax evasion and predict

incarceration rates.
•E-commerce: Data science can automate digital ad placement.

•Gaming: Data science can improve online gaming experiences.

•Social media: Data science can create algorithms to pinpoint compatible

partners.
•Fintech: Data science can help create credit reports and financial profiles,
run accelerated underwriting and create predictive models based on historical
payroll data.
 Data Analytics Lifecycle

• Data science projects differ from BI projects

– More exploratory in nature
– Critical to have a project process
– Participants should be thorough and rigorous
• Break large projects into smaller pieces
• Spend time to plan and scope the work
• Documenting adds rigor and credibility
 Data Analytics Lifecycle

• Data Analytics Lifecycle Overview

• Phase 1: Discovery
• Phase 2: Data Preparation
• Phase 3: Model Planning
• Phase 4: Model Building
• Phase 5: Communicate Results
• Phase 6: Operationalize
 Data Analytics Lifecycle
Overview

• The data analytic lifecycle is designed for Big

Data problems and data science projects
• With six phases the project work can occur in
several phases simultaneously
• The cycle is iterative to portray a real project
• Work can return to earlier phases as new
information is uncovered
 Key Roles for a
Successful Analytics Project
 Key Roles for a
Successful Analytics Project

• Business User – understands the domain area

• Project Sponsor – provides requirements
• Project Manager – ensures meeting objectives
• Business Intelligence Analyst – provides business domain
expertise based on deep understanding of the data
• Database Administrator (DBA) – creates DB environment
• Data Engineer – provides technical skills, assists data
management and extraction, supports analytic sandbox
• Data Scientist – provides analytic techniques and modeling
 Background and Overview of Data
Analytics Lifecycle

• Data Analytics Lifecycle defines the analytics process and best

practices from discovery to project completion
• The Lifecycle employs aspects of
– Scientific method
– Cross Industry Standard Process for Data Mining (CRISP-DM)
• Process model for data mining
– Davenport’s DELTA framework
– Hubbard’s Applied Information Economics (AIE) approach
– MAD Skills: New Analysis Practices for Big Data by Cohen et al.
 Overview of
Data Analytics Lifecycle
2.2 Phase 1: Discovery
 2.2 Phase 1: Discovery

1. Learning the Business Domain

2. Resources - technology, tools, systems, data, and people
3. Framing the Problem
4. Identifying Key Stakeholders
5. Interviewing the Analytics Sponsor
6. Developing Initial Hypotheses
7. Identifying Potential Data Sources
2.3 Phase 2: Data Preparation
 2.3 Phase 2: Data Preparation

• Includes steps to explore, preprocess, and condition

data
• Create robust environment – analytics sandbox
• Data preparation tends to be the most labor-intensive
step in the analytics lifecycle
– Often at least 50% of the data science project’s time
• The data preparation phase is generally the most
iterative and the one that teams tend to underestimate
most often
 Phase 2: Data Preparation

✓ Preparing the Analytic Sandbox

✓ Performing ETLT
✓ Learning about the Data
✓ Data Conditioning
✓ Survey and Visualize
✓ Common Tools for Data Preparation
 2.3.1 Preparing the Analytic Sandbox

• Create the analytic sandbox (also called workspace)

• Allows team to explore data without interfering with live
production data
• Sandbox collects all kinds of data (expansive approach)
• The sandbox allows organizations to undertake ambitious projects
beyond traditional data analysis and BI to perform advanced
predictive analytics
• Although the concept of an analytics sandbox is relatively new,
this concept has become acceptable to data science teams and IT
groups
 2.3.2 Performing ETLT
(Extract, Transform, Load, Transform)

• In ETL users perform extract, transform, load

• In the sandbox the process is often ELT – early load
preserves the raw data which can be useful to examine
• Example – in credit card fraud detection, outliers can
represent high-risk transactions that might be
inadvertently filtered out or transformed before being
loaded into the database
• Hadoop is often used here
Big ETL,OLTP,OLAP,API
 2.3.3 Learning about the Data

• Becoming familiar with the data is critical

• This activity accomplishes several goals:
– Determines the data available to the team early in the
project
– Highlights gaps – identifies data not currently
available
– Identifies data outside the organization that might be
useful
 2.3.3 Learning about the Data
Sample Dataset Inventory
Table demonstrates one way to organize this type of data inventory.
 2.3.4 Data Conditioning

• Data conditioning includes cleaning data, normalizing

datasets, and performing transformations
– Often viewed as a preprocessing step prior to data analysis, it
might be performed by data owner, IT department, DBA, etc.
– Best to have data scientists involved
– Data science teams prefer more data than too little
 2.3.4 Data Conditioning

• Additional questions and considerations

– What are the data sources? Target fields?
– How clean is the data?
– How consistent are the contents and files? Missing or
inconsistent values?
– Assess the consistence of the data types – numeric,
alphanumeric?
– Review the contents to ensure the data makes sense
– Look for evidence of systematic error
 2.3.5 Survey and Visualize

• Leverage data visualization tools to gain an overview of

the data
• Shneiderman’s mantra:
– “Overview first, zoom and filter, then details-on-demand”
– This enables the user to find areas of interest, zoom and filter
to find more detailed information about a particular area, then
find the detailed data in that area
 2.3.5 Survey and Visualize
Guidelines and Considerations

• Review data to ensure calculations are consistent

• Does the data distribution stay consistent?
• Assess the granularity of the data, the range of values, and the
level of aggregation of the data
• Does the data represent the population of interest?
• Check time-related variables – daily, weekly, monthly? Is this good
enough?
• Is the data standardized/normalized? Scales consistent?
• For geospatial datasets, are state/country abbreviations consistent
 2.3.6 Common Tools
for Data Preparation

• Hadoop can perform parallel ingest and analysis

• Alpine Miner provides a graphical user interface for
creating analytic workflows
• OpenRefine (formerly Google Refine) is a free, open
source tool for working with messy data
• Similar to OpenRefine, Data Wrangler is an interactive
tool for data cleansing an transformation
2.4 Phase 3: Model Planning
 2.4 Phase 3: Model Planning

1. Data Exploration and Variable Selection

2. Model Selection
3. Common Tools for the Model Planning
Phase
 2.4 Phase 3: Model Planning

• Activities to consider
– Assess the structure of the data – this dictates the tools and analytic
techniques for the next phase
– Ensure the analytic techniques enable the team to meet the business
objectives and accept or reject the working hypotheses
– Determine if the situation warrants a single model or a series of
techniques as part of a larger analytic workflow
– Research and understand how other analysts have approached this kind
or similar kind of problem
 2.4 Phase 3: Model Planning
Model Planning in Industry Verticals

• Example of other analysts approaching a similar problem

 Model Planning

1. Data Exploration and Variable Selection

2. Model Selection
3. Common Tools for the Model Planning Phase
 2.4.1 Data Exploration
and Variable Selection

• Explore the data to understand the relationships among the variables to

inform selection of the variables and methods
• A common way to do this is to use data visualization tools
• Often, stakeholders and subject matter experts may have ideas
– For example, some hypothesis that led to the project
• Aim for capturing the most essential predictors and variables
– This often requires iterations and testing to identify key variables
• If the team plans to run regression analysis, identify the candidate predictors
and outcome variables of the model
 2.4.2 Model Selection

• The main goal is to choose an analytical technique, or several candidates,

based on the end goal of the project
• We observe events in the real world and attempt to construct models that
emulate this behavior with a set of rules and conditions
– A model is simply an abstraction from reality
• Determine whether to use techniques best suited for structured data,
unstructured data, or a hybrid approach
• Teams often create initial models using statistical software packages such as
R, SAS, or Matlab
– Which may have limitations when applied to very large datasets
• The team moves to the model building phase once it has a good idea about
the type of model to try
 2.4.3 Common Tools for the
Model Planning Phase

• R has a complete set of modeling capabilities

– R contains about 5000 packages for data analysis and graphical presentation
• SQL Analysis services can perform in-database analytics of common data
mining functions, involved aggregations, and basic predictive models
• SAS/ACCESS provides integration between SAS and the analytics sandbox via
multiple data connections
2.5 Phase 4: Model Building
 2.5 Phase 4: Model Building

• Execute the models defined in Phase 3

• Develop datasets for training, testing, and production
• Develop analytic model on training data, test on test data
• Question to consider
– Does the model appear valid and accurate on the test data?
– Does the model output/behavior make sense to the domain experts?
– Do the parameter values make sense in the context of the domain?
– Is the model sufficiently accurate to meet the goal?
– Does the model avoid intolerable mistakes?
– Are more data or inputs needed?
– Will the kind of model chosen support the runtime environment?
– Is a different form of the model required to address the business problem?
 2.5.1 Common Tools for
the Model Building Phase

• Commercial Tools
– SAS Enterprise Miner – built for enterprise-level computing and analytics
– SPSS Modeler (IBM) – provides enterprise-level computing and analytics
– Matlab – high-level language for data analytics, algorithms, data exploration
– Alpine Miner – provides GUI frontend for backend analytics tools
– STATISTICA and MATHEMATICA – popular data mining and analytics tools
• Free or Open Source Tools
– R and PL/R - PL/R is a procedural language for PostgreSQL with R
– Octave – language for computational modeling
– WEKA – data mining software package with analytic workbench
– Python – language providing toolkits for machine learning and analysis
– SQL – in-database implementations provide an alternative tool (see Chap 11)
2.6 Phase 5: Communicate Results
 2.6 Phase 5: Communicate Results

• Determine if the team succeeded or failed in its objectives

• Assess if the results are statistically significant and valid
– If so, identify aspects of the results that present salient findings
– Identify surprising results and those in line with the hypotheses
• Communicate and document the key findings and major
insights derived from the analysis
– This is the most visible portion of the process to the outside stakeholders
and sponsors
2.7 Phase 6: Operationalize
 2.7 Phase 6: Operationalize

• In this last phase, the team communicates the benefits of the project
more broadly and sets up a pilot project to deploy the work in a
controlled way
• Risk is managed effectively by undertaking small scope, pilot
deployment before a wide-scale rollout
• During the pilot project, the team may need to execute the algorithm
more efficiently in the database rather than with in-memory tools
like R, especially with larger datasets
• To test the model in a live setting, consider running the model in a
production environment for a discrete set of products or a single line
of business
• Monitor model accuracy and retrain the model if necessary
 2.7 Phase 6: Operationalize
Key outputs from successful analytics project
 2.7 Phase 6: Operationalize
Key outputs from successful analytics project

• Business user – tries to determine business benefits and

implications
• Project sponsor – wants business impact, risks, ROI
• Project manager – needs to determine if project completed on
time, within budget, goals met
• Business intelligence analyst – needs to know if reports and
dashboards will be impacted and need to change
• Data engineer and DBA – must share code and document
• Data scientist – must share code and explain model to peers,
managers, stakeholders
 2.7 Phase 6: Operationalize
Four main deliverables

• Although the seven roles represent many interests, the interests

overlap and can be met with four main deliverables
1. Presentation for project sponsors – high-level takeaways for executive
level stakeholders
2. Presentation for analysts – describes business process changes and
reporting changes, includes details and technical graphs
3. Code for technical people
4. Technical specifications of implementing the code
 Reference

• David Dietrich, Barry Hiller, “Data Science & Big Data Analytics”,
EMC education services, Wiley publications, 2012, ISBN0-07-
120413-X