Introduction to Big Data

Whats Big Data?

No single definition; here is from Wikipedia:

Big data is the term for a collection of data sets so large and
complex that it becomes difficult to process using on-hand
database management tools or traditional data processing
applications.
The challenges include capture, curation, storage, search, sharing,
transfer, analysis, and visualization.
The trend to larger data sets is due to the additional information
derivable from analysis of a single large set of related data, as
compared to separate smaller sets with the same total amount of
data, allowing correlations to be found to "spot business trends,
determine quality of research, prevent diseases, link legal
citations, combat crime, and determine real-time roadway traffic
conditions.
2 2
Big Data: 3Vs

3
 Volume (Scale)

Data Volume
44x increase from 2009 2020
From 0.8 zettabytes to 35zb
Data volume is increasing exponentially

Exponential increase in
collected/generated data

4
 4.6
30 billion RFID billion
tags today
12+ TBs (1.3B in 2005)
camera
of tweet data phones
every day world wide

100s of
millions
data every day

of GPS
? TBs of

enabled
devices sold
annually

25+ TBs of 2+
log data
every day billion
people on
the Web
76 million smart meters by end
in 2009 2011
200M by 2014
 CERNs Large Hydron Collider (LHC) generates 15 PB a year
Maximilien Brice, CERN
 The Earthscope
The Earthscope is the world's largest
science project. Designed to track North
America's geological evolution, this
observatory records data over 3.8
million square miles, amassing 67
terabytes of data. It analyzes seismic
slips in the San Andreas fault, sure, but
also the plume of magma underneath
Yellowstone and much, much more.
(http://www.msnbc.msn.com/id/443635
98/ns/technology_and_science-
future_of_technology/#.TmetOdQ--uI)

7
 Variety (Complexity)
Relational Data (Tables/Transaction/Legacy Data)
Text Data (Web)
Semi-structured Data (XML)
Graph Data
Social Network, Semantic Web (RDF),

Streaming Data
You can only scan the data once

A single application can be generating/collecting

many types of data

Big Public Data (online, weather, finance, etc)

To extract knowledge all these types of

data need to linked together 8
 A Single View to the Customer

Social Banking
Media Finance

Our
Gaming
Customer Known
History

Purchas
Entertain
e
 Velocity (Speed)

Data is begin generated fast and need to be

processed fast
Online Data Analytics
Late decisions missing opportunities
Examples
E-Promotions: Based on your current location, your purchase history,
what you like send promotions right now for store next to you

Healthcare monitoring: sensors monitoring your activities and body

any abnormal measurements require immediate reaction

10
 Real-time/Fast Data

Mobile devices
(tracking all objects all the time)

Social media and networks Scientific instruments

(all of us are generating data) (collecting all sorts of data)

Sensor technology and networks

(measuring all kinds of data)

The progress and innovation is no longer hindered by the ability to collect data
But, by the ability to manage, analyze, summarize, visualize, and discover
knowledge from the collected data in a timely manner and in a scalable fashion

11
 Real-Time Analytics/Decision Requirement

Product
Recommendations Learning why Customers
Influence
that are Relevant Behavior Switch to competitors
& Compelling and their offers; in
time to Counter

Friend Invitations
Improving the Customer to join a
Marketing Game or Activity
Effectiveness of a that expands
Promotion while it business
is still in Play
Preventing Fraud
as it is Occurring
& preventing more
proactively
Some Make it 4Vs

13
 Harnessing Big Data

OLTP: Online Transaction Processing (DBMSs)

OLAP: Online Analytical Processing (Data Warehousing)
RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

14
 The Model Has Changed
The Model of Generating/Consuming Data has Changed

Old Model: Few companies are generating data, all others are consuming data

New Model: all of us are generating data, and all of us are consuming data

15
Whats driving Big Data

- Optimizations and predictive analytics

- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time

- Ad-hoc querying and reporting

- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets

16
 The Evolution of Business Intelligence
Interactive Business
Speed
Intelligence & Big Data:
In-memory RDBMS Scale
Real Time &
Single View
BI Reporting QliqView, Tableau, HANA
OLAP &
Graph Databases
Dataware house
Business Objects, SAS, Big Data: Speed
Scale
Informatica, Cognos other SQL Batch Processing &
Reporting Tools
Distributed Data Store
Hadoop/Spark; HBase/Cassandra

1990s 2000s 2010s

 Big Data Analytics

Big data is more real-time in nature

than traditional DW applications
Traditional DW architectures (e.g.
Exadata, Teradata) are not well-
suited for big data apps
Shared nothing, massively parallel
processing, scale out architectures
are well-suited for big data apps

Big data analytics is the process of

examining large datasets containing a
variety of data types -- i.e., big data -- to
uncover hidden patterns, unknown
correlations, market trends, customer
preferences and other useful business
information. 18
Why is big data analytics important?
Big data analytics helps organizations harness their data
and use it to identify new opportunities. That, in turn,
leads to smarter business moves, more efficient
operations, higher profits and happier customers. In his
report Big Data in Big Companies, IIA Director of
Research Tom Davenport interviewed more than 50
businesses to understand how they used big data. He
found they got value in the following ways:

Cost reduction. Big data technologies such as Hadoop

and cloud-based analytics bring significant cost
advantages when it comes to storing large amounts of
data plus they can identify more efficient ways of
doing business.
Faster, better decision making. With the speed of
Hadoop and in-memory analytics, combined with the
ability to analyze new sources of data, businesses are
able to analyze information immediately and make
decisions based on what theyve learned.
New products and services. With the ability to gauge
customer needs and satisfaction through analytics
comes the power to give customers what they want.
Davenport points out that with big data analytics, more
companies are creating new products to meet
customers needs.

6/16/2017
 The Data Analyst Role
The Data Analyst Role
High-level definition of the Data Analyst role

Data Analyst
Performs logical data modeling
Identifies patterns in data
Designs and creates reports

The Data Analyst is the professional whose focus of analysis and problem solving relates to data, types of
data, and relationships among data elements within a business system or IT system.

The data analyst role can vary but it can commonly involve the following:

Documenting the types and structure of the business data (logical modeling),
Analyzing and mining business data to identify patterns and correlations among the various data points,
Mapping and tracing data from system to system in order to solve a given business or system problem,
Design and create data reports and reporting tools to help business executives in their decision making,
Perform statistical analysis of business data.

6/16/2017
 What is Data Science

6/16/2017
 What is Data Science
Data science is an interdisciplinary field about
processes and systems to extract knowledge
or insights from data in various forms, either
structured or unstructured, which is a
continuation of some of the data analysis
fields such as statistics, data mining, and
predictive analytics, similar to Knowledge
Discovery in Databases (KDD)

6/16/2017
 Necessity for Data Science

6/16/2017
 Necessity for Data Science
Have a Masters /PhD/Graduate Degree in any
of the STEM fields.
Know the ABCs of programming.
Know the basics of SQL
Have passion to develop business acumen
Curious about playing with data
Familiar with the basic math and statistic
concepts
6/16/2017
 Role of Data Scientist
A data scientist represents an evolution from the business or data analyst role. The formal
training is similar, with a solid foundation typically in computer science and applications,
modeling, statistics, analytics and math. What sets the data scientist apart is strong business
acumen, coupled with the ability to communicate findings to both business and IT leaders in
a way that can influence how an organization approaches a business challenge. Good data
scientists will not just address business problems, they will pick the right problems that have
the most value to the organization.
The data scientist role has been described as part analyst, part artist. Anjul Bhambhri, vice
president of big data products at IBM, says, A data scientist is somebody who is inquisitive,
who can stare at data and spot trends. It's almost like a Renaissance individual who really
wants to learn and bring change to an organization."
Whereas a traditional data analyst may look only at data from a single source a CRM system,
for example a data scientist will most likely explore and examine data from multiple
disparate sources. The data scientist will sift through all incoming data with the goal of
discovering a previously hidden insight, which in turn can provide a competitive advantage or
address a pressing business problem. A data scientist does not simply collect and report on
data, but also looks at it from many angles, determines what it means, then recommends
ways to apply the data.
Data scientists are inquisitive: exploring, asking questions, doing what if analysis,
questioning existing assumptions and processes. Armed with data and analytical results, a
top-tier data scientist will then communicate informed conclusions and recommendations
across an organizations leadership structure.

6/16/2017
Big Data Technology

27
Thank you