GeoEvent & GeoAnalytics:

Leveraging the spatiotemporal big data store

Adam Mollenkopf Suzanne Foss

Real-Time & Big Data GIS Capability Lead, Esri Real-Time & Big Data Product Engineer, Esri
[email protected] [email protected]
@amollenkopf @sfoss_esri
Agenda
1 why a new type of data store?
2 visualizing observation data
3 writing observation data from GeoEvent Server
4 on-the-fly aggregations
5 writing analytic results from GeoAnalytics Server
6 deeper dive
1 why a new type of data store?
 ingest visualize
Web GIS 3K e/s
real-time
thousands

with real-time capabilities & big data

X
analyze store
3K e/s 200 e/s

Web GIS

X
200 e/s
GeoEvent
BYO-RDBMS
Server
 ingest visualize
Web GIS 3K e/s
real-time
thousands

with real-time capabilities & big data

X
analyze store
3K e/s 200 e/s

Web GIS

X
200 e/s
GeoEvent relational
Server data store
 ingest visualize
Web GIS 4K e/s
real-time
millions

with real-time & big data capabilities & big data

analyze store
4K e/s 10Ks e/s

Web GIS

4K e/s

GeoEvent spatiotemporal
Server big data store
 ingest visualize
Web GIS 4K e/s
real-time
millions

with real-time & big data capabilities & big data

analyze store
4K e/s 10Ks e/s

Web GIS

4K e/s

4K e/s

spatiotemporal
4K e/s big data store

GeoEvent
Server
 ingest visualize
ArcGIS Enterprise 4K e/s
real-time
millions

with real-time & big data capabilities & big data

analyze store
4K e/s 10Ks e/s

ArcGIS
Enterprise

IoT Big Data

GeoEvent spatiotemporal GeoAnalytics

Server big data store Server
 ArcGIS Enterprise
with real-time & big data capabilities

MINIMUM environment

1
ArcGIS
Enterprise

3 2 4
IoT Big Data

GeoEvent spatiotemporal GeoAnalytics

Server big data store Server

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!
 ArcGIS Enterprise
with real-time & big data capabilities

RECOMMENDED environment

1
ArcGIS
Enterprise

5 2 3 4 6 7 8
IoT Big Data

GeoEvent spatiotemporal GeoAnalytics

Server big data store Server

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!
ArcGIS Enterprise Web Device Desktop
with real-time & big data capabilities

• Ingest high velocity real-time

data into ArcGIS live features live & historic
aggregates & features
• Perform continuous analytics
on events as they are received
• Store observations in a visualization
spatiotemporal big data store
stream service map & feature service
• Run batch analytics on
stored observations
• Visualize high velocity & ArcGIS
Enterprise
volume data: ingestion
- as an aggregation analytics
analytics
- as discrete features

• Notify those who need to GeoEvent spatiotemporal GeoAnalytics

know about patterns of Server big data store Server
interest storage
2 visualizing observation data
Observation data
types of observation data

moving stationary discrete

something stands still but something that
that moves attributes change “just happens”

• planes • water gauges • crimes

• vehicles • weather stations • lightning
• animals • traffic sensors • accidents
• satellites • air quality
• storms
Observation data
defined

• An observation is a recording of a feature’s attribute values and location at a specific

moment in time.
- Observations are immutable, they happen and are typically not edited.
- Observations can be replayed over space & time.
- Moving observations are identifiable by a unique attribute, known as a TRACK_ID.
the tracks of moving observations
space & time tracks can be reconstructed

1 3
AM 3
3
2
1
2 2
T
PM
Observations can be Moving observations can
shown at a specific time be identified by a unique attribute
Moving observation illustration for two tracks
over space (X, Y) and time (T)
On-the-fly aggregations
of observations

• The spatiotemporal big data store enables you to aggregate data on-the-fly
- aggregating on-the-fly accommodates for real-time data
- you can perform exploratory analysis via any combination of space, time and attribute queries
- toggle from aggregation to raw feature view
- access feature level attributes in aggregate or feature level view
Demo

visualizing observation data

map & feature service
 writing observation data
3
from GeoEvent Server
 Writing observation data
from GeoEvent Server

ArcGIS
Enterprise
map & feature service

GeoEvent Service data source

input output
IoT

GeoEvent spatiotemporal
Server big data store
Demo

Writing observation data

into a spatiotemporal big data store
from GeoEvent Server
4 on-the-fly aggregations
Preparing data for on-the-fly aggregations
spatial indexing: geohash

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for geohash aggregation is continuously updated

geohash aggregation (based on a geohash index)

Preparing data for on-the-fly aggregations
spatial indexing: geohash & square

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for geohash aggregation is continuously updated
- a spatial index for square aggregation is continuously updated

geohash aggregation (based on a geohash index) square aggregation (based on a square index)
Preparing data for on-the-fly aggregations
spatial indexing: triangle

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for ‘pointy’ triangle aggregation is continuously updated
- a spatial index for ‘flat’ triangle aggregation is continuously updated

pointy triangle aggregation (based on a pointy triangle index) flat triangle aggregation (based on a flat triangle index)
Preparing data for on-the-fly aggregations
spatial indexing: hexagon (same as triangle)

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for ‘pointy’ hexagon (pointy triangle) aggregation is continuously updated
- a spatial index for ‘flat’ hexagon (flat triangle) aggregation is continuously updated

pointy hexagon aggregation (based on a pointy triangle index) flat hexagon aggregation (based on a flat triangle index)
Preparing data for on-the-fly aggregations
spatial indexing

• as data is written to a data source in the spatiotemporal big data store:

- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle
- this is in addition to a temporal index on the time field
- and an inverted index on each of the attribute fields

pointy pointy
geohash triangle hexagon

flat flat
square triangle hexagon
Visualizing on-the-fly aggregations
geohash
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
feature service
feature service lodType=geohash&lod=2
map service lodType=geohash&lod=2 &returnGeometry=false

geohash aggregation response geohash aggregation response geohash aggregation response

Visualizing on-the-fly aggregations
square
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
feature service
map service lodType=square&lod=2

square aggregation response square aggregation response

Visualizing on-the-fly aggregations
hexagon aggregation
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
feature service
map service lodType=flatHexagon&lod=2

flat hexagon aggregation response flat hexagon aggregation response

Visualizing on-the-fly aggregations
spatial projection

• some of the spatial indices support projection(s):

- the geohash spatial index only supports GCS WGS 1984, however you can project on-the-fly
- square, pointy & flat hexagon/triangle spatial indices are defined with a spatial reference

geohash aggregation in world mollweide projection (wkid = 54009) square aggregation in world mollweide projection (wkid = 54009)
 Demo

Visualizing on-the-fly aggregations

geohash, square & hexagon
spatial projection

http://github.com/esri/aggregation-viewer-server-map-service http://github.com/esri/aggregation-viewer-client-feature-layer
 writing analytic results
5
from GeoAnalytics Server
 Writing analytic results
from GeoAnalytics Server

ArcGIS
Enterprise

IoT Big Data

GeoEvent spatiotemporal GeoAnalytics

Server big data store Server
GeoAnalytics Server
analytic capabilities

Summarize Data Analyze Patterns

Aggregate Points Calculate Density
Summarize Nearby Find Hot Spots
Summarize Within Create Space Time Cube
Reconstruct Tracks
Create Panel Use Proximity
Create Buffers
Find Locations
Find Existing Locations Manage Data
Find Similar Locations Copy to Data Store
Join Features

* time aware tools

Writing analytic results
from GeoAnalytics Server

• GeoAnalytics Server has the ability to:

- perform analytics against data sources in the spatiotemporal big data store
- and write the analytic results to the spatiotemporal big data store

new Web GIS layers

Web GIS layers

use your Web GIS layers through Pro, Portal,

ArcGIS Python notebooks or the REST API
Enterprise

GeoEvent spatiotemporal GeoAnalytics

Server big data store Server
Writing analytic results
from GeoAnalytics Server

• GeoAnalytics Server has the ability to:

- perform analytics against files in an external big data file share
.csv .shp
- and write the analytic results to the spatiotemporal big data store
text files shapefiles

new Web GIS layers

Web GIS layers

Hive Hadoop
file system

ArcGIS
Enterprise

big data
file shares

GeoEvent spatiotemporal GeoAnalytics

Server big data store Server
feature
service
 Loading data
via ‘Copy to Data Store’ GeoAnalytics tool

.csv
text files
big data
file shares
.shp
shapefiles

spatiotemporal
big data store

Hive

relational
data store
Hadoop
file system

GeoAnalytics tool interface

feature
service

output features
 Writing analytic results
via ‘Reconstruct Tracks’ GeoAnalytics tool

.csv
text files
big data
file shares
.shp
shapefiles

spatiotemporal
big data store

Hive

relational
data store
Hadoop
file system

feature GeoAnalytics tool interface

service
Writing analytic results
via ‘Reconstruct Tracks’ GeoAnalytics tool

input features GeoAnalytics tool interface

output features
6 deeper dive
- operating system requirements
- resource requirements
- data reliability
- ObjectId
 Operating system requirements
64-bit Windows or Linux

resource minimum RECOMMENDED notes

64-bit Windows Server: Windows Server: ‘_’ characters are
operating system 2012 R2 Standard & Datacenter (Sep 2016) 2012 R2 Standard & Datacenter (Sep 2016) not allowed in
(Windows) 2012 Standard & Datacenter (Sep 2016) machine names
2008 R2 Standard, Enterprise & Datacenter (sp1)
2008 Standard, Enterprise & Datacenter (sp2)

64-bit Red Hat Enterprise Linux Server 7 (update 8) Use your Linux flavor of choice with the
operating system Red Hat Enterprise Linux Server 6 (update 2) latest update version Esri has certified.
(Linux) CentOS Linux 7 (7.2)
CentOS Linux 6 (6.8)
Scientific Linux 7 (7.2)
Scientific Linux 6 (6.8)
Ubuntu Server LTS (16.04.1)
SUSE Linux Enterprise Server 12 (sp1)
SUSE Linux Enterprise Server 11 (sp4)
Oracle Linux 7 (update 2)
Oracle Linux 6 (update 8)

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
Resource requirements
processors

resource minimum RECOMMENDED notes

More processors results in
faster writes (indexing time),
processors 8 cores 16 cores higher write throughput &
faster query response time.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
Resource requirements
processors, memory

resource minimum RECOMMENDED notes

More processors results in
faster writes (indexing time),
processors 8 cores 16 cores higher write throughput &
faster query response time.

Storage need is dictated by data velocity & size.

storage 400 GB 1 TB or more SSD results in much faster writes & queries.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
Resource requirements
processors, memory, storage

resource minimum RECOMMENDED notes

More processors results in
faster writes (indexing time),
processors 8 cores 16 cores higher write throughput &
faster query response time.

Storage need is dictated by data velocity & size.

storage 400 GB 1 TB or more SSD results in much faster writes & queries.

More memory results in

memory 16 GB 32 GB higher likelihood of query cache hits.
32 GB is the maximum due to JVM limitations.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
Resource requirements
processors, memory, storage & instances

resource minimum RECOMMENDED notes

More processors results in
faster writes (indexing time),
processors 8 cores 16 cores higher write throughput &
faster query response time.

Storage need is dictated by data velocity & size.

storage 400 GB 1 TB or more SSD results in much faster writes & queries.

More memory results in

memory 16 GB 32 GB higher likelihood of query cache hits.
32 GB is the maximum due to JVM limitations

With one instance there is risk of data loss.

# of With three instances replicas are written
increasing data reliability.
ISOLATED 1 3 or more The # of instances will vary based on the velocity of
instances incoming data as well as the data retention policy
configured on each data source.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
Data reliability
shards & replication factor

spatiotemporal
big data store
T1
node 1
T3

T2 r=1
node 5 node 2
GeoEvent GeoAnalytics
Server Server
T3
node 4 node 3
T1 T2
Data reliability
auto-rebalancing of data upon node membership changes, + or -, in the big data store

spatiotemporal
big data store
T1
x
T3

T2 r=1
node 5 node 2
GeoEvent T3 T1
GeoAnalytics
Server Server
T1 T3
node 4 node 3
T1 T2
Data reliability
data retention policies, configured per data source

spatiotemporal
big data store

node 1

r=1
node 5 node 2
GeoEvent GeoAnalytics
Server Server

node 4 node 3

purge based on
data retention
Data reliability
rolling index option, set appropriately to the velocity of your observation data

spatiotemporal
big data store

node 1
indices

r=1
node 5 node 2
GeoEvent GeoAnalytics
indices indices
Server Server

node 4 node 3
indices indices

purge based on
data retention
Data reliability
automatic data backups using periodic snapshots, including ability to restore from a snapshot

spatiotemporal
big data store

node 1

r=1
node 5 node 2
GeoEvent GeoAnalytics
Server Server

node 4 node 3

purge based on
data retention

snapshot-2016-05-17-11-0-0.snapshot
snapshot-2016-05-17-12-0-0.snapshot
…
ObjectId
choosing an Object Id option
ObjectId
choosing an Object Id option
ObjectId
choosing an Object Id option

Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …

events per day Int32

1,000 e/s 86,400,000 25 days
10,000 e/s 864,000,000 2.5 days
100,000 e/s 8,640,000,000 6 hours
1,000,000 e/s 86,400,000,000 36 minutes
10,000,000 e/s 864,000,000,000 4 minutes
ObjectId
choosing an Object Id option

Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps

events per day Int32

1,000 e/s 86,400,000 25 days
10,000 e/s 864,000,000 2.5 days
100,000 e/s 8,640,000,000 6 hours
1,000,000 e/s 86,400,000,000 36 minutes
10,000,000 e/s 864,000,000,000 4 minutes
ObjectId
choosing an Object Id option

Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps

events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years
10,000 e/s 864,000,000 2.5 days 29,247,200 years
100,000 e/s 8,640,000,000 6 hours 2,924,720 years
1,000,000 e/s 86,400,000,000 36 minutes 292,472 years
10,000,000 e/s 864,000,000,000 4 minutes 29,248 years
ObjectId
choosing an Object Id option

Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps
UniqueStringID n/a unlimited JavaScript, custom apps

events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years
10,000 e/s 864,000,000 2.5 days 29,247,200 years
100,000 e/s 8,640,000,000 6 hours 2,924,720 years
1,000,000 e/s 86,400,000,000 36 minutes 292,472 years
10,000,000 e/s 864,000,000,000 4 minutes 29,248 years
Summary
spatiotemporal big data store

• The spatiotemporal big data store enables:

- GeoEvent Server to write high velocity & volume observation data
- On-the-fly aggregations to be visualized using Map & Feature Services
- GeoAnalytics Server to read & write high volume analytic results

• To learn more:
- See the ‘Spatiotemporal Big Data Store’ tutorial
http://links.esri.com/geoevent-sbds
Real-Time & Big Data GIS
other sessions

• Applying Real-Time Analytics Mon, 1:45pm-2:45pm, Room 147B

Tue, 11:00am-12:00pm, Room 151B

• Leveraging the Spatiotemporal Big Data Store Mon, 3:00pm-4:00pm, Room 147B
Tue, 1:30pm-2:30pm, Room 151B

• ArcGIS GeoAnalytics: Intro to Big Data Feature Analytics Mon, 4:15pm-5:15pm, Room 147B
Tue, 2:45pm-3:45pm, Room 147A

• Leveraging Stream Services Tue, 2:45pm-3:45pm, Room 151A

• ArcGIS GeoEvent Server: Best Practices Tue, 4:00pm-5:00pm, Room 151A

• Real-Time & Big Data GIS at a Massive Scale Tue, 5:15pm-6:15pm, Room 151A
Please Take Our Survey on the Esri Events App!

Download the Esri Events Select the session Scroll down to find Complete Answers
app and find your event you attended the survey and Select “Submit”
Print Your Certificate of Attendance
Print stations located in the 140 Concourse

Monday Tuesday
12:30 PM – 6:30 PM 10:45 AM – 5:15 PM
GIS Solutions Expo, GIS Solutions Expo,
Hall B Hall B

5:15 PM – 6:30 PM 6:30 PM – 9:30 PM

Expo Social, Networking Reception,
Hall B Smithsonian National Air
and Space Museum
 Questions / Feedback?
To learn more:
http://links.esri.com/geoevent
http://links.esri.com/geoevent-forum

Adam Mollenkopf Suzanne Foss

Real-Time & Big Data GIS Capability Lead, Esri Real-Time & Big Data Product Engineer, Esri
[email protected] [email protected]
@amollenkopf @sfoss_esri