Using Splunk Enterprise Security Course Materials

Using Splunk Enterprise Security

turn data into doing™ 1
Copyright © 2023 Splunk, Inc. All rights reserved | 31 August 2023
Document Usage Guidelines
• Should be used only for enrolled students
• Not meant to be a self-paced document, an instructor is needed
• Do not distribute
August 31, 2023

Foundational Knowledge
• To be successful, students should have a solid understanding of
the following courses:
– What is Splunk?
– Intro to Splunk
– Using Fields
– Visualizations
– Search Under the Hood
– Introduction to Knowledge Objects
– Introduction to Dashboards

Course Goals
• Use Splunk Enterprise Security (ES) to detect and identify security-
related threats
• Create investigations to determine root causes of malicious or
anomalous events
• Discover previously unknown types of potential threats
Important!
All labs must be completed for
course credit.

Course Outline
Module 1: Introduction to ES Module 6: Security Domain Dashboards
Module 2: Security Monitoring & Module 7: User Intelligence
Incident Investigation Module 8: Web Intelligence
Module 3: Risk-Based Alerting Module 9: Threat Intelligence
Module 4: Assets & Identities Module 10: Protocol Intelligence
Module 5: Investigations Module 11: Supplemental Apps
Appendix A: Reports, Dashboards, Data Models, and Use Cases

Appendix B: Event Sequence Engine
Appendix C: Interfacing with Splunk Intelligence Management (TruSTAR)
Appendix D: Cloud Security Dashboards

Module 1:
Introduction to
Enterprise Security

Objectives
• Explain how Splunk Enterprise Security (ES) helps security
practitioners prevent, detect, and respond to threats
• Give an overview of the features and capabilities of ES
• Describe data models, correlation searches, and notable events
• Define ES user roles
• Explain Splunk Web access to Splunk for Enterprise Security

Overview of Splunk Enterprise Security
• Built on the Splunk Operational Intelligence platform
– ES is a Splunk app, installed on a Splunk server
• Leverages Splunk's powerful search capabilities

• Provides tools for security practitioners to detect and respond to
security threats and incidents
• Efficiently manage, analyze, and mitigate security breaches
• Highly customizable for your specific enterprise requirements
• Real-time, scalable, context-aware, focused on content
• Makes all data — not just your “security data”— relevant to
your security effort
ES Functional Areas
Preventive Analysis
Breach Response
Perimeter Defense
Traffic analysis Audit

Vulnerability alerts
Statistical analysis Investigation journaling
Unexpected processes
Prohibited traffic Anomaly detection Incident tracking
Threat activity Pattern matching Forensics tools

Known threats Asset & identity management
Risk framework

ES Use Cases
• Malware protection
Note
– Detection Refer to the Splunk Enterprise
Security Use Cases documentation
– Use DNS data to identify “Patient Zero” for a detailed list.
– Zero-day investigations
• Insider threat
– Dataexfiltration
– Suspicious privileged account activity
• User Behavior
– Track threatening user behavior
– Classify accounts based on privileged access

ES Detecting APTs
• Enterprise Security can help detect and prevent malicious cyber
attacks like Advanced Persistent Threats (APTs)
• An APT is a growing, global threat aimed at undetected insertion,
long-term viability, extraction/delivery of valuable information
– Focused attack on specific systems like Equifax (130 million people),
Petya, Wannacry
– Targets: business, government, individuals
– Many delivery methods
– Metamorphic/polymorphic coding
– Constantly changing and adapting
https://www.splunk.com/blog/2015/06/17/opm-apt-and-the-need-for-personalized-threat-intelligence.html

APT Attack - Example
Delivery Exploita0on & Installa0on Command & Control Accomplish Mission
Threat 1
intelligence
2 MAIL WEB
Download from WEB FW
Network infected site 7
Activity/Security 8
Email 6
3
Host
Activity/Security 4 5
Identity, Roles, Privileges, Location, Behavior, Risk, Audit scope, Classification, etc.
Auth - User Roles,
Corp Context
The Kill Chain
Attackers use the kill chain methodology to devise and implement
their attacks, but defenders can also use the kill chain to counter
and those prevent attacks
Stage Attacker Activity ES Countermeasures
Delivery Email, website malware, social Threat lists, vulnerability scanning, real-time
engineering, etc. monitoring, access monitoring
Exploitation / Installation Open attachment, download from Protocol Intelligence, file system alerts, intrusion
site, upload from memory stick, etc. detection, port monitoring
Command and Control Execute code, open/copy files, Malware tracking, process alerts, change alerts,
change configuration, etc. analytics
Accomplish mission Upload payload to remote server, Traffic alerts, network analysis, audits
disable services, etc.

Who Uses ES?
Security
Security Analysts Exec/Managers
Security Auditors
SOC Staff

How ES Works
ES Searches for
Raw Events are Threats and Anomalies
Indexed Data is available for ES
ES creates notable events
Data is generated, | tstats queries and
which are stored in
forwarded, and indexed dashboards can now use
summary indexes and are
into Splunk the data
searchable by data models
Data Model Summary ES Background Searches

Searches Run (content) Process Data
CIM DM normalization is Correlation Searches, trackers,
applied, CIM DM and threat intelligence search
key/value pairs are stored data models
in DM TSIDX

How ES Works (cont.)
• Security-related data is acquired by add-ons
in your enterprise from servers, routers, etc.
– This data is forwarded to Splunk indexers and
stored as events
• ES runs searches (real-time or scheduled) for
indicators of threats, vulnerabilities, or attacks
– Ifa search discovers something that needs
attention, ES displays it on one or more of its
dashboards
– You can then investigate the issue, track it,
analyze it, and take the appropriate action

ES Data Flow
• Security-related data is acquired by add-ons in your enterprise from
servers, routers, firewalls, and other network appliances
– This data is forwarded to Splunk indexers and stored as events
Vulnerability Scanners
(port scanning, testing
Firewalls/Proxies Intrusion Detection System
vulnerabilities)
• cisco-pix (packet sniffing)
• mcafee
• pa-networks • snort
• nessus
• juniper-networks • dragon-ids
• bluecoat • mcafee
Production Servers
(any operating system)
Network Capture
(Stream) • microsoft-av
• stream:tcp • linux-secure
• stream:udp • windows:*
• stream:http Splunk ES • access-combined
(events, data models)

Data Models
• Data models normalize data
and provide a more
meaningful representation of
unstructured raw data
• ES depends heavily on
accelerated data models
• Accelerated data models provide a “speedup” factor
• Use | tstats searches with summariesonly = true to search
accelerated data

ES Dashboards and Data Models
• How does raw security data become available to ES dashboards?
1. Splunk or a custom add-on indexes and sourcetypes the raw data
2. Events are mapped and normalized to Splunk Common Information
Model (CIM) data models
3. Events are referenced by the accelerated CIM data models
• Most ES correlation searches, dashboards, and reports use the

accelerated data models
• You can create your own custom searches based on the events in
your index(es) or associated with your accelerated data models

tstats Search Example
• Use | tstats to create reports based on accelerated data models

– Use | tstats summariesonly=t to restrict results to accelerated data for
performance improvement
• Use Search > Datasets to build datasets using ES data models

ES and The KV Store
• The KV Store stores data as key-value pairs in collections on the
search head
– Collections are containers for data, similar to a database table
– Collections exist within the context of a given app, like SA-
IdentityManagement or SA-ThreatIntelligence
– Provides a way to manage and maintain the state of the app
• ES utilizes the KV Store to: Important!

ES relies on the KV Store.
– track workflow of notable events Never disable the KV Store!
– store incident review status changes and comments

– manage lookups, like assets & identities and threat intel collections
(inputlookup / outputlookup)
Correlation Searches
• Correlation Searches run continually in the background
looking for known types of threats and vulnerabilities
– There are a number of built-in correlation searches in ES, and more in
the Use Case Library. You can also create your own searches
• When a correlation search detects any Indicators of Compromise (IOC),
ES creates an alert called a notable event
• When a notable event is assigned to an analyst it is referred to as an
incident
• ES enables you to track, update, and resolve incidents
– Security Posture dashboard provides a cross-domain SOC overview
– Incident Review dashboard is used to inspect and manage incidents
Notable Events
• Correlation searches create notable events in the notable index
–A notable event might indicate a breach, vulnerability, or other issue
• Notable events are created with fields, event types, and tags that
provide information necessary for incident investigation and a link
to the original source event(s)
• You can search for the notable events in the notable index
– InES, select Search > Search to run a manual search
– Run a search like index=notable for a given time period to see the
notable events
– Event source field shows the correlation search that created the
notable event
Assets and Identities
• Notable event urgency is based on the priority of the
assets and identities in your environment
– Assets: devices in your enterprise, like routers and servers
ê Identified by IP address or MAC address
– Identities are people in your enterprise
ê Identified by username, email address, etc.
• Both are managed in the KV Store with lookup tables
– ES can show a meaningful name and descriptive information
for a server or person instead of an IP address or user ID

Beyond Notable Events
• ES provides many advanced tools which can be used to examine
security data in detail, such as:
– Risk and threat analysis
– Web and user intelligence
– Protocol (stream) intelligence
– Other adaptive responses (send email, run script, etc.)
• These tools assist analysts to:

– Perform forensic investigation of existing breaches
– Analyze the environment for new threats
– Examine the history of old breaches, understand how they happened
and prevent them in the future
ES Roles
ES Roles (required for ES login)
ES User ES Analyst ES Admin

ess_user ess_analyst ess_admin
Runs real-time searches Owns notable events Configures ES system-
and views all ES and performs notable wide, including adding
dashboards event status changes ES users, managing
correlation searches, and
adding new data sources
User Power Admin
Standard Splunk Roles

Accessing ES
• Access Splunk Web using a URL similar to:
https://eshostname:8443
• To access ES a user must have an

assigned ES role on the ES server
(ess_admin, ess_analyst, ess_user)
• Once logged on, ES displays in the list of

apps on the Splunk home page

The ES Home Page
ES Menus
Security Posture: monitor status Incident Review: work on issues
Documentation site
Configuration tools
Community support Product tutorial

Module 1 Lab: Introduction to ES
• Time: 15 minutes
• Tasks:
– Log into your Splunk classroom server, configure your user
account, and navigate to the ES home page
– Examine the source events ES is using to monitor the security
environment and notable events

Module 2:
Security Monitoring and
Incident Investigation

Objectives
• Use the Security Posture dashboard to monitor ES status
• Use the Executive Summary dashboards to view security
operations at high level
• Use the Incident Review dashboard to analyze notable events
• Take ownership of a notable event and move it through the
incident workflow
• Create notable events
• Suppress notable events

Monitoring and Response
• ES continually runs correlation searches for known threats, vulnerabilities,
authentication patterns, malware, or suspicious network traffic
– There are over 60 built-in correlation searches, and more with the
Enterprise Security Content Update (ESCU) app installed
– Or you can create your own
• When a correlation search detects any Indicators of Compromise (IOC), ES
creates an adaptive response one of which is a notable event or incident
• ES enables you to track, update, and resolve incidents
– Security Posture dashboard provides a cross-domain SOC overview
– Executive Summary dashboards to evaluate security trends
– Incident Review dashboard to inspect and manage incidents
The Security Posture Dashboard
• An overview of your
Enterprise Security
status
• Key Indicators (KI)
at the top provide an
at-a-glance view of
notable event status
over the last 24 hours
• The four panels
provide additional summary information categorized by urgency, time,
and top notable event types and sources

Key Indicators (KI)
Edit key indicators
Large number = total number of notable events in that category

Small number = trend of events indicator, red
Black = no threshold set for increase green for decrease
Red = over threshold
Green = under threshold Total increase or decrease over the
past 48 hours (from previous 24-hr
period to last 24-hr period)

KI Drilldown to Incident Review
1
From the Security
Posture dashboard,
click a KI total value
2
The details for the KI
opens in Incident Review

Security Posture Panels

Notable Event Urgency
• Each notable event has an Urgency field, ranging from
Unknown to Critical
• Urgency is a combination of
two factors:
– Severity
• Based on the severity added to
the notable event by the
correlation search
– Priority
• Assigned to the associated assets or identities—i.e., the server or user
• If more than one asset or identity is involved in a single notable event, the
one with the highest priority determines the urgency
Calculating Urgency
• Table displays how urgency values are calculated by default in notable events
⎼ Can be overwritten by modifying asset/identity priority and rank, correlation
search syntax, or Urgency Levels lookup
Event Severity
Informational Unknown Low Medium High Critical
Asset/Identity Priority
Unknown Informational Low Low Low Medium High

Low Informational Low Low Low Medium High
Medium Informational Low Low Medium High Critical
High Informational Medium Medium Medium High Critical
Critical Informational Medium Medium High Critical Critical
Asset/Identity Priority + Event Severity = Urgency

Drilldown Support
Hover over an item to
preview details about its
underlying notable events
1
Click an item to open the related notable
events in the Incident Review dashboard
2
From the Incident Review dashboard:
a. Drilldown into the details of a notable
b. Take ownership
c. Work the issue

Incident Review Dashboard
Use charts, filters, and search to focus on specific notable events
Hide the charts

or filters
Add event(s) to an investigation Actions

Expand for menu
details Notable Events
Investigation bar

Incident Review Dashboard
Hide the charts
Use charts, filters, and search to or filters
focus on specific notable events
Add event(s) to an investigation
Notable Events
Expand for details Actions

menu
Investigation bar

Incident Review Filter Fields
• Saved filters: any filters created and saved on the IR dashboard

• Search: Splunk search language expressions
• Tag: tags configured for key/value pairs
• Urgency: Informational, Low, Medium, High, Critical, Unknown
• Status: New, In Progress, Pending, Resolved, Closed
– Along with Owner, is used to track the status of an incident
• Owner: user assigned to investigate and resolve an incident
• Security Domain: Access, Endpoint, Identity, Network, Threat, Audit
• Type: Notable or Risk Notables
• Search Type: The title of a correlation search or configured sequence template
• Time or Associations: time range, short ID or Running Template
Using the Incident Review Dashboard
• Search supports full SPL and wildcard search

• Adding one or more values per field, values are ORed together
• Urgency values can be toggled on and off
– Gray values are “off” and will not be searched
• If values are set for more than one field, the fields are ANDed together
• Status, Owner, Security Domain and Tag support multiple OR values

Notable Event Details
Expand Notable event

the Actions menu
notable
event for
details
Fields for the Field
notable event, Action
with Action menus
menus for
each field
Note
You cannot expand an event until the
search is complete. Not all incidents
have all the same detail items.

Create a Short ID from Event Details
Scroll to the bottom of the details for a notable event to see the
Event Details section and create a Short ID for the event
1
Click Create Short ID for ES to automatically
generate a short ID that makes it easier to
find and share a notable event
2
The Short ID replaces
the Create Short ID link

Create a Short ID: Notable Event Actions
1
From the notable event Actions drop-
down, creating a Short ID is possible
using Share Notable Event
2
In addition to creating a Short ID, it enables sharing the event via a link:
• Click the Bookmark button to copy the link for sharing
or
• Click and drag the Bookmark button to your Bookmarks bar to save the link

Search for a Short ID or Investigation
1
Select Associations from the Time or
Associations menu, and Short ID from
the Associations menu 2
Click inside the filter field and
enter all or part of a Short ID
(drop-down appears and filters as you type)
Or
Click and scroll to the Short ID
Note
You can search for one or
multiple Short IDs.
3
Click Submit

Field Action Menu
• Each notable event field has an
Action menu allowing you to:
– Investigate the asset, set tags
or search Google. Depending
on the field type other options
may be available
• Risk scores for systems or
users are displayed next to
fields
– Click a risk score to open the
Risk Analysis dashboard for
Note
that asset or identity
Scroll the menu to make sure you
see all the available field actions.

Notable Event Actions Menu
• Each notable event has an Actions menu with options related to
the event, such as:
– Adding the event to an investigation
– Suppressing the notable event
– Sharing the notable event with others
– Initiating further adaptive response actions

Incident Workflow: Concepts
1. Assign an owner
Analysts are responsible for
2. Examine the incident changing workflow status values
as they work incidents
3. Implement corrective measures
ES Admins can define, add new status values and assign values to
different roles, so the statuses in your environment may differ
New - not yet being worked
In Progress - analysis underway Note
Pending - various: work in progress, awaiting action, etc. When a notable is assigned an owner, it is
tracked as an incident in the KV Store.
Resolved - fixed, awaiting verification
Closed - fix verified

Incident Workflow: Procedures
As needed, add
2
Click Edit selected event(s) to an
Selected investigation. It will
appear under Related
Investigations in the
1 event details 3
Select one or Set Status, Urgency, Owner,
more events and Disposition. Optionally,
add a Comment
4
Click Save changes
As needed, click the + icon on the
Investigation Bar to view an investigation,
add a new one, or click the spy glass to
perform a quick search
Investigation Bar

Incident Review History
1
Select View all review activity for
this Notable Event to open a new Tip
search showing all “review” events
for the current issue The `incident_review` macro can
be used in custom searches and
reports for incident status tracking by
directly accessing the KV Store.
2 The results show the reviewer,

urgency, status, and owner
changes for the event throughout
the review process

Notable Event Adaptive Response
• Notable events may contain
further adaptive responses
that an analyst can initiate
(ping, nslookup, change risk,
run script, etc.)
• Depending on the type of

notable event, different Adaptive Responses: Previously
executed actions
actions are available
• Use Actions > Run Adaptive
Response Actions to trigger
Next Steps: If configured in the
an action correlation search, suggests
actions to trigger next

Triggering Actions
Actions > Run Adaptive Response Actions
• Choose from a list of
actions to run
• This list is configured

by an ES admin Enter some, or all the action
name to filter
(list filters as you type)
• You may see different

options depending on
availability and
permissions

Ping Example
As you investigate, you may need to see if the affected server is up
2 3
Host Field (event field with the host to
ping (i.e., dest, src, etc.)
4 Max Results: number of results for

the ping returns (default is 1)
5
Worker Set local is optional
6
Find the Ping action in the
Note
list of Adaptive Responses,
click Ping to view the If there is an investigation selected in the
results Investigation Bar, Adaptive Responses will
display an Action column with the option to add
the response to the current investigation.

Threat Intel Example
Similarly, you can add threat artifacts to a threat collection
(needs to be configured by an admin first)
Threat Group to attribute this artifact to

1
(i.e., iblocklist_logmein (threatlist))
2 Threat Collection to add

the threat artifact to (i.e.,
ip_intel)
Field from event: a field in the

event containing the
information (i.e., dest)

Send to UBA Example
Automatically send correlation
search results to Splunk User
Behavior Analytics (UBA)
Category in UBA
Severity sets the score

in UBA for the notable
1 event (optional)
Note
UBA must be installed on the ES
search head for this Response
Action to be available.

Adaptive Response Action Center
Audit > Adaptive Response Action Center

Tagging Incidents
• Associate significant incidents
with tags
– Example: quickly find all
incidents related to servers
infected with malware
• Add a tag to each server using

Action > Edit Tags for the dest,
src or ip field (for this example)
• Search for tag “Infected” using
the Tag filter on Incident Review
• Now only notable events with Important!
this tag value will display Tags are case sensitive.

Creating and Suppressing Notable Events
• Manual creation: useful when there is source event data that has not
(yet) been identified by ES as suspicious, and you want to create a
notable event that will identify the issue and allow it to be tracked
• Suppression: useful if getting false positives from a host or a user,

and you want to exclude future notable events from that host or user
• ES Analysts do not have permission to perform these actions

– AnES Admin must give the ess_analyst and ess_user roles the
Edit Notable Event Suppressions permission

Creating Notable Events
• Create ad-hoc notable events
– For example: you find an event in
Splunk that has not triggered a
correlation search's parameters, but
you feel it should be investigated
• Steps:
1. Run a search on the source events
2. Expand an event and select
Event Actions
3. Select Create notable event
4. Enter the desired data for the
notable event and click Save

Suppressing Notable Events
Suppress notable events that
are false positives, like a
server that has been
temporarily misconfigured
From Incident Review:

1. Click the Actions drop-down for a notable event
2. Select Suppress Notable Events
3. Give the suppression a name
4. Set description and dates
5. Click Save
Note
The end date is optional. If left blank, all
future notable events from the dest field
AND signature are suppressed.

Managing Notable Event Suppressions
• View, edit, and create suppressions Important
Users with the ess_analyst or
• Select a suppression Label to edit ess_user role must be given the
Edit Notable Event Suppressions
• Enable or disable a suppression permission to perform these tasks.
Configure > Incident Management > Notable Event Suppressions

The Executive Summary Dashboards
• Select Executive Summary or SOC Operations dashboards
• Provides summary of data over several time range options
• Action menus allow for search and refresh
Time Range
Action Menu

Key Metrics Section
• Available in Executive Summary and SOC Operations dashboards
• Mean Time to Triage: time between when a notable was detected, and any
action done on that notable
• Mean Time to Resolution: time between when a notable was detected and
when its status changed to end status
• Investigations Created: number of investigations created over time period

Notables Section

Notables Section (cont.)

Risk Section
• Shows the number of regular notables versus risk notables over time
• Displays number of risk events that generated risk notables versus risk
events that did not generate risk notables over time
• Lists risk event sources not contributing to any risk notables

Additional Metrics Section
• Displays number of adaptive response actions fired in the
system over time
• Shows how many enabled sources have risk actions versus
notable actions over time
• Displays distribution of correlation searches enabled versus
disabled over time

Workload Section
• Displays assigned versus unassigned notables over time
• Shows notables assigned versus notables resolved over time
• Presents assigned open versus closed notables by analyst over time

Dispositions Section

Audit > Incident Review Audit

Module 2 Lab: Monitoring & Investigating
Time: 30 minutes
Description: An expired user account has been detected
attempting to log on to high priority resources
Tasks:
– Use the Security Posture dashboard
– Continue researching unauthorized network access
– Begin working the issue
– Test workstation status
– Remove the false positives from the list of incidents
– Resolve your incident
– Suppress notable events

Module 3:
Risk-Based Alerting

Objectives
• Give an overview of Risk and Risk-Based Alerting
• View Risk Notables and risk information in Incident Review
• Explain risk scores and how to change an object’s risk score
• Review the Risk Analysis dashboard
• Describe annotations

Risk Overview
• A risk score is a single metric that shows the relative risk of an object
(system, user, or other) in the network over time
• Risk is increased by the adaptive response associated with the
correlation search
• ES Admins can configure an object’s risk value manually or by editing
the correlation search
– Edit the Risk Analysis Response Action in a correlation search to modify the
risk score that is assigned to an object
• How is risk different from priority, severity, or urgency?
– You can see cumulative risk caused by multiple events over time
– You can fine-tune the way you interpret threats or vulnerabilities

Risk Overview (cont.)
For example: a Risk Factor can
• ES Admins can configure an object’s risk add 5 to the risk score of any
value: identity with a user_category of
“contractor”
– by editing the Risk Analysis
Response Action in a correlation
search
– by creating a Risk Factor under
Content Management
• Risk Factors specify conditions to
dynamically adjust risk scores to
specific objects
• ES Admins and ES Analysts can add
ad-hoc risk scores for objects from the
Risk Analysis dashboard
Risk Activity Examples
• User risk
– An employee who begins moving files to a local workstation and
emailing attachments to external sites
– A contractor who begins logging on from many different
geographically remote systems throughout the organization
• Asset risk
–A restricted system (like a point-of-sale station) begins running new
processes
– A server shows connections to known malicious sites on the internet
• Correlation searches can detect these events and add to the

objects’ risk score automatically
Why Risk-Based Alerting?
• Address alert fatigue and improve detection of sophisticated threats like
low-and-slow attacks that traditional SIEMs miss
• Align to cyber security frameworks like MITRE ATT&CK, Kill Chain, CIS 20,
and NIST
• Scale analyst resources to optimize SOC productivity and efficiency
Dedicated Use Splunk Enterprise Security Risk-based Alerting manual

docs.splunk.com/Documentation/ES/latest/RBA/Overview
Going further with the Risk Framework
As we’ve seen, Correlation Enrich risk attributions by When an entity’s risk score or
Searches adaptive responses appending relevant context like behavioral pattern meets the
attribute risk scores to entities a risk score or a MITRE predetermined threshold, a
when something suspicious ATT&CK technique Risk Notable event is triggered
happens. Risk attributions are
written to the risk index

Risk Rules
• The Risk Analysis
adaptive response
action, if configured in a
correlation search, is
considered a Risk Rule
• A Risk Rule feeds

results (risk attributions)
into the risk index

Risk Correlation Searches
• Risk Incident Rules are the “risk” correlation searches that run against
the risk index
• Risk Incident Rules create “Risk Notables”
• There are two out-of-the-box Risk Incident Rules
– ATT&CK Tactic Threshold Exceeded for Object Over Previous 7 days
• Creates a notable when the number of MITRE attacks exceeds 3 over
the last 7 days
– Risk Threshold Exceeded for Object Over 24 Hour Period
• Creates a notable when the risk score for an object exceeds 100 over
the last 24 hours
• Custom risk incident rules can be created
Risk-Based Alerting Example
“Risk attributions” (risk score changes) along
the timeline are put into the risk index
Risk Incident Rule creates a

Use Incident Review to Risk Notable due to the risk score
view the Risk Notable of the user exceeding 100 over a
24-hour period

Risk Notables
Filter Incident Review to

show only Risk Notables
Fields display risk information for

risk objects

Risk Notable - MITRE ATT&CK Matrix
• Expand a Risk Notable in Incident Review
• Displays which MITRE ATT&CK techniques contributed to the Risk
Notable and the tactics those techniques fall under
Click a technique to be forwarded

to the MITRE website for the full
details and documentation

Risk Notable - Details
Click the Risk Events

number to view the details
Click an individual
event for the details
Expand for
details

Threat Topology Visualization
Toggle view to
Threat Topology
Click an object to see Hover to view the

relationships object details

Entity Zones
• The A&I framework allows for the configuration of Entity Zones, an
enrichment field to differentiate events which may share
common attributes
– Prevents risk events from different zones from being grouped together
– For example, events sharing a local IP may use location data to
differentiate from what office they originated
Visible in the details of

a Risk Notable

Risk Analysis Example
A priority firewall server is attacked periodically over time
1. A few attacks are expected, but as they accumulate over weeks, the
risk score for that server increases
2. If other low priority events are also accumulating for that server, like
minor vulnerabilities and low-grade anomalous network activity, they
also contribute to the risk score for the server
3. If the risk for that server increases more than other servers due to
this continuing activity, you can be alerted and investigate
These types of issues are difficult to detect without this cumulative
approach

Risk Analysis Dashboard
Security Intelligence > Risk Analysis
Risk scores by correlation search

Object and risk score

Ad-hoc Risk Entry
• An ES Analyst can perform
a one-time ad-hoc risk
adjustment for an object
• Useful to change an
Risk Message: a description
object’s risk based on an of the adjustment
investigation
• The risk value entered is Risk Score: positive or negative
added to (or subtracted Risk object: object name (user or system)
from) the object’s overall

risk score
Risk Object Type: use other
for an unspecified object

Ad-hoc Risk Entry – Threat Objects
• Threat objects can also
be added to an ad-hoc
Threat Object: a threat object that
risk adjustment poses a threat to the environment,
including a command or a script that
• Correlate threat objects you must run
with risk events to

make adjustments to
the risk score Threat Object Type: type of threat
object like script or file_hash

Ad-hoc Risk Entry - Annotations
• Also available on the
Create Ad-hoc Risk Entry
window is the ability to add Enter annotation attributes or choose
MITRE ATT&CK annotations from
annotations the included list
• Use annotations to enrich

correlation search results
with the context from
industry-standard
mappings Create a custom
annotation
• Used as field labels in the
Risk Analysis dashboard
Save the ad-hoc entry

Annotations
ES includes the following annotations for common security
frameworks. Custom annotations can be created
Example industry-standard mappings:

View Annotation Details
Hover over an annotation to view

the risk modifier count or risk score

Module 3 Lab: Risk-Based Alerting
Time: 25 minutes
Description: Examine risk-based information and high-risk assets
or users in your environment
Tasks:
– Review the risk-based information for a risk notable
– Examine user risk information
– Manually adjust a risk score

Module 4:
Assets & Identities

Objectives
• Give an overview of the ES Assets and Identities framework
• Show examples where asset or identity data is missing from ES
dashboards or notable events
• View the Asset & Identity Management Interface
• View the contents of an asset or identity lookup table

Assets & Identities Overview
Asset and identity configuration
enhances the information available for
users and systems in notable events
and ES dashboards
ES admins add the enhanced

data for assets and identities to
ES in lookup tables
Missing Data Example 1
If enhanced data is not included in the Assets & Identities configuration
for a user or system, notable event and dashboard data is still available
for the object, though the additional information is not provided
For example, objects may show as “not known”
in the Asset or Identity Investigator
Important!
If you are expecting to see enhanced
data for a particular object, double
check the configuration in the Assets
& Identities Management interface.

Missing Data Example 2
Unknown objects in the Investigators shows a problem with Asset
and Identity configuration
Dashboards or dashboard panels may be

empty if the Asset & Identity lookups are
not configured, mis-configured, or disabled

Asset & Identity Management Interface
Asset and identity lookups and settings are configured in
Configure > Data Enrichment > Asset and Identity Management
Important!
Default ess_analyst view. Users must have
the edit_modinput_identity_manager
capability to make changes in the
A&I Management interface.
Lookup name and status

View Asset & Identity Lookups
• View the contents of a lookup table using | inputlookup
For Example: | inputlookup demo_identities.csv
| inputlookup demo_assets.csv

Identity Center & Asset Center Dashboards
View the contents of the asset or identity configuration added to ES
in the Identity Center or Asset Center dashboard
Security Domains > Identity > Identity
Center
Lookup columns and information

Module 5:
Investigations

Objectives
• Use investigations to manage incident response activity
• Use the Investigation Workbench to manage, visualize and
coordinate incident investigations
• Add various items to investigations (notes, action history,
collaborators, events, assets, identities, files and URLs)
• Use investigation timelines, lists, and summaries to document and
review breach analysis and mitigation efforts

Investigations
An investigation is a collection of activities and notes related to work
done on a specific issue, such as a breach
Example: Customers are reporting unauthorized use of their account numbers (from
your store). Start an investigation and begin researching the issue:
1. Examine notable events related to the payment processing system and add them to an
investigation.
2. Add relevant artifacts (i.e., assets and identities) and explore them within the
Investigation Workbench – this will help identify other identities involved.
3. Add collaborators with expertise in a specific field. Note
4. Run ad-hoc searches and add the results to the timeline. By default, only ess_admin and
5. Add Action History items, such as a “source” or “non-notable event”. ess_analyst have permission to start
investigations.
6. Add notes detailing actions taken to mitigate the breach.
7. Modify the investigation status. Helpful for analysts in the future, especially if you solved
the problem!

Investigations Dashboard
Lists all investigations
Filter for a specific

investigation Start a new investigation
Click an investigation to open it

Investigation Workbench
Tabs
Time range
1
Select
Artifact(s)
Note
Workbench will be blank until you
select artifact(s) and click Explore.
2
Click Explore to display
selected artifacts in the
workbench Expand panel view

Tabs & Panels
Context Panels Endpoint Data Panels Network Data Panels Risk Panels
• Risk Scores • File System Changes • Web Activity • Risk Scores
• IDS Alerts • Registry Activity • Email Data • Recent Risk Modifiers
• Notable Events • Process Activity • Network Traffic Data • MITRE ATT&CK
Techniques
• System Vulnerabilities • Service Activity • DNS Data
• MITRE ATT&CK
• Latest OS Updates • User Account Changes • Certificate Activity tactics
• Computer Inventory • Port Activity • Network Session Data
• Authentication Data

Add a Tab to the Investigation
• Add other tabs to the investigation
– For example, add the Authentication tab
Content > Add single tab > Select a tab > Authentication
ê Imports cloud-authentication-
related notable events into the
investigation
ê Displays authentication related
data relevant to the investigation
• The tabs do not persist, you must

add them each time you view the
investigation Click Save

Add Artifacts to an Investigation
• Artifacts are assets or identities you may add to an investigation to
determine whether they are involved in the overall incident
• There are several ways to add an artifact to an investigation
– From a notable event (set up by an admin)
ê Actions > Add Event to Investigation
– Manually
ê Add Artifact button
ê Add Artifact icon on the Investigation Bar
– From a workbench panel (select any item)
– From an investigation event (Timeline View > Details > click a value)

Add Artifacts Manually
1. Click Add Artifact button or click
2. Select Add artifact or Add multiple artifacts and enter the artifact(s)
(all artifacts added must be the same type: assets or identities)
3. Select either Asset or Identity artifact
4. To separate multiple artifacts, click New Line or use a comma
5. Optionally, add a Description and Label(s) (separate labels with
<Enter> or <,>)
6. Optionally, Expand artifact (seeks correlated items from lookups)
7. Click Add to Scope

Add Artifacts within the Investigation
1
When exploring, click a 2
value to add it as an artifact Enter details and click
Add to Scope

Add Items to an Investigation
It is important to add items to investigations to document the purpose of
the steps you have taken to research the issue and to provide any details
that may be useful to your team’s future investigation work. You can add
several types of entries:
• Notes • Action History items:
- Dashboards viewed
• Search strings - Notable Event Updated
• Notable or source events - Notable Event Suppression Updated
- Panel Filtered
- Search Run
Enable Add Quick Add Action
Livefeed Artifact Search Notes History
Investigation Bar

Adding a Note
1
Enter a title 2
2 Modify time as
Click to add a note needed
default = now
Note
If you create a standard note, and do 3
Enter comments
not check the Show on Timeline box,
the note will show under Notes as a
“draft” note.
4
Add attachments (text or
1 binary format). 4MB max per
Click to view notes 5
file and are stored in KV Store.

Adding an Action History Item
4
Filter search
3 Modify time as needed as needed
2
Select type
5
Select items
6
1
Select Action History

Adding a Search String (Quick Search)
• Perform a search from the Investigation Bar and add the string to an
investigation 1
Click and drag to resize

the search window.
Double click to toggle full
screen to minimized
2 3
Enter search criteria
• Analyst can run the

saved search to
view the results 4
Determine whether the
while investigating results are useful to the
investigation

Adding Events
There are several ways to add events to an investigation
Add notable events

from Incident Review
or
or
Add source events from
a search result

Enabling Notable Event Livefeed
• Get a visual notification when a notable event occurs for assets or
identities included in the investigation
– Select an investigation, click the bell icon, and toggle Enable Notification
– Bell icon turns orange within five minutes of the next occurrence Enable
Livefeed
Review events and use the plus sign (+)

to add events to the investigation

Adding Collaborators to an Investigation
1
Click + to add a
collaborator
3
Click a collaborator
initial to remove or
change write
permissions
2
Search and/or click a
username to add as a
collaborator
4
Select whether they have “write
permission” and click Done

Updating Investigation Status
• When you open an 1
investigation, the
status is New
• Investigations can
only be deleted by
admins
2
• Analysts can delete Edit the Title, Status, and
Description of the investigation
investigation entries

Investigation Summary View
Expand for details
Examine the correlation search that

created the notable event
Click to open the event in Incident

Review
Click to examine the

source notable event

Timeline: Slide View
Filter by Type: Action History, Adaptive Response Action,
Search String, Notable Event, Note, Splunk Event Edit, delete, or
open in Incident
Review
Scroll left Scroll right

(newer) (older)
Click an item to view its

details in upper panel

Timeline Details View to Add Artifacts
Click Details for a detailed view of all fields and values
1
Add Artifacts view opens

and auto-populates
3 Enter a type,
2
Click a Value to add description and
it as an artifact label as needed

Timeline: List View
From Timeline, change

view to List View Use the Action menu to
delete selected entries
View
details
Edit or delete entries or open in
Incident Review

Edit Investigation Entry
1
Click Action and select Edit Entry

to change the title of the entry
2
Enter new title and Save

Investigation Bar and Inline Timeline View
Select an investigation from the
list or click + to add a new one
Toggle the
Investigatio
n Timeline
Inline Investigation Timeline
Investigation
Timeline Zoom Entries
Jump to start

Module 5 Lab: Working with ES Investigations
Time: 50 minutes
Description: Create and manage an ES investigation
Tasks:
– Create an investigation to monitor user Hax0r over time
ê Add notable events to your investigation
ê Add an alert for results of future related notable events
– Create an investigation to monitor Snort activity
ê Find Snort events and add a Quick Search
ê Create a notable event to track status
ê Investigate source systems
ê Review your investigation from Timeline and Summary views

Module 6:
Security Domain Dashboards

Objectives
• Use ES to inspect events containing information relevant to active
or past incident investigation
• Identify security domains in ES
• Use ES security domain dashboards
• Launch security domain dashboards from Incident Review and
from action menus in search results

ES and Forensic Investigation
• When a breach occurs, you need to examine the details related to
the incident to determine a root cause and eliminate the risk
• The Security Domain dashboards provide the necessary tools to
examine related log and stream data in depth
• You can also use these dashboards as part of a periodic security
status evaluation
• The dashboards are organized by security domain

Security Domains
• Access: authentication attempts and access control
related events (login, access allowed, access failure, etc.)
• Endpoint: malware infections, system
configuration, system state (CPU usage,
open ports, uptime), patch status and
history (which updates have been applied),
and time synchronization information
• Network: information about network traffic
provided from devices such as firewalls,
routers, network-based intrusion detection
systems, and hosts
• Identity: examine identity and asset collection data

How to Use Domain Dashboards
• Use the dashboards:
– During forensic investigation of current or past security incidents
– To drill down into root causes of notable events
– Examining events related to an asset or identity you are investigating
– To periodically evaluate the status of security-related events
• Access the Security Domain dashboards from:

– The Security Domains menu
– Field Action menu in Incident Review search results

Access Domain
• The Access domain focuses on user identity and authentication
• Dashboards provide tools to research:
– Brute force attacks
– Privileged account misuse (i.e., root)
– Access by rare or new accounts
– Access by expired or disabled accounts
– Access by unusual applications (i.e., SSH, VNC, etc.)

Access > Access Center
Large failure rates High access rates from a single

indicate app like sshd can be malicious
brute force probing
High rates of login activity (user_count) may

signal a compromised system

Access > Access Search
Clicking a src or user_count under the Top Access
By Unique Users panel on the Access Center displays
the details on who is accessing the Source
Expand the events to view more

details, create a notable event, or
add the event to an investigation

Other Access Domain Dashboards
Access Tracker Account activity over time for:
• first time access
• inactive accounts
• expired identities
Account Management Account actions, like:
• creation
• deletion
• lockout
Default Account Activity Usage of default accounts, which are built-in to an operating system,
such as:
• root/administrator
• SYSTEM
• guest
**Splunk is adding new Correlation Searches all the time. Check the Enterprise Security documentation to view the specific searches
available for your version ES

Endpoint Domain
• The Endpoint domain watches over user
systems, such as:
– Workstations, PCs, notebooks
– Handheld devices
– Point-of-sale systems
• Potential issues include:

– Vulnerabilities:
missing updates or patches
– Malware: spyware, ransomware, or other
malicious code
– Unexpected running processes or services
– Unexpected registry changes

Endpoint > Malware Center
Overview of malware in your environment
Statistics on most common
infection types
Click a signature (for instance

TROJ_JAVA.BY) to drill down to
that malware type on the
Malware Search dashboard
New malware
identification

Endpoint > Malware Search
The search drills down into

the TROJ_JAVA.BY signature
showing useful information
on the infected systems

Endpoint > Endpoint Changes
Track changes on your systems by type (file, registry, etc.) or by system

Endpoint > Malware Operations
Malware status overview
Malware client information (i.e., antivirus)
Statistics on repeat and aging Infection duration statistics

infections

Endpoint > Time Center
Report the status of time synchronization in your environment
Systems not properly synchronizing will not send

correct time-stamped data to Splunk
Can lead to search failure and false negatives in ES

Other Endpoint Domain Dashboards
System Center: O/S statistics & versions in use
Update Center: patches and other software update statistics
Update Search: search interface for update events

Network Domain
• Many Network domain scenarios
are preventative in nature:
– Suspicious activity spotted by
intrusion detection systems (IDS)
– Vulnerabilities
– Unusual ports being opened
– Suspicious DNS activity
– Port scanning

Network > Intrusion Center
Events logged from intrusion detection systems (IDS)
Use filters to focus on types of

attacks. The example focuses on
trojan activity on the network
Click a result to drill down to the

Intrusion Search dashboard

Network > Intrusion Search
Clicking on a result in the Intrusion Center displays the
specifics of the attack in the Intrusion Search dashboard
Drilling down from the A Network Trojan

was Detected entry on the Intrusion Center
populates the search fields

Network > Vulnerability Center
Statistics on system security settings from vulnerability scanners
Results are looking at critical events
Click a result to drill down to the

Vulnerability Search dashboard
like MDKSA-2004:029:kernel

Network > Vulnerability Search
Clicking on a result in the Vulnerability Center displays the
specifics of the issue in the Vulnerability Search dashboard
Drilling down from the MDKSA-2004:029:kernel

entry on the Vulnerability Center populates the
Signature field

Network > Web Center
HTTP activity insights from web server, proxy, and firewall logs
Click a result to drill down to the Web Search

dashboard like the POST method
Focus in on a specific status like

404 - Page Not Found or 503 –
Service Unavailable
Focus in on a specific method like POST

Network > Web Search
Drilling down from the POST entry on the
Web Center populates the search fields

Other Network Domain Dashboards
Vulnerability Statistics on vulnerability aging and scan activity
Operations
Network Changes Events recording changes to network configurations on routers,
firewalls, etc.
Port and Protocol Analysis of network activity by port type or protocol type
Tracker
**Splunk is adding new Correlation Searches all the time. Check the Enterprise Security documentation to view the specific searches
available for your version ES

Identity Domain
• Identity domain dashboards provide
information about the assets and
identities defined in ES
• Use the Asset Center or Identity
Center to view lists of objects used
by the Assets and Identity framework
• View assets or identities by priority
level, business unit, or category
• Troubleshoot network sessions by
device or user

Identity > Asset Center
Security Domains > Identity > Asset Center
Distribution of
Distribution of
Distribution of assets by category
assets by
assets by priority
business unit
Asset lookup information

Identity > Identity Center
Security Domains > Identity > Identity Center
Identities by priority Identities by Identities by category

business unit
Identity lookup information

Identity > Session Center
Security Domains > Identity > Session Center
Use the Session Center to view an

overview of network sessions
Correlate network activity to a user

using session data provided by
DHCP or VPN servers
Review session logs and

identify the user or machine
associated with an IP address
used during a session

Module 6 Lab: Using Security Domain Dashboards
• Scenario:
– Workin the role of a network analyst performing forensic analysis on
an open incident
• Tasks:
– Use the Access Domain dashboard
– Use the Malware Search and Center dashboards
– Use the Vulnerability Center and Search dashboards
– Use the Intrusion Center dashboard

Module 7:
User Intelligence

Objectives
• Understand and use user activity analysis
• Use investigators to analyze events related to an asset or identity
• Use access anomalies to detect suspicious access patterns

Security Intelligence > User Intelligence
User intelligence tools
provide the security
practitioner with
analytical tools to find
potential internal threats
Asset Investigator Examine a specific asset, such as a server or workstation, and compare events over time
in parallel lanes showing different types of activity
Identity Investigator Examine a specific identity and compare events over time in parallel lanes showing
different types of activity
Access Anomalies A survey of network activity by users, highlighting anomalous access (one user account
being used multiple times)
User Activity A survey of people and their actions, focused on watchlisted or high-risk users

Asset and Identity Investigators
• Investigator dashboards allow you to search by an asset or
identity for a specific time range
• Both return a time-sequenced set of swim lanes showing activity
for that asset or identity
Individual bars represent

Swim lanes groups of events
Time Range Picker

Accessing the Investigators
Access investigators
From the User Intelligence menu
From a field Action menu

in an Incident Review event
ê Asset: dest, src, ip, host
ê Identity: user, src_user

Asset Investigator
Search by asset name
Asset information
Details about the

selected event
Selecting an individual bar

(set of events) shows the details in the right panel.
A darker bar has more events
Area graph shows activity over time period

Identity Investigator
Search by
identity name
Same tools and

functionality as the
Asset Investigator

Pan and Zoom
Start End
Dragging the pan/zoom controls changes the time frame for the
search and re-executes the search, showing only the activity in the
selected range

Missing Information
• If an Investigator does not find the enhanced data for an asset/identity,
you will see an error that the asset or identity is “unknown”
• Check that the asset or

identity configuration is correct
and enabled in the Assets
and Identity Management
interface

Configuring Swim Lanes
• Click Edit and select a collection
of swim lanes
• Use the Custom collection to
select specific swim lanes
• Customize swim lane colors
• ES Admins can add new swim
lanes and set overall defaults and
permissions per role Drag swimlanes up
and down into the
• Changes are not saved order you prefer

User Intelligence > User Activity
Risk assigned by various correlation

searches on user activity
Sorted by risk
Sorted by
size
Click a user to open the

Identity Investigator
Users accessing external sites

that have been added to a
watchlist
Connecting from remote locations
Incident opened in an external tracking system

Access User Activity from a Source Event
From a Splunk search result, click the
Actions menu for the user field and
select User Activity
User Activity dashboard only displays

activity for the user selected

Access Anomalies Dashboard
Security Intelligence > User Intelligence > Access Anomalies
View authentication
attempts from different IP
addresses and
improbable travel
anomalies using internal
user credentials and
location-relevant data
This dashboard is dependent on the

Important! gia_summary index, which is filled by
This search is disabled by default;
the Access - Geographically
enable it to use this dashboard. Improbable Access - Summary Gen
scheduled search hourly

Module 7 Lab: User Intelligence
• Scenario:
– You are investigating potential internal threats
• Tasks:
– Examine and learn more about the Hax0r user account
– Investigate the server that Hax0r is attempting to access
– Use the User Activity and Access Anomalies dashboards
– Use the Access Anomalies and Access Search dashboards

Module 8:
Web Intelligence

Objectives
• Use the Web Intelligence dashboards to analyze your network
environment
• Filter and highlight events

Security Intelligence > Web Intelligence
Web Intelligence contains
analytical dashboards that
are useful for inspecting
various aspects of your
website network activity
HTTP Category Explore the types of websites being accessed in the network
HTTP User Agent Examine the web user agents being used on the network
New Domain See what external domains are being accessed
URL Length Examine request URLs for unusual contents

Uses for Web Intelligence Dashboards
• Find URLs associated with unwanted activity
– HTTP Category Analysis
• Identify malicious activity in the form of long or malformed user
agent strings
– HTTP User Agent Analysis
• Detect botnet or trojan attacks by high counts of new domains
– New Domain Analysis
• Look for embedded SQL, cross-site scripting, etc.
– URL Length Analysis
docs.splunk.com/Documentation/ES/latest/User/ThreatListActivitydashboard

HTTP Category Analysis
Gives an overview of websites used by category
http://www.websense.com/content/support/library/web/v85/siem/siem.pdf
HTTP User Agent Analysis
Investigate user agent strings in proxy
data to detect potential threats

Per Panel Filter
• Some ES dashboard
views have a Per-Panel
Filter button which is
used to highlight or filter
items out of the
dashboard search
• Unavailable by default
for ES Analysts but can
be enabled by an ES
Admin
Note
In the lab environment for this course,
the Edit Per Panel Filters permission
has been enabled for ES analysts.

Filtered vs. Highlighted Events
• Filtered events are no longer displayed
• Highlighted events are:
– Highlighted in the Per-panel Filter column
– Displayed at the top of the list by default

Managing Per Panel Filtering Lookups
1
3
To remove a highlighted or filtered field,
right-click on the row and click Remove
rows. In the example, the category
”unknown” has been filtered out and is
shaded in blue in the lookup

Unhighlighting an Event
If an event is already highlighted
1. Select it
2
2. Click Per-panel Filter
3. Remove the highlight
or change to filtering 1
4. Click Save
Note
Events can only be "unfiltered" directly 3
from the lookup by removing the
corresponding row. (Filtered events
are not visible from the UI). 4

Module 8 Lab: Web Intelligence
• Scenario:
– Youare using the HTTP User Agent dashboard and notice some
unusual activity
• Tasks:
– Perform HTTP User Agent analysis
– Use a per-panel filter
– Use the HTTP Category Analysis dashboard

Module 9:
Threat Intelligence

Objectives
• Give an overview of the Threat Intelligence framework and how
threat intel is configured in ES
• Use the Threat Activity dashboard to see which threat sources are
interacting with your environment
• Use the Threat Artifacts dashboard to examine the status of threat
intelligence information in your environment

Security Intelligence > Threat Intelligence
Threat Intelligence provides
tools to help security
practitioners find and
prevent potential external
threats in your environment
Threat Activity Examine activity from a threat perspective:

• which threats have been identified
• which systems or users are affected
Threat Examine the details of threat intel that has been downloaded
Artifacts from online threat libraries, or have been added locally

Threat Intelligence Framework
• Threat intel is downloaded regularly from external and internal
sources by the Threat Download Manager modular input
– Data is parsed into KV store collections with “_intel” suffixes
– Collections are used as lookups during threat generation searches
• Threat Gen searches run by default every 5 minutes and scan for
threat activity related to any of the threat collections
– Whenthreat matches are found, events are generated in the
threat_activity index and appear in the Threat Intelligence data model
• The data model is scanned by the Threat Activity Detected
correlation search and new notables for threat activity are created

Threat Intelligence Administration
• ES Admins are tasked with managing ES threat intelligence
• Analysts and users can be given the Edit Intelligence Downloads
permission to manage threat intelligence downloads
• Threat Intelligence can be added to ES by
– downloading a feed from the Internet
– uploading a structured file
– inserting threat intelligence directly from events in ES
– as an “Add Threat Intelligence” adaptive response action in a
correlation search

Threat Intelligence Configuration
• Threat Intel is configured in the Threat Intelligence Management
interface
Configure > Data Enrichment > Threat Intelligence Management
• ES can download the following types of threat intelligence
– Threat lists: IP addresses of known malicious sites
– STIX/TAXII: details about known threats, including threat type and source
– OpenIOC: additional information about known threats
• Many intel sources require regular refresh from external sources

• This information is used by the Threat Activity Detected
correlation search

Threat Activity
The Threat Activity dashboard displays events related to known threat
sites over the desired time
Threat activity over the last 24 hours

and which collection it is from
(file_intel, ip_intel, etc.)
Per-panel Filter can be used to filter out

events that are not considered a threat, or
highlight rows that are more of a threat
Which sources (download name/URL) are
most active and the details of the threat

Using the Threat Activity Dashboard
Filter the Threat Activity dashboard
Choose a field from the
Search drop-down and and
enter a value for the search
Threat category, such

as advanced
Threat source:
persistent threat
download feed or
(APT), financial
local file name
threat, backdoor, etc.

Threat Artifacts
Filter to select a threat artifact type or filter by
fields relevant to the selected artifact type
Use the Threat Artifact menu to search drill down into the categories to see more
details about each type of threat (Network, Endpoint, Certificate, Email)
Threat Overview displays the items that have been

downloaded from threat lists or STIX/TAXII sources

Using the Threat Artifacts Dashboard
• Get more information about an active threat with the Threat
Artifacts dashboard
• Example: On the Threat Activity dashboard Most Active Threat
Sources panel, you see that iblocklist_proxy is one of the most
common threat sources
– In Threat Artifacts, you enter iblocklist_proxy in the Intel Source ID
field and search
– You learn that iblocklist_proxy is a CSV type threat list
– Use the Network tab to inspect the full list of known IP addresses
from this threat list, including locations when known

Add Threat Intelligence from a Search
• Admins can insert threat intel directly from a Splunk event
– Write a search that produces threat indicators, add the following to
the end of the search:
| outputlookup local_<threat intelligence type>_intel append=t
– Typesof <threat intelligence type> include ip, email, or

certificate
• For example: write a search that produces a list of IP addresses that
are testing a web server for vulnerabilities and add them to the
local_ip_intel lookup to be processed by the modular input and
added to the ip_intel KV Store collection

Module 9 Lab: Threat Intelligence
• Scenario:
– You are investigating potential external threats
• Tasks:
1. Review threat activity
2. Add a local IP address to the ip_intel KV Store

Module 10:
Protocol Intelligence

Objectives
• Explain how network data is input into Splunk events
• Describe stream events
• Give an overview of the Protocol Intelligence dashboards and
how they can be used to analyze network data

Security Intelligence> Protocol Intelligence
Protocol Intelligence is
ES’s set of tools for
analyzing network traffic
Protocol Center An overview dashboard showing protocol activity across the network
Traffic Size Analysis An analytical dashboard showing network traffic rates and trends
DNS Dashboards showing an overview of activity of DNS queries and a search interface
SSL Dashboards for analyzing SSL certificate activity
Email Dashboards for analyzing email activity

Getting Data In
• Capture network traffic using the Splunk Stream app or by
normalizing network log data (DNS, SSL, SMTP, HTTP)
• Uses Cases:
– Monitor suspicious network traffic
– Correlate logged vs. actual activity
– Gain direct access to network traffic for SSL, HTTP, DNS, and SMTP
– Configure correlation searches that can monitor network traffic

Splunk Stream
• Traffic can be captured using the
Splunk Stream add-on
docs.splunk.com/Documentation/StreamApp
• Deployed on forwarders and listens to traffic
• Traffic data is forwarded to indexers and made available to ES
• Additional captures can be set up within ES

Stream Data Flow
Production Servers
with forwarders and
Stream add-on or
network data
Capture network data and
forward to indexers
Splunk ES Indexers
If using the Stream app, Store captured data
it is installed here
Execute and display
search results
Captured data does not
include message content
unless specifically configured

Stream Events
• Stream events are generated from the Splunk Stream app, or other
streaming apps like Zeek (Bro) IDS
• Splunk Stream events in the
notable index are stored
with the orig_sourcetype field
as stream:xxxx (stream:tcp, stream:http etc.)
• Standard fields are extracted, as well as additional fields for the
specific source type
– HTTP: cookies, request parameters, etc.
– SMTP: sender, receiver, subject, summary of body
– DNS: DNS query, query type, DNS host, etc.

Protocol Intelligence > Protocol Center
Displays an overview of security-relevant
network protocol data
An exploited protocol
may display a
disproportionate
number of connections
for its service type
TCP connections sustained longer than 3

minutes. A long duration connection
between hosts may represent
unusual or suspicious activity

Protocol Intelligence > Traffic Size Analysis
Compare traffic data with statistical data to find outliers. Displays
traffic data from firewalls, routers, switches, or network flows
Standard Deviation Index - Percentage (%) shows the amount

of data that will be filtered out. The higher the percentage, the
fewer traffic size anomalies and details are displayed

Protocol Intelligence > DNS Activity
Displays an overview of data relevant to the DNS
infrastructure being monitored
For example, a host initiating a large number of DNS queries to

unknown or unavailable domains will report a large number of DNS
lookup failures with some successes. That pattern of DNS queries
may represent an exfiltration attempt or suspicious activity

Protocol Intelligence > SSL Activity
Provides an overview of the traffic and connections
that use SSL. Analysts can use these dashboards to
view and review SSL encrypted traffic by usage,
without decrypting the payload

Protocol Intelligence > Email Activity
Provides an overview of the data relevant to the email
infrastructure being monitored. Data can be used to
find suspect emails including, top email sources, large
emails, and rare senders or receivers

Creating a Stream Capture
• Investigating a notable or
source event?
– You can create a temporary 1
stream capture for the
source or destination server
2
ê Then investigate the
stream data for that server
• You can also Stream
capture using:
– Correlationsearch
– Adaptive response action 3
4

Scenarios: Data Exfiltration
• Detect data exfiltration using protocol intelligence dashboards:
– Email Activity? Examine Top Email Sources
ê Look for sudden spikes in email output from single accounts or
ê Spikes in the Large Emails display
– DNS Activity? Examine Queries per Domain
ê Look for unfamiliar domains getting large numbers of lookups
• See an endpoint / server that may be involved in data exfiltration?
1. Create a stream capture for it and analyze the data
2. Look for sensitive information, intellectual property, etc.

Module 10 Lab: Protocol Intelligence
• Tasks:
– UseProtocol Intelligence and related tools to investigate a suspected
data exfiltration event

Module 11:
Supplemental Apps

Objectives
• Review apps to enhance the capabilities of ES
– Splunk Mission Control
– Splunk Security, Orchestration, Automation and Response (SOAR)
– Splunk User Behavior Analytics (UBA)
– Cloud-Based Streaming Analytics
– Splunk App for PCI Compliance
– Splunk App for Fraud Analytics
– Splunk App for Lookup File Editing

Splunk Mission Control

Splunk Mission Control
• A modern, cloud-based, unified security operations platform
⎼ Brings together data, analytics, and
operations to modernize and transform the
Security Operations Center (SOC)
⎼ A modern, elegant user experience to
manage security events
⎼ Detect, manage, investigate, respond,
contain, and remediate threats – all from one
common work surface
https://www.splunk.com/en_us/products/mission-control.html

Mission Control Capabilities
• Security Analytics: A single queue of all high-fidelity incidents consisting of
prioritized Splunk ES notables and risk notables
• Standardized SOC Processes: Speed up investigations with OOTB
response templates with embedded searches, actions, and playbooks
• Orchestration, Automation, and Response: Launch SOAR playbooks to
automate tasks and actions across your security stack
• Case Management: Use response templates with custom notes containing
intelligence data to document work within an investigation
• Metrics and Reporting: Historical data from tasks are stored within Mission
Control allowing for SOC metrics, reporting, and auditability
• Embedded Splunk Search: Access native Splunk search within an incident

Integrated Capabilities
• Mission Control unifies SIEM, SOAR, and 3rd party security tools
⎼ Notable event ingestion from any data source
⎼ Notable event investigation, management, and triage
⎼ Search against Splunk Enterprise Artifact actions
⎼ Common Information Model (CIM) mappings
⎼ Dashboarding, reports, collaboration
⎼ Response templates
⎼ SLAs

Integrating with SOAR & UBA

Security Orchestration & Automation (SOAR)
• Use the Splunk App for SOAR to
send ES events to SOAR using an
ES Adaptive Response Actions
– Send to SOAR sends
ES search results to SOAR
– Run Playbook in SOAR
Runs a SOAR playbook
on the ES event
https://www.splunk.com/en_us/products/splunk-security-orchestration-and-automation.html
User Behavior Analytics (UBA)
• Splunk User Behavior Analytics (UBA)
is a separate solution that extends your
ability to detect insider threats
– Sendthreats and anomalies from UBA to ES to adjust risk scores
and create notable events
– Send correlation search results from ES to UBA to be processed

for anomalies
– Retrieve user and device association data from UBA to view it in ES
https://www.splunk.com/en_us/products/user-behavior-analytics.html

Cloud-Based Streaming Analytics
Behavioral Analytics Service in Enterprise Security

Detect Suspicious Behavior in Real Time
• Real-time insider threat detections for common threat use cases
• Seamless integration with ES to take advantage of the ES Risk-
based Alerting framework to improve visibility, true positive
detection, and reduce alert fatigue
• Scalable analytics – analyze data as it is sent, scale concurrent
searches, evolving beyond traditional scheduled searches
• Simple to deploy, cloud-native, and a low maintenance solution

Key Use Cases
A set of features embedded within ES that allows Splunk Cloud customers to
detect unknown threats within their organization

Implementation
• To enable the Behavioral Analytics Service

– On the provisioned tenant,
access the Enterprise Security
app to display the modal and
enable the service
– Select the checkbox to allow
ES to generate a token for
authentication and the ability
to send data to the Behavioral
Analytics Service Can also be enabled from
Configure > General > General Settings

Splunk App for PCI Compliance

Payment Card Industry Data Security Standard
• Payment Card Industry Data Security Standard (PCI DSS)
– An industry standard for organizations that transmit cardholder data
including, credit cards, debit and ATM cards, and
Point of Sale (POS) systems
– Protects cardholder data and minimizes the possibility of cardholder data
theft and/or loss
– Requires that all merchants, service providers, and financial institutions
meet minimum levels of security and monitoring of the systems in their
cardholder data environment (CDE)

Splunk App for PCI Compliance
The Splunk App for PCI Compliance provides the
compliance practitioner with visibility into
compliance-relevant threats found in the
cardholder data environment (CDE)
⎼ Capture, monitor, and report on data from enterprise devices, systems,
and applications in the CDE
⎼ Monitor access attempts to PCI assets
⎼ Monitor traffic between PCI domains
⎼ Identify vulnerabilities found on PCI assets
⎼ Notify administrators of malware found on PCI assets
⎼ Investigate and resolve compliance issues
⎼ Enable PCI compliance managers to monitor and report on PCI DSS compliance
by producing views and reports of significant activity

PCI Dashboards
Reports are built-in reports organized

by PCI DSS requirement
Scorecards display a
real-time summary view
of your compliance with
the PCI data security
standard in each of the
Incident Review links to the requirement areas
dashboard filtered to notable events
with governance = “pci”

Splunk App for Fraud Analytics

Splunk App for Fraud Analytics
• Leverages Splunk Enterprise Security

– Analyst can work in a familiar
incident review tab
– Fraud Incident Review includes workflow link to Investigate dashboard
– Visual link analysis to make fraud investigations quick
– Leverages Risk-based Alerting (RBA) principles
• Extensible and configurable

– All fraud rules available as correlation searches and can be modified
– Application designed with data models as the source of all searches
– Macros used to define constraints (sources for data models)

Splunk App for Lookup File Editing

Lookup Editor
• A user interface to create and edit lookups within the Splunk platform
– Provides an Excel-like interface for editing, importing, and exporting lookup files
(KV store and CSV-based)
– Ensures that lookups work in Search Head Clustered environments (edits to
lookups are propagated to search heads)
– Maintains a revision history for lookups that provides the capability to view or
restore older lookups quickly in the interface
https://docs.splunk.com/Documentation/LookupEditor

Wrap Up
• Understand how to use ES • Use risk-based alerting to monitor
• Define correlation searches and risk in your security environment
notable events • Analyze network events for
• Use the Security Posture and suspicious behavior
Incident Review dashboards • Detect insider threats
• Use the Asset and Identity • Use the Threat Intelligence
Investigators framework
• Perform forensic investigation on • Use Protocol Intelligence to
current and past incidents examine live network data
• Use adaptive response actions

What’s Next?
Become a Splunk Enterprise Security Certified Admin
This certification demonstrates an individual's ability to install, configure,
and manage a Splunk Enterprise Security deployment
Splunk Education Course(s) (recommended, but not required for this certification track). Either course path is acceptable
Exam registra>on assistance here. Study Guide here

Splunk Security Courses
• For more Splunk security training, please review these courses on
https://www.splunk.com/en_us/training.html
– Splunk User Behavior Analytics

– Administering Splunk SOAR
– Developing SOAR Playbooks
– Advanced SOAR Implementation

Community
• Splunk Community Portal • Slack User Groups
splunk.com/en_us/community.html splk.it/slack
– Splunk Answers • Splunk Dev Google Group
answers.splunk.com groups.google.com/forum/#!forum/splunkdev
– Splunk Apps • Splunk Docs on Twitter
splunkbase.com twitter.com/splunkdocs
– Splunk Blogs
splunk.com/blog/ • Splunk Dev on Twitter
twitter.com/splunkdev
– .conf
conf.splunk.com

Splunk How-To Channel
Free, short videos on a variety of Splunk topics: splk.it/How-To

Support Programs
• Web
– Documentation: dev.splunk.com and docs.splunk.com
– Splunk Lantern
Guidance from Splunk experts lantern.splunk.com
• Global Support
Support for critical issues, a dedicated resource
to manage your account – 24 x 7 x 365
– Web: splunk.com/index.php/submit_issue
– Phone: (855) SPLUNK-S or (855) 775-8657
• Enterprise Support
– Access customer support by phone and manage your
cases online 24 x 7 (depending on support contract)
Thank You

Appendix A:
Reports, Dashboards,
Data Models & Use Cases

Objectives
• Use and customize ES reports
• Use and customize ES dashboards
• Explore ES Correlation Searches
• Understand ES data models
• Use ES Content Updates or Use Case Library to pinpoint potential
issues and share them in ES

ES Reports
• Select Search > Reports
• Over 200 reports in more than 20
categories
• Execute any report by selecting
its name
Share Export
• Select Edit to open in search,
Print
modify, and save as a new report
• Use Share, Print or Export as
appropriate

ES Datasets to Build New Reports
• Access ES data models via the Search > Datasets menu
• Explore > Visualize with Pivot to quickly build new reports
– Reports can then be enhanced with charts and saved for future use

Lists of ES Correlation Searches
Table of app, security domain, name and description

of all correlation searches in your environment
Enabled correlation searches and

the adaptive response actions
Note
For a list of all enabled and disabled
correlation searches, remove
| where disabled=0.

Common Information Model
• The Common Information Model (CIM) is a library of data models
• The CIM is built into the data models that are included with ES
– Manyof the data models used by ES are actually configured in the
CIM app
• One important service provided by the CIM is normalization
• Different data sources might use different names for one logical
field name
– Example:
“Sev”, “Severity”, “SevCode”, etc. all map to the logical field
name “Severity”
docs.splunk.com/Documentation/CIM/latest/User/Overview

ES Content Updates: Analytic Stories
• Self-documented searches that
solve a specific problem
– Threat / focus of the Analytic
Story
– How to implement the
searches (data required)
• Tied to security frameworks
(Critical Security Controls, Kill
Chain, Mitre ATT&CK) https://splunkbase.splunk.com/app/3449/
• Works in ES, Splunk Enterprise

or Splunk Cloud

Content Library > Analytics Stories Stats
Search > Dashboards > Content Library
Overview of content by Analytics Stories or the searches that comprise them
Total ESCU
stories Version
Applicability to frameworks:
Kill Chain, CIS Critical
Security Controls, etc.
1
Click items in visualizations or
use drop-downs to filter details
2 Click a story row to go to its Analytics Story Detail

page

Content Library > Search Summary
The Search Summary tab has a similar structure and
process to the Analytics Stories Stats but for searches

Analytic Story Detail
Select a different story to explore
Metadata Runs all
searches in
the story and
provides a
count of
results
Detailed explanation of the story
Associations to
various security
Nested list of the story’s searches by category
frameworks
Selected
Search
(Admin) Opens the search in Edit Correlation Search in ES
Deselected
Searches
Get context, Investigate or access support

Configure > Content > Use Case Library
If your admin has
installed ESCU and
enabled Use Case
Library, you can
view ESCU analytic
stories from within
ES, bookmark
them, and add
your own

Appendix B:
Event Sequence Engine

Event Sequencing Engine
• The Event Sequencing Engine allows you to create a group of
correlation searches looking for specific fields and values
• Admins create a workflow called a Sequence Template that defines the
correlation searches and variables, and if configured, the order in which
the notable events need to occur
• A Sequenced Event is created when the workflow
triggers notable events with the configured fields
and values Note
Only ES admins, or users given the
• Similar to writing a script to automate things that Edit Sequence Templates
permission can create Sequence
you would have to do manually when tracking a Templates. ES analysts can view
the resulting events and add them
variety of notable events and variables through a to investigations.
variety of correlation searches

Example Use Case
Start (correlation search)
1. Brute Force Access Behavior Detected
Transitions (correlation searches)
2. Unusually Long Command Line
3. Uncommon Processes On Endpoint
4. Web Uploads to Non-corporate Sites by Users
5. Suspicions Reg.exe Process
End (correlation search)
6. Abnormally High Number of Endpoint Changes by User

Example Configuration
Configure > Content > Content Management > Create New Content
> Sequence Template
– The Content Management window has been filtered to show only
Sequence Templates

Example Configuration (cont.)

Resulting Sequence Event
The results of the

Sequence
Templates are
Sequenced Transitions display the
Events, which are correlation searches
matched in the template
viewed in
Incident Review

Appendix C:
Interfacing with Splunk Intelligence
Management (TruSTAR)

Splunk Intelligence Management
In addition to the Ingest of Indicators
from multiple Intelligence Sources into
Splunk KV stores for alerting, the
TruSTAR Unified app enables two Adaptive Response actions:
– Enrichthreat activity notable events
– Submit events to TruSTAR

Enrich Threat Activity Notable Events
Urgency updated by
TruSTAR normalized score
The Indicator
TruSTAR Report
Summaries of reports about

the indicator

Enrich Threat Activity Notable Events (cont.)
Pass-through/Original Score
Actor(s)
Malware Families
Normalized Score

Appendix D:
Cloud Security Dashboards

Cloud Security Dashboards
Visualize the security of your cloud infrastructure (AWS, Azure)
through several dashboards Important!
To onboard Cloud data sources and examine
your Cloud Security environment, you must
install and setup Splunk Add-on for Amazon
Kinesis Firehose and Splunk Add-on for
Microsoft Office 365 from Splunkbase.

Prerequisites
1. Create indexes to populate the Cloud Security dashboards
2. Provide index name in ES app settings
– Select Configure > General > General Settings
– Navigate to AWS Index or Microsoft 365
– Populate index name
3. Install Amazon Kinesis Firehose and Microsoft 365 add-ons

4. Configure add-ons to send data to Splunk and prepare Splunk to
receive data Note
If you are already using AWS or
Microsoft 365 TAs, you can use
these instructions to configure your
existing indexes rather than create
a new one.

Available Dashboards
• Use the Security Group dashboard to monitor activity in your
AWS environment
• Use the IAM Activity dashboard to monitor user activity in your
AWS environment
• Use the Network ACLs dashboard to monitor your network ACL
activity in the AWS environment
• Use the Access Analyzer dashboard to monitor your AWS public
facing queues, lambdas, and S3 buckets
• Use the Microsoft 365 Security dashboard to monitor security
activity in your Microsoft 365 applications

Using Splunk Enterprise Security Course Materials

Uploaded by

Copyright:

Available Formats

Using Splunk Enterprise Security Course Materials

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Using Splunk Enterprise Security Course Materials

Uploaded by

Copyright:

Available Formats

Using Splunk Enterprise Security