BRKSEC-3771 Yazji v6 Final v2 2

Advanced Web Security
Deployment & Troubleshooting with

WSA in IPv4 & IPv6 Networks
Bill Yazji – Consulting Security Engineer

BRKSEC-3771
Agenda
• Introduction
• Deploying WSA
• Troubleshooting HTTPS Issues
• Troubleshooting Performance Issues
• Cognitive Threat Analytics & WSA
• ThreatGRID & WSA
BRKSEC-3771 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
About Me
Bill Yazji
Consulting Security Engineer
byazji[at]cisco.com
Lab Details
 https://labops-out.cisco.com/labops/ilt
 Email address: [email protected]
 lab Class ID : adaron_v23172
Deploying WSA
Explicit Proxy At-A-Glance
• Pro’s • Con’s
• Most web-enabled applications support • Requires a solid AD architecture
NTLM enabled proxy-style authentication design to facilitate an easy staged
• Easier for Network/Security admin to deployment
troubleshoot, as the traffic flow is easy to • More interaction with server team for
review via packet capture GPO rollouts
• Ability to offer load balancing without any • Deployment back-out dependent on
external hardware (via proxy.pac) AD/GPO update policy/frequency
• “AutoDetect” proxy setting will work simply • Manual configuration required for non-
with DNS and DHCP (Option 252) settings domain controlled workstations
• May be able to remove default route on • Easier to circumvent
network, which can help with security by
keeping rogue applications from finding
their way out
• Easy to test during pre-deployment
Explicit Proxy with IPv4 & IPv6
 Client requests a website
 Browser connects first to WSA using IPv4 or IPv6
 WSA does DNS lookup
A record returned and/or AAAA record returned
 Depending on WSA setting, WSA builds outgoing connection either on IPv4 or IPv6
 Firewall usually only allows web traffic for proxy
Web Security Appliance

Internet Web
server
IPv6
IPv4
Internet
ASA Firewall
Explicit Mode with IPv4 & IPv6
Setting IPv6 Addresses on Interface
Setting Routes
Setting DNS Server
Can add IPv4 and
IPv6 DNS Servers
Which Protocol should be prefered in

case of A and AAAA record returned?
Redundancy using CARP
Common Address Redundancy Protocol
• CARP provides virtual IP

• Works with IPv4 and IPv6
Internet
• Requires L2 Connectivity
• Communication done via Multicast
• One Master, multiple Slaves
• Useful when no hardware load balancer

exists for explicit deployments
L2 Network
Virtual IP
Redundancy using CARP (2)
Redundancy Group for IPv4 & IPv6
Higher Value = Master
Testing via CLI – “FAILOVERCONFIG”
Testing via CLI – “TESTFAILOVERGROUP”
Transparent Proxy At-A-Glance
• Pro’s • Con’s
• Generally no need to touch end user • Auth, auth, auth!!!. Generally, only
device for authentication (possibly web browsers are able to handle the
may need to edit Intranet zone security style of authentication required for
settings) transparent connections. Requires
• Able to force all traffic to proxy if heavier use of IP surrogates, which
desired, without end user interaction may not be favorable/possible due to
• Staggered/Staged deployment easy network configuration – and cookie
surrogates aren’t shared between
with the usage of WCCP ACLs
browsers/applications.
• May reduce the need for usage of • WCCP can sometimes be
SOCKs Client
cumbersome to enable, requires
• Load balancing inherent without usage review into routing/switching code.
of hardware load balancers/pac file
• Easy to back-out during deployment
(simply remove redirection)
Transparent Proxy via WCCP
 Client requests a website
 DNS Resolution is done by the client
 Browser tries to connect to Website (follows default route)
 Network Device redirects traffic to WSA using WCCP
 WSA proxies the request (and does its own DNS query)
Web Security Appliance
IPv6 Internet Web

server
IPv4
Internet
ASA Firewall
WCCP Details
Assignment
The WCCP assignment method is used to determine which WCCP traffic and which
WCCP device is chosen for the destination traffic.
WCCP can use two types of Assignment Methods: Hash and Mask.
 Hash Based Assignment

Uses a software based hash algorithm to determine which WCCP appliance
receives traffic. In hardware based platforms the Netflow table is used to apply
hardware assistance.
 Mask Based Assignment
Uses the ACL TCAM to assign WCCP entities. This method is fully handled by
hardware.
WCCP input redirect (wccp-in)
WCCP Input redirect
Ingress Egress
Interface Interface
WCCP output redirect and input exclude
WCCP Output redirect
Ingress Egress
Interface Interface
WCCP Exclude-in
How WCCP registration works
1. Registration
2. Here I am
3. I see you WCCP Client

WCCP Server
(WSA/Proxy)
 The WCCP client registers at the WCCP Server
 Both, Server and Client need to use the same WCCP Service Group ID
 One WCCP Server usually can server multiple Clients
 Server and Client exchange Here I Am and I See You Packets to check availability
UDP/2048, unicast
Multicast possible
 Traffic is redirected from Server to one or multiple Clients using the hash or mask algorithm
WCCP Protocol – Load balancing and Redundancy
 When a WCCP client fails, the portion of the load handled by that client is automatically
redistributed to the remaining WCCP clients in the service group
 If no other WCCP clients are available in the service group, the service group is taken offline
and packets are forwarded normally
Buckets 86–128 Buckets 129–170

Buckets 1–85 Buckets 86–170 Buckets 171–255
A
X
B C
Using WCCP for Traffic Redirection
• WCCPv2 support is available on many Cisco Platforms
• WCCPv2.01 is IPv6 Capable
• WSA supports & negotiates all redirect and assign methods (software
implementation)
• Multiple WSAs elect a “Designated Web Cache” (DWC), lowest IP in Cluster,
negotiates method
• How to force GRE? Set WSA to “Allow GRE”
Using WCCP for Traffic Redirection
• Load Balance based on CLIENT address for best performance
Using WCCP for Traffic Redirection (2)
• Performance Considerations:
• MASK (HW) > HASH (SW)
• HW has to take TCAM Resources into consideration
• L2 (HW) > GRE (SW)
• Use GRE if WSA is located in other subnet
• Check if Device can do GRE in HW
• Use L2 if WSA and WCCP Device are in same subnet
WCCP Protocol - Service Group
• The routers/switches and WCCP clients participating in a WCCP service

constitute a Service Group
• Up to 32 routers per service group
• Up to 32 WCCP clients per service group
• Each service group is established and maintained using separate protocol
message exchanges
• Service definition must be the same for all members of the service group
WCCP with L3 Switch
L2 Redirect
Use template “access”, “routing” or “dual-

Internet ipv4/ipv6 routing”
WCCP shares same TCAM Region than
PBR!
VLAN10
sdm prefer routing
ip routing
ip wccp 91 redirect-list wsa
ip access-list extended wsa
VLAN11 permit tcp any any eq www
permit tcp any any eq 443
!
interface Vlan10
ip address 172.16.10.10 255.255.255.0
ip wccp 91 redirect in
WCCP with L3 Switch
L2 Redirect
Internet
 Recommendations:
Assign seperate VLAN for the
VLAN40 connection to the WSA!
 Redirect ACL only allows „permit“

statements on 3560/3750 Series!
VLAN10 12.2(58) added support for „deny“
 If 3560/3750 is stacked, configure

WCCP on the Stack Master!
WCCP IPv6
Internet
ipv6 wccp 91 redirect-list wsav6

VLAN40 !
interface Vlan10
ip address 172.16.10.10 255.255.255.0
ipv6 address 2001:db8:1:10::66/64
VLAN10 ipv6 nd ra suppress
ipv6 wccp 91 redirect in
ipv6 access-list wsav6

permit tcp 2001:DB8:1:10::/64 any eq www
permit tcp 2001:DB8:1:10::/64 any eq 443
BRKSEC-3771 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34 34
WCCP IPv6 & IPv4
Different service groups for IPv4 & IPv6
Internet
ip wccp 90 redirect-list wsav4
ipv6 wccp 91 redirect-list wsav6
!
interface Vlan10
ip address 172.16.10.10 255.255.255.0
VLAN40
ipv6 address 2001:db8:1:10::66/64
ipv6 nd ra suppress
VLAN10
ipv6 access-list wsav6
permit tcp 2001:DB8:1:10::/64 any eq www
permit tcp 2001:DB8:1:10::/64 any eq 443
!
ip access-list extended wsav4
WCCP IPv6 & IPv4 – WSA Side of things….
In Dual-Stack Environments, two WCCP Service Groups are required.
WCCP IPv6 & IPv4 – WSA Side of things….
IPv6 Address of the Switch / Router
WCCP with L3 Switch – IPv4
Redirect - Verification
munlab-3560X#show ip wccp 91 detail
WCCP Client information:
WCCP Client ID: 172.16.10.100 Version & State
Protocol Version: 2.0
State: Usable
Redirection: L2
Redirect
Packet Return: L2
Packets Redirected: 0
Method
Connect Time: 01:02:16
Assignment: MASK Assignment
Method
Mask SrcAddr DstAddr SrcPort DstPort
---- ------- ------- ------- -------
0000: 0x00000000 0x00000526 0x0000 0x0000 Mask Value
Value SrcAddr DstAddr SrcPort DstPort CE-IP
----- ------- ------- ------- ------- -----
0000: 0x00000000 0x00000000 0x0000 0x0000 0xAC100A64 (172.16.10.100)
0001: 0x00000000 0x00000002 0x0000 0x0000 0xAC100A64 (172.16.10.100)
0002: 0x00000000 0x00000004 0x0000 0x0000 0xAC100A64 (172.16.10.100)
WCCP with L3 Switch – IPv6
Redirect - Verification
munlab-c6504#sh ipv6 wccp 90 det
WCCP Client information:
WCCP Client ID: 2001:420:44E6:2013::45
Protocol Version: 2.01 Version & State
State: Usable
Redirection: L2
Packet Return: L2 Redirect
Assignment: MASK Method
Connect Time: 00:13:25
Redirected Packets:
Process: 0 Assignment
CEF: 0 Method
GRE Bypassed Packets:
Process: 0
CEF: 0
Mask Allotment: 4 of 4 (100.00%)
Assigned masks/values: 1/4
Mask SrcAddr DstAddr SrcPort DstPort
---- ------- ------- ------- ------- Mask Value
0000: :: 300:: 0x0000 0x0000
WCCP with ASA  ASA allows only redirect-in
 Client and WSA must be on same interface - No DMZ Deployment
possible....
Internet  Inside ACL is checked before redirection
Destination Server must be allowed in ACL
 Redirection Method is GRE based
 Redirect ACL allows permit and deny
 No TCP Intercept, Inspect Engine or internal IPS is applied to the
redirected flow.
 IPS HW/SW Module however does inspect traffic
access-list WCCPRedirectionList extended deny ip 172.16.10.0

255.255.255.0 172.16.10.0 255.255.255.0
access-list WCCPRedirectionList extended permit tcp any any eq www
access-list WCCPRedirectionList extended permit tcp any any eq https
!
wccp 90 redirect-list WCCPRedirectionList
wccp interface INSIDE 90 redirect in
WCCP Dual-Stack with Router – ISRG2
Lab-Setup with ISR G2
ip wccp source-interface GigabitEthernet0
ip wccp 91 redirect-list IPv4-WCCP
Internet ipv6 unicast-routing
ipv6 cef
ipv6 wccp source-interface GigabitEthernet0
P2 ipv6 wccp 90 redirect-list IPv6-WCCP
!
P1 interface GigabitEthernet0
description WCCP-REDIR
Gi0 ip address 172.16.201.1 255.255.255.0
duplex auto
speed auto
ipv6 address FD00:ABCD:1:2::1/64
Fa0 ipv6 nd ra suppress all
!
WCCP Dual-Stack with Router – ISRG2 (2)
Lab-Setup with ISR G2
interface Vlan200
Internet
description WCCP Inside
ip address 172.16.200.1 255.255.255.0
P2 ipv6 address FE80::1 link-local
ipv6 address FD00:ABCD:1:1::1/64
P1
ipv6 nd prefix D00:ABCD:1:1::/64 no-advertise
!
Gi0 interface FastEthernet0
switchport mode trunk
no ip address
Fa0
WCCP with IP Spoofing
 Some Designs require that the Client IP is
e2 preserved after being proxied
 Problem to solve:
e1 Traffic coming back from the Internet needs to be
e0 redirected to the WSA by the network because the
Destination is now the Client Network, no longer the
WSA
 IP Spoofing mostly used in transparent mode
 Caution – adds complexity in troubleshooting
 Activated on the WSA in the WCCP Config:
IP Spoofing Design in Transparent Mode
ip cef
WCCP 92 ip wccp version 2
ip wccp 91 redirect-list Redirect-Client
e2 ip wccp 92 redirect-list Redirect-back
!
interface e0
e1 ip wccp 91 redirect in
e0 !
interface e2
WCCP 91 !
ip access-list extended Redirect-Client
permit tcp 145.16.0.0 0.0.255.255 eq www
permit tcp 145.16.0.0 0.0.255.255 eq 443
!
145.16.0.0 /16 ip access-list extended Redirect-back
permit tcp any eq www 145.16.0.0 0.0.255.255
permit tcp any eq www 145.16.0.0 0.0.255.255
IP Spoofing Design in Transparent Mode
WCCP 92
e2
e1
e0
WCCP 91
145.16.0.0 /16
Management Functions
Features Support for IPv6 Support over IPv6
WebUI (HTTP, HTTPS) Yes Yes
CLI (SSH) Yes Yes
FTP No No
Logging, Log Push Yes No
SNMP Yes No
Upgrades / Updates N/A No
Reporting, Tracking Yes N/A
Support Functions
Features Support for IPv6 Support over IPv6
Support Tunnel N/A No
Packet Capture Yes N/A
Policy Trace Yes N/A
WBNP, Telemetry Yes No
CWS Cloud Connector Mode No No
CLI
• Neighbor Cache in IPv6 is equivalent to the arp cache in IPv4
Display the arp-cache (v4)
Display the neighbor table (v6)
Troubleshooting:
HTTPS Issues
Flow for Decryption
Identity
Authentication
HTTP Proxy HTTPS Proxy
Access Pol Decryption Policy
Block Monitor Warn Pass Decrypt Drop Monitor

Cont. eval of
Block Cont eval Encrypted Decryption Policies
Warn Goto
Page of Access Page Page
Access If “Decrypt for EUN”
displayed Policies displayed displayed
Policy Selected (in 7.7)
Page Page Block Page
blocked allowed displayed
Flow for Decryption (2)
Access Pol Decryption Policy
Monitor Monitor
Applications WBRS Check : has Score
Granular Block Monitor Pass-thru Decrypt Block

Control (if
available)
Block Continue
page Eval of
displayed Access WBRS Check : has No Score
Policies
Default Action
Troubleshooting HTTPS Issues
Certificate Errors
Scenario: You enable HTTPS
decryption on WSA
• After, users complain of errors
• Complaints of “strange” issues on
sites
Time to Investigate
Compare Certificate Details – Without Proxy and With Proxy
Trusted
Trust full
certification path
Untrusted
WSA certificate not
trusted authority
Install Trusted Root Certificate
Install WSA certificate as trusted root
• From IE, view and install
• Push through GPO
Verify Certificate is Now Trusted
WSA Certificates
WSA rewrites SSL
certificates dynamically
WSA Cert is now
trusted root
Compare Certificate Details – With Decryption
Trusted
Trust full
certification path
With and Without
Decryption
HTTPS Troubleshooting – Access Logs
HTTPS Request Denied
• 407 – Proxy Authentication Required
#grep google.com accesslogs

1421790456.333 2 192.168.202.114 TCP_DENIED/407 0 CONNECT tunnel://www.google.com:443/
- NONE/- - OTHER-NONE-ContractorsAD-NONE-NONE-NONE-NONE
<IW_srch,5.8,1,"-",-,-,-,-,"-",-,-,-,"-",-,-,"-","-",-,-,IW_srch,-,"-","-","Google","Search
Engine","Encrypted","-",0.00,0,Local,"-","-",-,"-",-,-,"-","-”>
Identity Matched
ContractorsAD
identity triggers auth
HTTPS Troubleshooting – Access Logs
Request is Decrypted Allowed by Access Policy
• HTTPS working as expected • User unauthenticated, default policy
1421791245.780 215 192.168.202.114 TCP_CLIENT_REFRESH_MISS_SSL/200 39 CONNECT

tunnel://www.google.com:443/ "(Unauthenticated)192.168.202.114" DIRECT/www.google.com -
DECRYPT_WEBCAT_7-DefaultGroup-ADauth-NONE-NONE-NONE-DefaultGroup
<IW_srch,5.8,1,"-",-,-,-,-,"-",-,-,-,"-",-,-,"-","-",-,-,IW_srch,-,"-","-","Google","Search
Engine","Encrypted","-",1.45,0,Local,"-","-",-,"-",-,-,"-","-”>
1421791334.748 740 192.168.202.114 TCP_MISS_SSL/200 94933 GET

https://www.google.com:443/search?safe=active&site=&source=hp&q=cisco&oq=cisco
"(Unauthenticated)192.168.202.114" DIRECT/www.google.com text/html DEFAULT_CASE_12-
DefaultGroup-ADauth-NONE-NONE-NONE-DefaultGroup
<IW_srch,5.8,0,"-",0,0,0,1,"-",-,-,-,"-",1,-,"-","-",-,-,IW_srch,-,"Unknown","-","Google","Search
Engine","Unsafe Rewrite","ensrch",1026.30,0,Local,"Unknown","-",1,"-",-,-,"-","-”>
Certificates and HTTPS Proxy Settings
Signing Certificate
Self signed or
by trusted internal CA
Must be trusted by clients
Options
Additional decryption
Invalid certificates
Decryption Policies and Bypassing Decryption
Troubleshooting HTTPS Sites
• Web Applications break from HTTPS
decryption and MITM certificates
• Adding sites to Pass Through may be
required
• Pass Through can be temporary
troubleshooting step as well
Thirsty for more?
• BRKSEC-3006 (The Trip to TLS Land using the WSA)
• This Session will go into a Deep Dive on all things around TLS Encryption on the WSA.
We will first get into an overview on how TLS works, then check the process and
functionalities required to deal with TLS on the WSA. This will include best practice
recommendations. In the last Section we will discuss general issues with TLS 1.3 and
HTTP/2 and what is developing on a industry standard perspective and how things like
certificate pinning are influencing our deployment. This Session is targeted at Security
Administrators and Security Architects dealing with WSA or proxies/ngfw in general and
want to get more insight on TLS Decryption best practices.
• Wednesday, 7/13 @ 1:30p
• Toby Mayer, Consulting System Engineer
Performance Troubleshooting and
Optimization
Troubleshooting Performance Issues
Common Examples of Slowness Causes
Going to specific site is slow

Internet is slow for everybody
Going to a specific site is slow intermittently
Internet is slow for everybody intermittently
HTTPS sites are slow
Downloads are slow for a site
WSA Performance Analysis
Internet
WWW Server Cisco TALOS
DNS Server
Client AD Server
Troubleshooting Slowness – Gathering Information
Important questions to ask: Other information to gather:
• Is the issue occurring right now? • Proxy deployment (Transparent vs
Explicit)?
• Does the issue occur for all users?
• Does the issue occur for all websites? • Authentication being used (NTLM,
CDA/TUI, Kerberos, LDAP, No
• When did the issue first start to occur? authentication)?
• Does the issue occur all the time or only • HTTPS Proxy enabled?
during certain times?
• If you bypass the WSA, does the issue still
• Features enabled on the WSA (AVC, AV
occur? Engines, Adaptive Scanning, etc…)?
• Does the slowness occur for both HTTP and • Policy Construct – lots of Regex?
HTTPS sites?
• Identify Issues
• Easy to identify using CLI commands:
• proxystat : Easy to get CPU Utilization and
corresponding RPS – instantaneous
• status detail: Number of active connections &
response times. Active connections, not above 40k
• trackstats: Files stored with useful stats of WSA proxy
and other engines
• Typically prox CPU utilization above 80% will
start to impact performance (most noticeable
above 90%)
• Proxy Connection Backlog – Ideally 0 or near 0
• Response time – variable – BUT consistency is
key
• Packet Capture
• Need to get a capture that contains the client -> wsa
socket as well as the wsa -> destination server socket
• If not a lot of traffic, run the capture unfiltered
• If too much traffic, run the capture filtered based on
the client IP address and the destination server IP
address
• If possible, run a capture on the client machine as well
• When running a capture using a custom filter and the
traffic contains GRE, it will not capture the client side
traffic
• Using the predefined filter is better for this scenario
Packet Capture with IPv6
 Packet Capture shows
additional interfaces for IPv4
& IPv6
 Filter can be applied to IPv6
addresses
Troubleshooting Slowness – Common Scenarios
• Slowness occurs only during peak hours
• Typically a capacity issue on WSA
• Requests per second are too high for the
configuration, causing prox process to
spend a high percentage of time using CPU
Troubleshooting Slowness – Troubleshooting Process
If issue is occurring at the time of troubleshooting:
• See if it is possible to test by bypassing the WSA to see if the issue still
occurs. If the issue still occurs when bypassing the WSA, then in most
scenarios you can rule out the WSA being the cause.
• If going transparent and issue is occurring, try explicit to see if the issue still
occurs. Load balancer? DNS? WCCP Issue? PBR or L4 redirection issue?
• Check to see if there is a capacity issue by looking at the rate command or
by checking trackstats/shd_logs.
Troubleshooting Slowness – Troubleshooting Process
If issue is occurring at the time of troubleshooting:
• Run a packet capture on the WSA and a grep on the access logs while
reproducing the issue. If issue occurs for all websites, use a basic website
like example.com or purple.com.
• If the delay appears to be coming from the WSA, either by packet capture
verification or being unable to rule out the WSA as the source, check
trackstats to see if any features may be causing a delay.
• Check logs on the WSA to see if there are any warning/critical alerts that
stand out.
• If possible, set up an access log subscription with latency custom fields to
see where the delay is occurring.
Troubleshooting Performance
SNMP Monitoring
Enable by CLI command snmpconfig

• SNMPv1, v2 and v3 supported
• SNMPv3 username is v3get
MIBs available on cisco.com

• http://
www.cisco.com/c/en/us/support/security/web-secur
ity-appliance/tsd-products-support-series-home.ht
ml
• AsyncOS Web MIB

• AsyncOS SMI MIB for WSA
• AsyncOS Mail MIB for WSA (not a mistype!)
SNMP Monitoring
• OID’s that are the most useful

• Proxy CPU in Percent
• 1.3.6.1.4.1.15497.1.2.3.1.2 (cacheCPUUsage)
• Request Throughput
• 1.3.6.1.4.1.15497.1.2.3.7.1.1 (cacheThruputNow)
• Although both the OID’s have the keyword cache, it has nothing to do with the
WSA ‘cache,’ other than that the values are always stored in cache.
SNMP Monitoring
• SNMPwalk example (other options too!)
• snmpwalk -v3 -l authNoPriv -a MD5 -A <passphrase> -Os -u v3get <IP_of_WSA> <OID>
The WSA is showing a higher than expected CPU load either via
the CLI proxystat command or SNMP polls. As the WSA reaches
its maximum capacity, administrators may experience sluggish
response to the GUI and most notably latency will increase for
traffic being proxied through the appliance.
Policy Best Practices
Identity Usage
Keep Identity Usage To A Minimum
• The primary goal of identities is to specify which traffic should be authenticated
on the appliance (or to specify individual appliances when deploying with an
SMA). If creating policies for membership groups or specific subnets, it is best
to define these in the access policy itself – especially when the access policies
make use of the “All Identities” selection.
Access Policies
• Use “All Identities” Sparingly
• Whenever a policy specifies “All Identities” it is actually replicated in the policy flow for every defined
identity. For example, if there are 5 identities, what appears to be a single access policy would
actually equal 5 policies if “All Identities” is selected.
• Place Most Used / Least Complex policies near top of list

• Since the WSA stops executing policy decision after the first match, attempt to keep the most
generic and most used policies at the top of the access policy list.
• Place most taxing rules near end of list

• Keep the most granular/least used rules near the bottom of the access policy list. These items
include authentication policies or policies with custom categories or especially custom categories
that contain regular expressions.
• User Policies
• Avoid defining multiple policies specifying individual users. When possible, create a group for these
users on the authorization server and consolidate the policies into a single (or a simpler) policy set.
Custom Categories
• Avoid regular expressions, especially “.*”
• Having the ability to filter uri’s with regular expressions is a great advantage of the
product, but should be used judiciously in order not to adversely effect performance
• If given the choice to either stipulate one (or several) domains or create a regular
expression, choose the domain definition. Always avoid ending a regular expression
with a .* - generally this is not needed and has a very large impact on scalability.
• Keep custom categories to a minimum
• Utilize the predefined URL categories on the appliance
• Avoid creating a custom categories for URLs that are already properly categorized in the
filtering databases.
• Seeing lots of uncategorized sites? Turn on WBNP participation!!!!!
Debugging Performance Issues
• Download file “prox_track.log” from appliance via FTP
• File is written every 5 minutes with timestamp
• Setting can be changed in advancedproxyconfig in CLI
prox_track.log content
• Contains various statistical data around proxy performance
• Please do NOT consider all number of packets 100% accurate!
• Just gives a good hint what problem might be happening
General Statistics
 Traffic Statistics:
If you have numbers increasing on “throttled transactions” this could indicate that the
appliance can not handle the load
How to read Prox_track.log
 Statistics are snapshots of total number of Packets
Counters are reset after reboot / restart of proxy
 Take statistic from time X and time Y, then compare change:
Important Statistics
 Client time:  Hit time:
Total time that the client was waiting Time that the WSA is using to fetch
until his request was fulfilled content from the local cache
 Miss time:
Time that the WSA takes to fetch all
Data from the server
Important Statistics (2)
 Server Transaction time:  Server wait time:
Time for the total transaction to the Time until WSA gets the first byte
Server to be finished. from the Server
High Values can mean “upstream” problems
(firewall, router, ISP, upstream proxy)
 DNS Time:
Time for the WSA to do a DNS Resolution
High time does indicate a problem with the DNS Server
Shameless OpenDNS Plug
http://info.opendns.com/rs/opendns/images/FB-Why-Point-DNS-to-OpenDNS.pdf
https://www.opendns.com/enterprise-security/resources/solution-briefs/covering-your-dns-blind-spot/
 Auth Helper Wait:  Auth Helper Service:
Time to wait for an authentication Time until an authentication request
request until its validated from the AD is fully validated
/ LDAP
Check if IP address is already authenticated,
High time indicates a problem with the check surrogates, etc…
connection to the authentication Server
 WBRS Service Time:  AVC Header Scan Service Time:
Time for the WSA to check the Time to check the Header of a request
reputation score against the AVC Signatures
 Webcat Service time:  AVC Body Scan Service time:
Time for the WSA to check the URL Time to check the body of a request against
Category the AVC Signatures
 Sophos/McAfee/Webroot Service Time:  Adaptive Scanning Service Time:
Time that the Scanner used to scan the object Time for the adaptive scanning
process to scan an object:
 Service Queue Time:
Time that the object stayed in the queue to be
scanned
Adaptive Scanning
 Each type of object gets a RISK Score assigned
 Score is based on Type of object, effectiveness of malware scanner for this type and WBRS
(WBRS must be enabled on WSA)
 Appliance will scan objects with the Scanner that is most appropriate for this object type
 If appliance has a performance problem with the Anti Malware Scanners, it will drop objects
not to be scanned
Example: Don’t scan *.jpg files with McAfee when they are coming from Websites with a good reputation.
Customizing the Access Log
Add custom field like:

“%m” (=Authentication Method) to the
access_log
 Variables can be appended in the Access Logs

 Variables are to be found in the GUI, some older Versions of WSA
Software might not have the full list
Customizing the Access Log - Example
%m AUTH: %:>a DNS: %:>d REP: %:>r %m : Authentication Method
%:>a : Authentication Wait time
%:>d : DNS Wait time
Any Text acting as a comment for %:>r : Reputation Wait time
readability
Customizing the Access Log – Example(2)
Destination IP %k Extremely useful in Dual-Stack
Environments to find out whether WSA
makes the outgoing connection on IPv4 or
IPv6!
Destination IP = v4 Source IP from Client = IPv6
Customizing the Access Log – Example (3)
Other useful Parameters:
%L <- human readable local time

%k <- Destination IP
%g <- group memberships
%u <- User Agent
Example for detailed Performance logs:

Request Details: ID = %I, User Agent = %u, AD Group Memberships = ( %m ) %g ] [ Tx Wait Times (in
ms): 1st byte to server = %:<1, Request Header = %:<h, Request to Server = %:<b, 1st byte to client =
%:1>, Response Header = %:h>, Client Body = %:b> ] [ Rx Wait Times (in ms): 1st request byte =
%:1<, Request Header = %:h<, Client Body = %:b<, 1st response byte = %:>1, Response header =
%:>h, Server response = %:>b, Disk Cache = %:>c; Auth response = %:<a, Auth total = %:>a; DNS
response = %:<d, DNS total = %:>d, WBRS response = %:<r, WBRS total = %:>r, AVC response =
%:A>, AVC total = %:A<, DCA response = %:C>, DCA total = %:C<, McAfee response = %:m>, McAfee
total = %:m<, Sophos response = %:p>, Sophos total = %:p<, Webroot response = %:w>, Webroot total
= %:w<, Anti-Spyware response = %:<s, Anti-Spyware total = %:>s; Latency = %x; %L
Summary for WSA Performance Analysis
 WSA has very detailed logs/GUI to troubleshoot

performance issues
 Use prox_stat.log file for general performance checks
 Use customizing the Access Logs for detailed checking of
single requests
 Always able to add more processing power either with
hardware or virtual appliances
WSA Performance Analysis
Internet Cisco Talos

WWW Server
DNS Server
Client AD Server
Hardware / Virtual Appliance
Update & Sizing
Cisco Content Security Virtual Appliance
• Free of charge for customers with • Licensing handled via license file vs.
Security Bundle Licensing/Contract ID cloud keys as physical
• Offered for all Content Security • No perpetual licensing options, VM
Products – WSA, SMA, and ESA expires when security features expire
• KVM and ESXi support • Full appliance import – no underlying
OS requirements
• http://
www.cisco.com/c/dam/en/us/td/docs/security/content_security/virtual_appliances/
Cisco_Content_Security_Virtual_Appliance_Install_Guide.pdf
• http://
www.cisco.com/c/en/us/support/docs/security/email-security-virtual-appliance/118
301-technote-esa-00.html
WSAv Sizing & Performance
Processor
Model Disk Space Memory Sustained RPS*
Cores
S000v** 250 GB 4 GB 1 65-180
S100v 250 GB 6 GB 2 120-340
S300v 1024 GB 8 GB 4 170-480
Minimum specs:
Two 64-bit x86 processors of at least 1.5 GHz each, 8 GB of physical RAM, A 10k RPM SAS hard drive disk
*Sustained RPS variant on security features enabled, and policy complexity.

** Only consider the S000v for lab/very small environments
WSAv vs. WSA Appliance
Processor
Model Disk Space Memory Sustained RPS*
Cores
S000v 250 GB 4 GB 1 65-180
S100v 250 GB 6 GB 2 120-340
S300v 1024 GB 8 GB 4 170-480
S190 1200 – 2400 GB 8 GB 6, 1.9GHz 210-450
S390 2400 – 4000 GB 32 GB 8, 2.4GHz 650-1000
S690 4800 – 9600 GB 64 GB 24, 2.5GHz 1385-1800
*Sustained RPS variant on security features enabled, and policy complexity.
Virtual Installation Considerations
• Do not clone the VM, Do not alter the hardware configuration
• No snapshots older than 72 hours (save disk!)
• Local disk is recommended
• Thin provisioning sounds great, but don’t use outside of a lab.
• Running on ESXi v4? You’ll need to create a new datastore for <1 TB disk
space allocations (4 MB vs 8 MB block)
• Cisco UCS Hardware is supported end-to-end. Other hardware platforms are
supported on a “Best Effort” basis: Cisco will try to help you, but it may not be
possible to reproduce all problems, and we cannot guarantee a solution.
• Oh, and do not clone or alter the HW config on the VM  - and – LOCAL DISK!
Cognitive Threat Analytics & WSA
Talos CTA on Cisco Web Security (CWS / WSA)
Before
During
After
Web Web
Filtering Application
Reputation Visibility & Webpage Anti- File Outbreak Cognitive
Control Malware Dynamic File
www.website.co Reputation Intelligence Threat Analytics
m
Malware Retrospection
Analysis
Admin Management
Reporting
CTA
STIX / TAXII (APIs)
Partial
Allow Warn Block
Block
Cognitive Threat Analytics
Layered Processing Engine & Scalable Cloud Infrastructure
A ft e r
CTA
Layer 1
CTA Layer 2
CTA Layer 3
10B 20K
Anomaly Trust Event Relationship incidents
requests Entity modeling
detection modeling classification modeling per day
per day
Anomalous Malicious Threat

Web requests (flows) Events (flow sequences) Incidents (aggregated events)
Recall Precision
Configuration Notes
• CTA Analytics are run in the cloud
• Proxy logs are ingested on regular basis by setting up regular
proxy upload over HTTPS or SCP (the only thing required on
customer side)
• Data is generally ready for review in one week
Architecture of CWS and CTA
Proxy logs
CWS CTA
STYX/TAXII Export
CWS Portal
WSA Connector
ISR G2 Connector CTA
AnyConnect…..
Splunk, ArcSight,
Q1 Radar, Logpoint… SIEM
Architecture of CWS and CTA – Additional Inputs
Proxy logs
CWS CTA Service listening on ports 22
(SCP) and 443 (HTTPS) for
incoming data from WSA or 3rd
party proxies
VMs – scalability and reliability
STYX/TAXII Export
CWS Portal
WSA Connector
ISR G2 Connector CTA Cisco WSA
AnyConnect… Blue Coat ProxySG
CTA
Service is configured
(Standalone from the CTA interface
Splunk, ArcSight,
Q1 Radar, Logpoint… SIEM portal)
Adding and Managing WSA Device List in CTA UI
Selecting Upload Type
New Device Configuration
WSA Device Config 1
WSA Device Config 2
1. Setup parameters
according to the manual
2. Copy the key that you
get
3. Paste the key back into
the CTA portal
WSA Device Config 3
Adding and Managing WSA Device List in CTA UI
• Maximum file size is 1GB, recommend smaller uploads incase of failure
• Recommended upload frequency (10-60 minutes)
• Log upload happens from the WSA M1 interface so it may be necessary to allow
traffic from the management interface to the internet (or to the cloud service).
• Log upload activity is visible in the WSA system log & CTA Console
• Warning: When committing the configuration change, the WSA proxy process
restarts, so users connected via the proxy may be temporarily disconnected. If
WSAs are not operating in high-availability (HA) mode, we recommended you
configure the WSA during an off-hour maintenance window to avoid impacting
users during production hours.
Confirm Device Log Upload
CTA Reporting
CTA Reporting
Thirsty for more?
• BRKSEC-1002 – CTA: From Exploitation to Exfiltration
• This session is an introductory session that provides beginner level walk through
Cisco Cognitive Threat Analytics (CTA) and its detections. On this example driven
session, you will learn: 1. About malware life cycle - from exploit kits, through
infections, monetization, and data exfiltration 2. How CTA uses anomaly detection to
highlight such behavior in your network 3. How to perform risk assessment and
mitigation
• Thursday, July 14, 1:00p

• Michal Svoboda, Engineer
ThreatGRID & WSA
Threat Grid Integration on WSA
• Requirements
• Cisco Threat Grid v1.4.2 or newer
• Cisco Web Security Appliance AsyncOS v8.8 or newer
• Before You Begin

• Ensure code levels appropriate
• Ensure WSA can reach the CLEAN interface over network
• WSA requires feature keys for “File Reputation” and “File Anaysis”
• Configure Threat Grid appliance first, then WSA
• If you will deploy a self-signed certificate: Generate a self-signed SSL certificate from
the Cisco AMP Threat Grid appliance to be used on your WSA. Be sure to generate a
certificate that has the hostname of your AMP Threat Grid appliance as CN. The default
certificate from the AMP Threat Grid appliance does NOT work
Cisco Threat Grid & Web Security Appliance
Multiple Configuration Options
Threat Grid Integration – Step 1
• Click Regenerate
• Download SSL Certificate
• ThreatGRID Application
• “Clean Interface”
• Administration Portal
• “Admin Interface”
• Supports
• TLSv1.0, TLSv1.1, TLSv1.2
• Security Services / Anti-Malware
and Reputation
• Edit Global Settings
• Advanced
• Select Private Cloud
• Enter DNS of Threat Grid Server
• Select Use Uploaded CA

• Upload the .cert file downloaded from TG
• When the WSA connects and registers itself with the Threat Grid Appliance, a
new Threat Grid user is created automatically. The initial status of this account
"de-activated“
• Login will match Client ID from WSA
• Click Re-Activate User
Troubleshooting Integration
• Ensure you have appropriate Feature Keys on WSA
• Check if the port 443 communication to TG server (clean interface) over TCP is
healthy
• Check if there is any "API Key Error" printed in the AMP debugs logs
• Invalid API Key
• Check if the account is re-activated, if not – re-activate the account
• Account Inactive
• Check if the account is re-activated, if not re-activate the account
Supporting Documentation
• Cisco AMP Threat Grid Install Guides
• http://www.cisco.com/c/en/us/support/security/amp-threat-grid-appliances/products-installation-guides-list.html
• Connecting Cisco ESA/WSA Appliances to Threat Grid Appliances

• http://www.cisco.com/c/dam/en/us/td/docs/security/amp_threatgrid/connecting-esa-wsa-to-tga.pdf
• Cisco Web Security Appliance (WSA) User Guide

• http://www.cisco.com/c/en/us/support/security/web-security-appliance/products-user-guide-list.html
We made it….. Thank you!
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Security Joins the Customer Connection Program
Customer User Group Program
19,000+
Members
• Who can join: Cisco customers, service Strong
providers, solution partners and training partners
• Private online community to connect with
Join in World of Solutions
peers & Cisco’s Security product teams
Security zone  Customer Connection stand
• Monthly technical & roadmap briefings via
WebEx  Learn about CCP and Join
 New member thank-you gift*
• Opportunities to influence product direction  Customer Connection Member badge ribbon
• Local in-person meet ups starting Fall 2016

Join Online
• New member thank you gift & badge ribbon
*
www.cisco.com/go/ccp
when you join in the Cisco Security booth
Come to Security zone to get your new member gift*
• Other CCP tracks: Collaboration & Enterprise and ribbon
Networks
* While supplies last
Thank you
Security Cisco Education Offerings
Course Description Cisco Certification
CCIE Security Expert Level certification in Security, for comprehensive understanding of security CCIE® Security
architectures, technologies, controls, systems, and risks.
Implementing Cisco Edge Network Security Solutions Configure Cisco perimeter edge security solutions utilizing Cisco Switches, Cisco CCNP® Security
(SENSS) Routers, and Cisco Adaptive Security Appliance (ASA) Firewalls
Implementing Cisco Threat Control Solutions (SITCS) Deploy Cisco’s Next Generation Firewall (NGFW) as well as Web Security, Email
Security and Cloud Web Security
Implementing Cisco Secure Access Solutions (SISAS) Deploy Cisco’s Identity Services Engine and 802.1X secure network access
Implementing Cisco Secure Mobility Solutions Protect data traversing a public or shared infrastructure such as the Internet by
(SIMOS) implementing and maintaining Cisco VPN solutions
Implementing Cisco Network Security (IINS 3.0) Focuses on the design, implementation, and monitoring of a comprehensive CCNA® Security
security policy, using Cisco IOS security features
Securing Cisco Networks with Threat Detection and Designed for security analysts who work in a Security Operations Center, the Cisco Cybersecurity Specialist
Analysis (SCYBER) course covers essential areas of security operations competency, including event
monitoring, security event/alarm/traffic analysis (detection), and incident response
Network Security Product Training For official product training on Cisco’s latest security products, including Adaptive
Security Appliances, NGIPS, Advanced Malware Protection, Identity Services
Engine, Email and Web Security Appliances.
For more details, please visit: www.cisco.com/go/securitytraining or http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKSEC-3771 Yazji v6 Final v2 2

Uploaded by

Copyright:

Available Formats

BRKSEC-3771 Yazji v6 Final v2 2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKSEC-3771 Yazji v6 Final v2 2

Uploaded by

Copyright:

Available Formats

Advanced Web Security

Deployment & Troubleshooting with

Bill Yazji – Consulting Security Engineer

Web Security Appliance

Which Protocol should be prefered in

• CARP provides virtual IP

• Useful when no hardware load balancer

Redundancy Group for IPv4 & IPv6

Higher Value = Master

IPv6 Internet Web

 Hash Based Assignment

WCCP Input redirect

WCCP Output redirect

3. I see you WCCP Client

Buckets 86–128 Buckets 129–170

• The routers/switches and WCCP clients participating in a WCCP service

Use template “access”, “routing” or “dual-

 Redirect ACL only allows „permit“

 If 3560/3750 is stacked, configure

ipv6 wccp 91 redirect-list wsav6

ipv6 access-list wsav6

In Dual-Stack Environments, two WCCP Service Groups are required.

IPv6 Address of the Switch / Router

access-list WCCPRedirectionList extended deny ip 172.16.10.0

WebUI (HTTP, HTTPS) Yes Yes

CLI (SSH) Yes Yes

Logging, Log Push Yes No

Upgrades / Updates N/A No

Reporting, Tracking Yes N/A

Support Tunnel N/A No

Packet Capture Yes N/A

Policy Trace Yes N/A

WBNP, Telemetry Yes No

CWS Cloud Connector Mode No No

Display the neighbor table (v6)

HTTP Proxy HTTPS Proxy

Access Pol Decryption Policy

Block Monitor Warn Pass Decrypt Drop Monitor

Applications WBRS Check : has Score

Granular Block Monitor Pass-thru Decrypt Block

#grep google.com accesslogs

1421791245.780 215 192.168.202.114 TCP_CLIENT_REFRESH_MISS_SSL/200 39 CONNECT

1421791334.748 740 192.168.202.114 TCP_MISS_SSL/200 94933 GET

Going to specific site is slow

Enable by CLI command snmpconfig

MIBs available on cisco.com

• AsyncOS Web MIB

• OID’s that are the most useful

• Place Most Used / Least Complex policies near top of list

• Place most taxing rules near end of list

Add custom field like:

 Variables can be appended in the Access Logs

Destination IP = v4 Source IP from Client = IPv6

%L <- human readable local time

Example for detailed Performance logs:

 WSA has very detailed logs/GUI to troubleshoot

Internet Cisco Talos

S100v 250 GB 6 GB 2 120-340

S300v 1024 GB 8 GB 4 170-480