Day 1 - MSP Bootcamp Training 201

Welcome
MSP Bootcamp Training 201

By: Davis Altamirano
Class Starts at 8:00 am PDT / 9:00 CDT / 11:00 am EDT
©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION

Zscaler™, Zscaler Internet Access™, Zscaler Private Access™, ZIA™ and ZPA™ are either (i) registered trademarks or service marks or (ii) trademarks
or service marks of Zscaler, Inc. in the United States and/or other countries. Any other trademarks are the property of their respective owners.
Prerequisite for Bootcamp
One week technical course on core functions of Zscaler for new hires
• Completed the 101 training

• ZCCA and ZCCP
• Installed required tools to complete exercises
• Wireshark
• HAR file reader e.g. Fiddler
2 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
What is Bootcamp Training About?
Interactive technical course on core functions for new hires
• Goal is to extend your technical knowledge on Zscaler in live instructor-led sessions by

SME’s (Subject Matter Experts) within the GCS organization.
Agenda
Five day dive into ZIA
• ZIA (Zscaler Internet Access)

• Day 1
• Intro/Zscaler Core Function Overview
• Architecture Overview
• Traffic Forwarding (GRE/PAC)
• Day 2
• Traffic Forwarding cont. (IPSEC/ZCC)
• Day 3
• Authentication (ZCC/IDP/SAML/LDAP/COOKIES)
• Virtual Machines Overview (VZEN/NSS/EDM/ZAB)
• Day 4
• Admin Portal (Administration/Policy/Analytics, etc..,)
• Mobile Admin Portal
• Threat Prevention
• Enforce policy
• Day 5
• Troubleshooting 101/Tools
• Hands-on
Intro/Zscaler Core Function Overview
What services do we offer as a company?
Zscaler Core Functions
• We act as a secure web gateway (proxy) for customers’ outbound internet traffic
• Type of services we offer
• ZIA - Zscaler Internet Access
• ADP, AV, Cloud, FW, Cloud IPS, Sandbox, BW Control, Cloud, Browser Isolation, CASB, DLP
• ZPA - Zscaler Private Access
• Alternate for VPN, Browser Access
• ZDX - Zscaler Digital Experience
• Monitoring Tool
• ZCP - Zscaler Cloud Protection
• Zscaler Workload Segmentation, Cloud Connectors, Security Posture Management
Where are we in the industry?
Gartner Quadrant (We are the best!)
• Zscaler continues to define the Secure Web

Gateways (SWG) category, enabling secure
digital transformation for thousands of global
customers with an industry-leading Zero Trust
architecture. We’ve been named the only
Leader in the Magic Quadrant for Secure Web
Gateways, further extending 10 years of
Magic Quadrant leadership.
ZS Evolution
2019
2018 2020
2017
2016
2015
2014
2013
2012
2011
2010
8 ©2019 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION Securing your cloud transformation
ZIA Architecture Review
Module 1 Objectives
By the end of the module, you will be able to describe all of the
Zscaler cloud components, functions, and architecture.
● Explain SMCA and its role and functionalities
● Describe the different clouds that Zscaler customers can be provisioned in
● Explain the use case of Feed Central Cloud(FCC).
● Explain how traffic is forwarded to Zscaler
● Explain how Zscaler processes customer transactions
● Explain how to log customer transactions
● Define the components of a Zscaler Datacenter
● Describe the Zscaler Load Balancers and explain the different types of
load balancing algorithms.
● Explain Zscaler Cloud Firewall.
Major components of ZIA
What powers the cloud
• SMCA : Central Authority

• Acts as the brain of the cloud.
• Central database holding all the sensitive information about the organization and their policies.
• Monitors all the components in the cloud and much more.
• SMUI : Administrator User Interface or Policy Portal
• The UI where the admin creates security policies.
• ZEN : Zscaler Enforcement Nodes
• High performance, multi-instance web gateway that takes traffic from the customers.
• Downloads the config from CA and applies the policies on the traffic.
• Cloud Routers/Relay
• Job is to route the transaction logs to appropriate Nanolog servers.
• SMTP
• Send DLP violation emails, Alerts
Major components of ZIA cont.
What powers the cloud
• SMSM or Nanolog Servers:
• Servers that store the transaction logs received from the ZEN’s.
• Writes the data in compressed id form.
• High speed data read and writing.
• PAC servers:
• Delivers the PAC file contents.
• Serves PAC file over http and https.
ZIA Architecture Overview(Per cloud view)
SMCA
• SMCA being the brain behind the entire operation is the most critical piece in this entire
puzzle.
• Responsibilities of SMCA
• Stores the entire database of the cloud and each organization.
• Takes care of user authentication/provisioning.
• Monitors all nodes in cloud.
• Keeps an up-to-date view of entire cloud. Removes systems/VIP’s which are found unhealthy.
SMCA
• User/Company config management.

• All cloud config related to users/company level config is stored in CA Database.
• When a user sends traffic to a SME, SME identifies the user/company/location and downloads
the necessary information from SMCA.
• SME connects to SMCA over two channels.
• Primary Channel 🡪 Time sensitive messages like health messages, Location/User/Company config
fetch etc. flow through this channel.
• Secondary Channel 🡪 Larger messages like MIP(Surrogacy) table sync, time quota sync messages etc
flow through this.
• Once a request comes on SMCA, SMCA keeps a track of all the SME’s having the company’s
config downloaded.
• When an Admin makes any changes in UI, the changes are committed to the CA DB without any
queueing. Upon activation in the UI, these changes are now updated to all the SME’s identified
in the above step.
SMCA
• Taking care of user authentication/provisioning.

• For any user to pass traffic through the cloud, its necessary that the user is first provisioned on
Zscaler.
• We support many types of user provisioning on Zscaler.
• Hosted DB
• AD LDAP
• OpenLDAP
• SAML Auto Provisioning
• SCIM Provisioning
• Apart from taking care of provisioning, SMCA also takes of user authentication.
• For Hosted DB, the credentials are locally stored and validated on CA.
• For LDAP, SMCA initiates the user bind request to the AD server.
• For SAML, SMCA validates the SAML response.
• For Kerberos, KDC role in SMCA validates the cross-realm request.
• SMCA also acts as IdP for the case where Zscaler is an IdP for certain applications.
SMCA
Monitor all nodes in cloud

• Monitor all nodes in cloud: This is a very important role taken care of by SMCA.
• SMCA monitors all the components in the cloud and keeps an up-to-date view of state of every system.
• SMCA monitors the nodes in two ways:

• Passive Health Monitoring: Each system updates its health stats to the SMCA every second.
• Active Health Monitoring: When Passive Health monitor fails, SMCA triggers Active health monitoring by initiating
an inbound connection to the node. When Active Health monitoring fails, SMCA raises a Alarm and continues to
monitor the node passively. AHM is done for select node types only and not for every node type. Fox ex:
Vzen/NSS/ZAB/SMSM nodes etc do not have AHM.
• What happens if a LB VIP/system is marked unhealthy.

• When a LB VIP is marked unhealthy, then SMCA removes that from rotation and that VIP is not provided to
users in any PAC file requests.
SMCA
Taking part in Master CA election process

• Now that you know how important is SMCA to the entire operation, its very essential that SMCA is available all the time.
• For this purpose, we always have a cluster of 5 or more SMCA nodes, participating in HA.
• Master CA has very specific roles:

• Has write access to the database
• All other CA’s will keep a local copy of database.
SME+ZVPN+SVPN+SMLB= A Typical Zscaler DC
Types of Zscaler Nodes handling customer traffic
• Types of nodes that handle customer traffic.
• SME(Zen nodes) 🡪 These are called ZEN(Zscaler Enforcement Nodes). They handle customer
traffic and apply configured policies.
• ZVPN 🡪 These are IPSEC terminating nodes. They terminate both IKEv1 and IKEv2 traffic.
• SVPN 🡪 These are SSLVPN terminating nodes. They terminate ZCC Tunnel2.0 traffic.
• SMLB 🡪 All of the above said nodes sit behind a LB node. We can have up to 128 instances of
same type behind a SMLB node.
Types of Zscaler Nodes handling customer traffic
• Before we dig deep into the types of nodes, lets be clear on following terms:
• Service IP & Service Interface
• Management IP & Management Interface
• Management interface is handled by the kernel directly, so it's handled by ZscalerOS.
• Our driver software is written such a way that, our OS(ZscalerOS) does not get a copy of the packets coming on the service interface. Instead,
the process(SME process) directly handles the traffic.
• We run our own TCP/IP stack in user space, which we call as SMNET.
SME/ZEN nodes
• Enforces customer policies.
• After the traffic is processed by SME nodes, they are sent out to the internet to the actual
destination.
• Single SME node can be part of multiple SMLB clusters.

• For example, say you have 3 GRE VIP in Chennai DC.
• A SME can listen for traffic destined to any of the 3 VIP’s.
• Each ZEN node can have anywhere between 3-6 SME instances.
SME/ZEN nodes
• Each SME is typically assigned one 1G interface. As we are scaling we are having SME’s with 10G nic as well.
• This interface is used for both sending and receiving data by the SME.
• All Zscaler instances follow the same model.
ZVPN nodes
• Terminates both IKEv1 and IKEv2 traffic.
• ZVPN instances can come in two flavors:

• One where the interface is bridged.
• smnet based IPSEC instances.
• SA information is not synced between the ZVPN instances.

• If your IPSEC traffic terminates on instance A, and if the instance goes down, then the traffic will
be sent to a new ZVPN instance by the Load Balancer.
• Since, SA information is not synced, you will have to establish a new tunnel with this newly
allocated instance.
SVPN nodes
• Terminates SSLVPN tunnels from ZCC client running tunnel2.0.
• The tunnel information is not synced between the SVPN instances.

• If your SSLVPN traffic terminates on instance A, and if the instance goes down, then the traffic
will be sent to a new SSLVPN instance by the Load Balancer.
• Since, tunnel information is not synced, you will have to establish a new tunnel with this newly
allocated instance.
SMLB nodes
• SMLB nodes do not terminate any traffic.

• They simply redirect the traffic to a SME/ZVPN/SVPN node from the cluster.
• We employ DSR(Direct Server Returns) form of load balancing. See below diagram.
• Is tasked with ensuring that the unresponsive SME instances within a DC do not get any traffic
SMLB nodes
• SMLB nodes are provisioned in either of:

• Active-Passive(Master-Backup)
• Active-Active with single cluster
• Active-Active with multiple cluster
• In Master-Backup setup, we use CARP(Common Address Redundancy Protocol) to achieve VIP failover.
• Although we are migrating to Active-Active LB setup slowly, we still use Active-Passive LB setup in following
deployments:
• Vzen
• Pzen
• Smaller DC’s where we do not expect capacity to hit high numbers.
SMLB nodes
• Active-Passive Sample output:
• The output will show:

• SMLB current state
• Cluster View as per the LB. It will show all the member nodes in the cluster.
• It will show health of each instance that LB has in its cluster. States will be Up/Down.
• Service IP of each instance that the LB monitors.
SMLB nodes(Active-Active in one cluster)
• Router uses ECMP based on source IP and destination IP to decide the next hop
• AA LB in one cluster means that all the SME and SMLB instances are in one broadcast domain.
• All the LB instances will be active for the given VIP.
Active-Active multi cluster
• In multi cluster setup, multiple clusters spread across multiple networks, share
same VIP IP address.
• As shown in last diagram, internet router leverages ECMP to decide the next hop
and sends the traffic to LB instance. Now depending on which cluster you land on,
your egress IP will vary.
• Since internet routers always use outer source IP + outer Destination IP load
balancing, all the traffic from one public IP always land on one LB.
• In the output below, you can see that the VIP is in 104.129.194.32/24 subnet,
while SME instances are in 136.226.48.0/24 subnet.
SMLB nodes
• We support the following hashing algorithms:

• Source IP (h1)
• Source IP + Source port. (h2)
• Source IP + Destination IP ---------- > Default hashing algorithm.(h3)
• Source IP + Source port + Destination IP + Destination port. (h4)
• Source IP + Destination IP + Destination port. (h5)
Packet Flow Within An SME.
About Policy Enforcement
• About Policy Enforcement:

• ZIA Public Service Edges feature Single-Scan, Multi-Action (SSMA) technology. SSMA handles the
inspection and then the execution of the policy takes place in the ZIA Public Service Edge's web and
firewall modules.
• Once the SSMA inspection process is complete, the ZIA Public Service Edge executes policies with
specific precedence. Each ZIA Public Service Edge has two main modules for applying policies:
• A web module &
• A firewall module.
• At a high-level, this is how traffic flows through the modules:

• Outbound web traffic: When the ZIA Public Service Edge receives outbound web traffic from your
organization to the internet it sends the traffic to its web module for policy evaluation. If the traffic violates a
web policy, it blocks the transaction. If the traffic does not violate any web policies, it sends the traffic to the
firewall module for policy evaluation. In the firewall module, if the traffic violates a firewall policy, it blocks
the transaction. If the traffic does not violate any firewall policies, it allows the traffic to the internet.
• Outbound non-web traffic: When the ZIA Public Service Edge receives outbound non-web traffic going to
ports other than 80/443 (or other HTTP/S ports) it sends the traffic directly to the firewall module for policy
evaluation. If the traffic violates a firewall policy, it blocks the transaction. If the traffic does not violate any
firewall policies, it allows the traffic to the internet.
• Inbound web traffic: When the ZIA Public Service Edge receives inbound web traffic (HTTP/HTTPS traffic for
ports 80/443) from the internet in response to HTTP GET/POST requests it sends the traffic to its web module
for policy evaluation. If the traffic violates a web policy, it blocks the transaction. If the traffic does not
violate any web policies, it allows the traffic into your organization.
About Zscaler Firewall
• There are two licenses for Firewall module.

• Standard
• Advanced
• Standard license allows the following:

• Create and apply FW policies based on Layer3, Layer 4 information.
• Create FQDN based allow or block policies.
• Advanced license allows the following:

• Create policies to allow or block based on network application.
• DNS control which allows allow/block on DNS queries.
• Cloud IPS(Intrusion prevention) which allows applying signature-based block on all ports and protocols.
• Destination NAT.
Detailed Next Gen Firewall Packet Flow (Client)
Break
ZIA Traffic Forwarding
PAC/GRE/IPSEC/ZCC
Module 2 Objectives
By the end of the module, you will be able to explain the different ways of
forwarding traffic to Zscaler.
● Define PAC files and PAC servers.
● Describe the use cases of all Zscaler specific PAC variables.
● Explain how to forward traffic to Zscaler while bypassing certain domains.
● Explain the concepts of subcloud.
● Explain GRE(Generic Routing Protocol).
● Explain IPSec as a traffic forwarding.
● Identify the differences between IKEv1 and IKEv2.
● Setup a IKEv1 tunnel to Zscaler.
● Explain how the packet flow changes, depending on the traffic forwarding method.
● Describe the concepts and functions of IKEv2-ALG.
● Describe the internal working of Zscaler Client Connector(ZCC).
● Deploy ZCC in various forwarding methods.
● Explain the Zscaler Mobile portal.
Traffic Forwarding - PAC Files
PAC file
What is a PAC file
A Proxy Auto-Configuration (PAC) file is a JavaScript function that determines whether web browser
requests (HTTP, HTTPS, and FTP) go directly to the destination or are forwarded to a web proxy server.
The JavaScript function contained in the PAC file defines the function:
function FindProxyForURL(url, host) { // ... }
Return value format

The JavaScript function returns a single string
• If the string is null, no proxies should be used
• The string can contain any number of the following building blocks, separated by a semicolon:
• DIRECT 🡪 Connections should be made directly, without any proxies
• PROXY host:port 🡪 The specified proxy should be used
Example: return "PROXY maa2.sme.zscaler.net:12321; PROXY bom6.sme.zscaler.net:12321; DIRECT";
PAC Files
Traffic Flow
HTTP CONNECT TUNNEL A HTTP CONNECT is sent by client.
The CONNECT shows client's intent to connect to
Traffic Flow through the eyes of Wireshark www.google.com on port 443.
Destination IP would be proxy IP
Proxy responds with 200OK.

This response means proxy is ready to accept data
for the destination www.google.com on port 443.
Client now sends data. In case of SSL, this would be Client Hello.
PAC file bypassing
Force some traffic to go out direct bypassing the proxy. Samples below
• Adding exception for a FQDN (Fully Qualified Domain Name):

if (dnsDomainIs(host, "www.google.com"))
return "DIRECT";
• Adding exception for a server IP address:

if (isInNet(dnsResolve(host), "1.2.3.4", "255.255.255.255"))
return "DIRECT";
• Adding exception for a particular source private IP

if (isInNet(myIpAddress(), "192.168.1.106", "255.255.255.255"))
return "DIRECT";
• Adding exception for a destination IP subnet

if (isInNet(dnsResolve(host), "1.2.3.4", "255.255.255.0"))
return "DIRECT";
• PAC file writing best practices: https://help.zscaler.com/zia/best-practices-writing-pac-files
Zscaler PAC variables
Gateway Variable
• Gateway Variable:
• You can use the ${GATEWAY} and ${SECONDARY_GATEWAY} variables to determine the ZIA Public Service
Edge closest to the client.
return "PROXY ${GATEWAY}:80; PROXY ${SECONDARY_GATEWAY}:80; DIRECT";
• The Zscaler service uses its geo-location technology to find the closest ZIA Public Service Edge with the
quickest response time. These variables provide the optimal user experience.
• If the organization has a large number of users behind a single egress IP address, then all the traffic lands on the
same SME. This can cause issues like overloading a particular SME.
Gateway Index Token
• Zscaler recommends this method if you want the users to be distributed across multiple SME's IP
addresses in a datacenter, for load balancing purposes.
• For example, ${GATEWAY_F0} corresponds to the first healthy IP address that is available,
${GATEWAY_F1} corresponds to the second healthy IP address and so on. We support from F0 till
F7.
• Use the following syntax:

return "PROXY ${GATEWAY_F0}:80; PROXY ${SECONDARY_GATEWAY_F0}:80; DIRECT";
• If the data center has less than eight healthy gateway IP addresses, then the PAC server allocates the
available healthy VIPs to all the eight variables in a round-robin fashion.
Dynamic Gateway Tokens
• Use the suffix, _FX to the ${GATEWAY} variable in the PAC file for the PAC server to dynamically
issue gateway IP addresses based on the client fingerprints, i.e., all users coming from a single egress
IP address are given an IP address from a pool of healthy gateway IP addresses.
• Use the following syntax:

return "PROXY ${GATEWAY_FX}:80; PROXY ${SECONDARY_GATEWAY_FX}:80; DIRECT";
• This variable is effective only for Zscaler Client Connector (formerly Zscaler App or Z App)
Source IP Variable
• You can use the ${SRCIP} variable to determine the client's public IP address
• Example:
var egressip = "${SRCIP}";
if (shExpMatch(egressip,"203.0.113.10")) {
/* User is in the office */
return "PROXY 10.84.0.188:80;DIRECT";
}
Country Gateway Variable
• You can use the ${COUNTRY_GATEWAY} and ${COUNTRY_SECONDARY_GATEWAY} variables
to determine the closest ZIA Public Service Edge in the client's country.
return "PROXY ${COUNTRY_GATEWAY}:80; PROXY ${COUNTRY_SECONDARY_GATEWAY}:80";
• When there is no ZIA Public Service Edge configured in your country, the ${COUNTRY_GATEWAY}
variable behaves just like the ${GATEWAY} variable.
• If your organization has a large number of users behind a single egress IP address, you can configure
the PAC file to choose from multiple gateway IP addresses within a country to distribute load.
return "PROXY ${COUNTRY_GATEWAY_F0}:80; PROXY ${COUNTRY_SECONDARY_GATEWAY_F0}:80; DIRECT";
return "PROXY ${COUNTRY_GATEWAY_FX}:80; PROXY ${COUNTRY_SECONDARY_GATEWAY_FX}:80; DIRECT";

Country Variable
• You can use the ${COUNTRY} variable to determine the client's country as shown in the following sample.
• The ${COUNTRY} variable supports the countries listed on Maxmind.
• You can also set exceptions for specific URLs when accessed from a specific country using the ${COUNTRY} variable
as shown in the following sample:
PAC Files
Subcloud
• Why subcloud:
• Customers want users to only go through a small subset of Zscaler nodes.
• Customers want to use PZEN’s along with public ZEN’s.
• Customers want to use DC’s that are not in public rotation. (Regional Surcharge nodes).
• In such cases, customers will ask Zscaler to provision a subcloud for them.
• Customers will then use return variables that is specifically created for subclouds.
Zscaler PAC variables(For subcloud)
• Gateway Variable
• ${GATEWAY.<Subcloud>.<Zscaler Cloud>}
• ${SECONDARY.GATEWAY.<Subcloud>.<Zscaler Cloud>}
• Gateway Host Variable(For Kerberos PAC)

• ${GATEWAY.<Subcloud>.<Zscaler Cloud>_HOST}
• ${SECONDARY.GATEWAY.<Subcloud>.<Zscaler Cloud>_HOST}
• FX Variable for subcloud:

• ${GATEWAY.<Subcloud>.<Zscaler cloud>.net_FX}
• ${SECONDARY.GATEWAY.<Subcloud>.<Zscaler cloud>.net_FX}
• Country Variable for subcloud:

• ${COUNTRY_GATEWAY.<Subcloud>.<Zscaler cloud>.net}:80
• ${COUNTRY_SECONDARY.GATEWAY.<Subcloud>.<Zscaler cloud>.net}:80
PAC Files
Public API’s
• http://pac.cloudname.net/getGatewayEndpoints?srcip=115.112.60.134
• pac.<cloudname>/getGatewayEndpoints?subcloud=<Subcloudname>"
• pac.<cloudname>/getVpnEndpoints?subcloud=<Subcloudname>"
• pac.<cloudname>/getGatewayEndpoints?lat=-<lat>&long=<long>
• pac.zscaler.net/getGatewayEndpoints?lat=-11&long=170“
• pac.zscaler.net/getGatewayEndpoints?subcloud=<subcloud>&srcip=2.2.2.2
Lab Exercise #1
Issue: Traffic is going to the wrong Zscaler Datacenter
• Problem Scenario:
a. Customer is using PAC files to route their internet traffic to Zscaler using the ${GATEWAY}
variable. Their office is physically located in San Jose, CA but their traffic is being routed to
Zscaler’s Dallas, TX datacenter. The customer expectation is that their traffic be routed to our
San Jose datacenter for best performance.
b. PAC file -
• Task:
a. Analyze the pac file and determine where the problem might be.
b. From your analysis, how would you communicate the results to the customer?
c. What would you advise the customer as next steps?
Tip - Check if customer has support access already open, you can check many things before asking questions or gathering data from them.
Lab Exercise #2
Issue: cnn.com not being bypassed using an exception
a.Customer is trying to bypass cnn.com using a wildcard, for all users who have private IP in range
172.16.0.0/12.
• Task:
a.The objective is to write a PAC file which helps achieve this objective.
b.Achieve this objective using shExpMatch
c.Repeat the same using dnsDomainIs
Traffic Forwarding - GRE
What is a tunnel in networking terms
• In the physical world, tunneling is a way to cross terrain or boundaries that could not normally be
crossed.
• Similarly, in networking, tunnels are a method for transporting data across a network using some
protocols.
• Example:
• Suppose you have an office location in Bangalore, India and another in San Jose, USA.
• Say you have an internal webserver hosted in Bangalore, that employees working in San Jose want to access.
• Since private IP's can’t communicate over the internet directly, we use tunneling protocols like GRE/IPSEC.
• So, your tunneling protocol like GRE/IPSEC would communicate over a public network, but inside it would carry
data, which preserves the internal IP addresses.
• Tunneling works by encapsulating packets: wrapping packets inside of other packets.
Through the eyes of Wireshark
Public IP to Public IP communication over the internet
Tunneling internal IP traffic. Here you can see 172.30.39.175 is communicating to 197.90.201.10.
This traffic is received by 197.98.201.9(Zscaler DC), over the internet.
Tunneling and Zscaler
• Zscaler supports 3 tunneling protocols.

• GRE Tunnels
• IPSEC Tunnels
• SSLVPN tunnels 🡪 Used by Zscaler Client Connector.
• In case of Zscaler, we are internet facing proxy. Which means we only receive internet bound
traffic.
• Customer should not traffic which is destined to a private IP through any tunnels established to Zscaler.
• Customer is recommended to send all internet bound traffic to Zscaler via these tunnels.
• When customers send traffic via a tunnel, we get to know the internal IP(Client Private IP). This helps us
better log the traffic.
GRE Tunnels
What is GRE
• The GRE (Generic Routing Encapsulation) protocol will encapsulate the actual traffic as payload
and utilizes the transport protocol like IP to forward the data to the destination
GRE Tunnels
What is GRE
• A GRE tunnel, just encapsulates data. It does not encrypt the data between endpoints.
GRE Tunnels
GRE Keep-Alive
• GRE tunnels typically use keepalive packets to determine if a tunnel is up. The GRE tunnel source
creates the keepalive request and response packets that are encapsulated and sent together to
the tunnel destination. When the tunnel destination receives an encapsulated packet, it just
decapsulates the original packet and sends the inner response packet back to the originating
peer.
GRE Tunnels
GRE Keep-Alives
You can see here how a GRE Keep-Alive looks. Source 103.220.234.45 is sending a GRE packet to 165.225.124.32.
Inside you will see the response packet which the client sends.
Source for that packet is 165.225.124.32 and destination is client(103.220.234.45).
GRE Tunnels
GRE Keep-Alives
You can see the keep-alive response here. Zscaler would just open the GRE headers and see that the
packet is destined to 103.220.214.45.
At this point, Zscaler would just send that packet back.
This completes one iteration of GRE Keep-Alive.
GRE Provisioning:
How do we provision tunnels.
• Customer must own at least one public IP.
• We attach that public IP to customers organization.
• We will provision a primary & secondary DC per public IP of the customer.
• Customer now must create a location on the portal, attaching the public IP to the location.
• Starting 6.1r, customers can self provision the public IP and attach GRE tunnels in Admin UI.
• To provision tunnels to RS(Regional Surcharge) DC’s, the provisioning is done by Zscaler

support.
Deploying GRE Tunnels
• Zscaler recommends that customers configure two GRE tunnels from an internal router
behind the firewall to the ZIA Public Service Edges.
• Zscaler requires building primary and backup GRE tunnels from every Internet egress
location and, if applicable, from each Internet service provider.
• While GRE Self Provisioning via Admin UI, Zscaler identifies the endpoints of the tunnels
based on your geolocation information. Customers can request alternative locations for the
tunnels by contacting Zscaler Support.
• Zscaler recommends that customers calculate the maximum transmission unit (MTU) and
maximum segment size (MSS) values on the GRE tunnel based on the MTU and MSS
configuration of the WAN interface.
• An incorrectly configured MTU results in higher fragmentation, leading to performance

degradation.
MTU Calculation
• Example calculation for deciding GRE tunnel MTU/MSS.
• In this example, the tunnel MTU and MSS values are calculated for a WAN interface that is 1500
bytes.
WAN Interface MTU = 1500
WAN Interface MSS = MTU (1500) – IP (20) – TCP (20) = 1460 (40 bytes TCP+IP Header)
GRE = 4 bytes header
GRE MTU = MTU (1500) – IP (20) – GRE (4) = 1476
GRE MSS = GRE MTU (1476) – IP (20) – TCP (20) = 1436
Supported Bandwidth for GRE Tunnels
• Zscaler supports a maximum bandwidth of 1 Gbps for each GRE tunnel if its internal IP
addresses aren't behind NAT.
• If the internal subnet is behind NAT, Zscaler can only support up to 250 Mbps of traffic for
each tunnel.
• If any organization wants to forward more than 1 Gbps of traffic, Zscaler recommends
configuring more GRE tunnels with different public source IP addresses.
• For example, if the organization forwards 2 Gbps of traffic, they can configure two primary GRE
tunnels and two backup GRE tunnels.
• If the organization forwards 3 Gbps of traffic, they can configure three primary GRE tunnels and
three backup GRE tunnels.
Monitoring GRE Tunnels
• Zscaler does not monitor customer GRE tunnels.
• Zscaler requires customers to monitor their GRE tunnels so that failover between the
primary and backup tunnels will trigger if a tunnel goes down.
• Customer must at least enable GRE keepalives to serve as a basic detection mechanism.
• The GRE keepalives monitor the interface, but not the service beyond the interface.
• To perform service monitoring, deploy Layer 7 health checks like IPSLA.

• Customers must perform HTTP Raw Request to the following URL:
• http://gateway.<Zscaler Cloud>.net/vpntest
• Zscaler requires customers to connect tunnel monitoring to tunnel failover.

• When service monitoring is down, the primary tunnel should failover to the backup tunnel, and when
monitoring is available, switch back to the primary tunnel.
GRE Tunnels
How the tunnel configuration looks like
• The customer must register the public static IP address from which they are building the GRE tunnel.
• As of 6.1r, customers can self provision their public IP onto Zscaler.
• If the DC’s listed in GRE Self-Provisioning utility is not as per expectation, then customer can open a
support case with Zscaler for the same.
• Zscaler provisioning team will add the IP address to the organization and configure GRE tunnel on
Zscaler side.
• Sample configuration:
GRE Router Configuration
Example of a Cisco 881
ip route <Primary DC VIP> 255.255.255.255 <Default GW>

ip route <Secondary DC VIP> 255.255.255.255 <Default GW>
!
interface Tunnel2700
description "Zscaler Primary Tunnel"
ip address <Primary Internal Router IP> 255.255.255.252
ip tcp adjust-mss 1436
keepalive 10 3
tunnel source <Tunnel Source IP>
tunnel destination <Primary DC VIP>
!
interface Tunnel2800
description "Zscaler Backup Tunnel"
ip address <Secondary Internal Router IP> 255.255.255.252
ip tcp adjust-mss 1436
keepalive 10 3
tunnel source <Tunnel Source IP>
tunnel destination <Secondary DC VIP>
72 !
©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
GRE Troubleshooting
Most common issues and what to look for
• Type of issues we hear from our customers

• Performance issues
• Tunnels are not coming up
• Tunnels are flapping

• Please remember that GRE tunnel on Zscaler is always up. It’s the customers monitoring that brings down the
tunnel.
• To debug, please check the following:
• CPU and general health of customer router.
• MTR towards Zscaler VIP IP during the issue time.
• Provide IP-SLA logs mentioning the exact time when the issue happened, along with TZ the logs are in.
• If possible, attach the IP-SLA config on the support case. This will help our support engineer better understand your customers
setup.
Lab Exercise #1
Issue: Customer internet performance is slow using GRE
a.Customer has GRE tunnels to Zscaler, but all their users are complaining about slowness. If the
customer disables the tunnels, meaning bypassing Zscaler and going out direct, the issue goes away.
Below are the mtr traces during the slowness.
b.Client-side MTR trace to Zscaler -
• Task:
a.Analyze the MTR traces and determine where the problem might be.
b.From your analysis, how would you communicate the results to the customer?
c.What would you advise the customer as next steps?
Lab Exercise #2
Issue: Customer GRE tunnels are not coming up
a.Customer GRE tunnels to Zscaler are not coming up and they provided you a packet capture from their
GRE router during the failure. Below are the pcaps from the SMLB and SME on Zscaler side
a.Zscaler SME pcap file -
• Customer config:
• Task:
a.Analyze the pcap files and cross verify with the config & determine where the problem might be.
b.From your analysis, how would you communicate the results to the customer?
c.What would you advise the customer as next steps?

Day 1 - MSP Bootcamp Training 201

Uploaded by

Copyright:

Available Formats

Day 1 - MSP Bootcamp Training 201

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Day 1 - MSP Bootcamp Training 201

Uploaded by

Copyright:

Available Formats

Welcome

MSP Bootcamp Training 201

Class Starts at 8:00 am PDT / 9:00 CDT / 11:00 am EDT

©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION

• Completed the 101 training

• Goal is to extend your technical knowledge on Zscaler in live instructor-led sessions by

• ZIA (Zscaler Internet Access)

• Zscaler continues to define the Secure Web

• SMCA : Central Authority

• User/Company config management.

• Taking care of user authentication/provisioning.

Monitor all nodes in cloud

• SMCA monitors the nodes in two ways:

• What happens if a LB VIP/system is marked unhealthy.

Taking part in Master CA election process

• Master CA has very specific roles:

• Types of nodes that handle customer traffic.

• Management interface is handled by the kernel directly, so it's handled by ZscalerOS.

• Enforces customer policies.

• Single SME node can be part of multiple SMLB clusters.

• Terminates both IKEv1 and IKEv2 traffic.

• ZVPN instances can come in two flavors:

• SA information is not synced between the ZVPN instances.

• Terminates SSLVPN tunnels from ZCC client running tunnel2.0.

• The tunnel information is not synced between the SVPN instances.

• SMLB nodes do not terminate any traffic.

• SMLB nodes are provisioned in either of:

• Active-Passive Sample output:

• The output will show:

• All the LB instances will be active for the given VIP.

• We support the following hashing algorithms:

• About Policy Enforcement:

• At a high-level, this is how traffic flows through the modules:

• There are two licenses for Firewall module.

• Standard license allows the following:

• Advanced license allows the following:

Return value format

Example: return "PROXY maa2.sme.zscaler.net:12321; PROXY bom6.sme.zscaler.net:12321; DIRECT";

Proxy responds with 200OK.

• Adding exception for a FQDN (Fully Qualified Domain Name):

• Adding exception for a server IP address:

• Adding exception for a particular source private IP

• Adding exception for a destination IP subnet

• PAC file writing best practices: https://help.zscaler.com/zia/best-practices-writing-pac-files

return "PROXY ${GATEWAY}:80; PROXY ${SECONDARY_GATEWAY}:80; DIRECT";

• Use the following syntax:

• Use the following syntax:

return "PROXY ${COUNTRY_GATEWAY_F0}:80; PROXY ${COUNTRY_SECONDARY_GATEWAY_F0}:80; DIRECT";

return "PROXY ${COUNTRY_GATEWAY_FX}:80; PROXY ${COUNTRY_SECONDARY_GATEWAY_FX}:80; DIRECT";

• The ${COUNTRY} variable supports the countries listed on Maxmind.

• Gateway Host Variable(For Kerberos PAC)

• FX Variable for subcloud:

• Country Variable for subcloud:

• Tunneling works by encapsulating packets: wrapping packets inside of other packets.

Public IP to Public IP communication over the internet

• Zscaler supports 3 tunneling protocols.

• We attach that public IP to customers organization.

• We will provision a primary & secondary DC per public IP of the customer.

• To provision tunnels to RS(Regional Surcharge) DC’s, the provisioning is done by Zscaler

• An incorrectly configured MTU results in higher fragmentation, leading to performance