High Availability

BRKRST-2042
Highly Available
Wide Area Network Design
David Prall, Principal Systems Engineer

CCIE #6508
Agenda
• Introduction
• Cisco IOS and IP Routing
• Convergence Techniques
• Design and Deployment
• Final Wrap Up
BRKRST-2042 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Your speaker
• David Prall
• Principal Systems Engineer
• World Wide Enterprise Networking
• [email protected]
• CCIE 6508 (R&S/SP/Security)
• Started at Cisco July 10, 2000
• Washington, DC
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKRST-2042
Goals
• Efficiently utilize available bandwidth
• Dynamically respond to all types of disruptions
• Leverage most effective design
techniques that meet the design
requirements
• Review today’s technology
Where Can Outages Occur?
Link or Device Failure
MPLS - SP A
C-A-R2 Link or Device Degraded
C-A-R1 C-A-R4
C-A-R3
HQ-W1 BR-W1
MPLS - SP B
HQ-W2 BR-W2
C-B-R1 C-B-R4
• How does outage manifest?

• How quickly can network detect?
• How long is bidirectional reconvergence?
Session Scope
• What methods are used for path selection and packet forwarding
• How does the network detect outages
• Focus on network survivability and effective utilization rather than
sub-second convergence
• Modern Design using SD-WAN
• Does not address “zero loss” considerations
• Please review BRKRST-2365 Unified HA Network Design - The Evolution of the
Next Generation Network
• Other sessions delivered by Matt Birkner
Defining Availability
Availability Downtime / Year
• System Availability: a ratio of the 98.000000% 7.3 Days

expected uptime to the experienced 99.000000% 3.65 Days
downtime over a period of time of 99.500000% 1.825 Days
the same duration 99.900000% 8.76 Hrs
99.990000% 52.56 Min Branch WAN
• Branch WAN High Availability:
Between 99.99% and 99.999% 99.999000% 5.256 Min HA Targets
99.999900% 31.536 Sec
• Ultra High Availability: Between 99.999990% 3.1536 Sec Ultra HA
99.9999% and 99.999999% 99.999999% .31536 Sec Targets
Building Highly Available WANs
Redundancy and Path Diversity Matter
Downtime
SINGLE Downtime Downtime 99.90%*
per Year 99.95%*
per Year per Year
ROUTER, MPLS 4 Hours 8 Hours Internet
SINGLE PATH 4–9 Hours 22 Minutes 46 Minutes
ISR ISR
Branch WAN
HA Solution
99.995% 99.995% 99.995%
SINGLE
ROUTER, 26+ Minutes
DUAL PATHS MPLS MPLS MPLS Internet Internet Internet
ISR ISR ISR
99.999% 99.999% 99.999%
DUAL
ROUTERS, 5+ Minutes
MPLS MPLS MPLS Internet Internet Internet
DUAL PATHS
ISR ISR ISR ISR ISR ISR
* Typical MPLS and Business Grade Broadband Availability SLAs and Downtime per Year, calculated with Cisco AS DAAP tool.
Deployment Options
MPLS/MPLS
MPLS/Internet
MPLS/LTE
Internet/Internet
Internet/LTE
MPLS MPLS
Internet Internet LTE/LTE
LTE LTE
100’s of Combinations
Agenda
• Introduction
• Multiple Links/Multiple Paths
• Load Sharing
• Final Wrap Up
Routing Table Basics
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
p 10.0.0.0/8 is variably subnetted, 14 subnets, 5 masks

B p 10.0.0.0/8 [20/0] via 172.16.0.6, 00:12:36
B p 10.3.0.0/16 [20/0] via 172.16.0.6, 00:12:36
B p 10.4.0.0/16 [200/0], 00:13:52, Null0
C p 10.4.0.41/32 is directly connected, Loopback0
D p 10.4.1.0/24 [90/307200] via 10.4.49.2, 00:14:32, Ethernet0/0
C p 10.4.49.0/30 is directly connected, Ethernet0/0
L p 10.4.49.1/32 is directly connected, Ethernet0/0
B p 10.9.0.0/16 [20/0] via 172.16.0.6, 00:12:36
100.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
B 100.64.0.0/24 [20/0] via 100.64.3.1, 00:13:43
C 100.64.3.0/24 is directly connected, Ethernet0/2
L 100.64.3.2/32 is directly connected, Ethernet0/2
172.16.0.0/16 is variably subnetted, 9 subnets, 2 masks
B 172.16.0.0/31 [20/0] via 172.16.0.6, 00:12:36
C 172.16.0.6/31 is directly connected, Ethernet0/1
L 172.16.0.7/32 is directly connected, Ethernet0/1
INFORMATIONAL
Administrative Distance
Default
Route Source Distance
• The distance command is used to configure Connected
a rating of the trustworthiness of a routing 0
Interface
information source, such as an individual Static Route 1
router or a group of routers EIGRP Summary
5
Route
• Numerically, an administrative distance is a BGP External
20
positive integer from 1 to 255. In general, the (eBGP)
higher the value, the lower the trust rating EIGRP Internal 90
OSPF 110
• An administrative distance of 255 means the IS-IS 115
routing information source cannot be trusted RIP 120
at all and should be ignored EIGRP External 170
BGP Internal
200
(iBGP)
Unknown 255
Route Selection OSPF EIGRP OSPF
• How is administrative
distance used to 10.0.14.0/24 10.0.14.0/24 10.0.14.0/25
determine which route 10.0.14.128/25
These Two Routes
should be installed? Are Identical
EIGRP Internal = 90
• Only identical routes OSPF = 110
are compared EIGRP Internal Installed
• Identical prefixes with router#show ip route 10.0.14.0 255.255.255.0
Routing entry for 10.0.14.0/24
different prefix lengths Known via "eigrp 1", distance 90, metric 307200, type internal
are not the same route Redistributing via eigrp 1
Last update from 10.0.121.2 on Ethernet0/1, 00:01:32 ago
• The route from the Routing Descriptor Blocks:
* 10.0.121.2, from 10.0.121.2, 00:01:32 ago, via Ethernet0/1
protocol with the lower Route metric is 307200, traffic share count is 1
administrative distance Total delay is 2000 microseconds, minimum bandwidth is 10000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
is installed Loading 1/255, Hops 1
Route Selection
OSPF EIGRP OSPF
• What about longest prefix
comparison?
10.0.14.0/24 10.0.14.0/24 10.0.14.0/25
• Only identical routes are 10.0.14.128/25
These Two Routes
compared Are Identical
• Identical prefixes with
different prefix lengths
are not the same route
OSPF Installed
• The route with the longest Longer Prefixes
prefix is router#show ip route 10.0.14.0 255.255.255.0 longer-prefixes
installed 10.0.0.0/8 is variably subnetted, 9 subnets, 3 masks
D 10.0.14.0/24 [90/307200] via 10.0.121.2, 00:01:35, Ethernet0/1
O 10.0.14.0/25 [110/20] via 10.0.122.2, 00:00:50, Ethernet0/2
O 10.0.14.128/25 [110/20] via 10.0.122.2, 00:00:50, Ethernet0/2
More Specific OSPF Override EIGRP

Agenda
• Introduction
• Multiple Links/Multiple Paths
• Load Sharing
• Final Wrap Up
Load Sharing
• Assume the same routing process attempts to install two routes for the
same destination in the RIB
• The routing process may allow the second route to be installed based on
its own rules
IGP OSPF IS-IS EIGRP
Route Cost Must be equal to Must be equal to Must be less than the
installed route installed route variance times the
lowest cost installed
route
Maximum Paths Must be fewer than maximum-paths configured under the routing
process (default = 4, maximum = 32)
Note: BGP default value for maximum-paths = 1

CEF Load Sharing
Per-Destination Per-Packet1
Default behaviour of IOS Universal Requires “ip load-sharing per-packet”
Algorithm “show cef state” interface configuration1
Per-flow using destination hash Per-packet using round-robin method
Packets for a given source/destination Packets for a given source/destination
session will take the same path session may take different paths
More effective as the number of Ensures traffic is more evenly
destinations increase distributed over multiple paths
Ensures that traffic for a given session Potential for packets to arrive out of
arrives in order sequence
1Not available in IOS-XE based images

Load Sharing
router#show ip route 192.168.239.0
Known via "eigrp 100", distance 170, metric 3072256, type external
Redistributing via eigrp 100
Last update from 192.168.245.11 on Serial0/2/1, 00:18:17 ago
Routing Descriptor Blocks:
* 192.168.246.10, from 192.168.246.10, 00:18:17 ago, via Serial2/0
Route metric is 3072256, traffic share count is 1
....
192.168.245.11, from 192.168.245.11, 00:18:17 ago, via Serial2/1
....
The Traffic Share Count Is Critical to

Understanding the Actual Load Sharing of
Packets Using These Two Routes
3072256/3072256 = 1
Load Sharing – with EIGRP Variance
Known via "eigrp 100", distance 170, metric 3072256, type external
Redistributing via eigrp 100
Last update from 192.168.245.11 on Serial0/2/1, 00:18:17 ago
* 192.168.246.10, from 192.168.246.10, 00:18:17 ago, via Serial2/0
....
192.168.245.11, from 192.168.245.11, 00:18:17 ago, via Serial2/1
....
If the Lower Metric Is Less than the Second
Metric, the Traffic Share Count Will Be
Something Other than 1 (EIGRP with Variance
Configured)
3072256/3072256 = 1
3072256/1536128 = 2
2x Faster Link Gets 2 Flows vs. 1 Flow 23
BRKRST-2042 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Load Sharing – with eBGP dmzlink-bw
Only routes learned
Known via "bgp 1", distance 20, metric 0 via eBGP Neighbors
Tag 2, type external
Last update from 10.0.122.2 00:00:16 ago
10.0.122.2, from 10.0.122.2, 00:00:16 ago
....
* 10.0.121.2, from 10.0.121.2, 00:00:16 ago
....
router#show ip bgp 192.168.239.0
BGP routing table entry for 192.168.239.0/24, version 9
Paths: (2 available, best #2, table default)
Multipath: eBGP
.... 2x Faster Link Gets 2 Flows vs. 1 Flow
10.0.122.2 from 10.0.122.2 (10.0.0.2)
Origin IGP, metric 0, localpref 100, valid, external, multipath(oldest)
DMZ-Link Bw 312 kbytes
rx pathid: 0, tx pathid: 0
....
10.0.121.2 from 10.0.121.2 (10.0.0.2)
Origin IGP, metric 0, localpref 100, valid, external, multipath, best
DMZ-Link Bw 625 kbytes
rx pathid: 0, tx pathid: 0x0
CEF Hashing and Exact Route
• Now that we have load balancing

• Which exact path are the flows using
• “show ip cef exact-route <src-addr> [src-port] <dest-addr> [dest-port]”
#show ip cef exact-route 1.1.1.1 2.2.2.2

1.1.1.1 -> 2.2.2.2 =>IP adj out of GigabitEthernet1, addr 10.255.0.1
Agenda
• Introduction
• Interface Detection
• Routing Protocols
• Static Routing and EOT
• First Hop Redundancy Protocols
• Cisco SD-WAN (Viptela)
Interface Detection – Carrier-Delay
• Carrier-delay
• If a link goes down and comes back up before the carrier delay timer expires, the
down state is effectively filtered, and the rest of the software on the router is not
aware that a link-down event occurred.
• Imposes a default 2 second pause before processing interface events
• Disabling carrier-delay speeds convergence upon interface events
• Disabling carrier-delay can increase control-plane usage during repetitive
interface events (flapping)
Interface Detection - Dampening
• Dampening
• Imposes a logarithmic delay based on interface events
• Coupled with carrier-delay, dampening protects the control-plane from
repetitive events by increasing the delay before processing up events
should the interface flap.
#conf t
(config-if)#interface GigabitEthernet1
(config-if)#carrier-delay 0
(config-if)#dampening
(config-if)#end
#show dampening interface
1 interface is configured with dampening.
No interface is being suppressed.
Features that are using interface dampening:
IP Routing
Agenda
• Introduction
Routing Protocol Timers INFORMATIONAL
Keepalive (B) Holdtime (B,E,I)

Hello (E,I,O) Invalid (R) Dead (O) Flush (R)
Update (R) Holddown (R)
BGP 60 180
EIGRP
5 (60) 15 (180)
(< T1)
IS-IS
10 (3.333) 30 (10)
(DIS)
OSPF
10 (30) 40 (120)
(NBMA)
RIP/RIPv2 30 180 180 240
Note: Cisco Default Values

Routing Protocol Neighbor Behavior INFORMATIONAL
R2
R1 R4
R3
Recovery Times by Protocol
Link Down Link Up Link Up Link Up

Line Protocol Down Loss 100% Neighbor Down Loss ~5%
BGP ~1s 180 180 Never
EIGRP
~ 1s 15 (180) 15 (180) Never
(< T1)
IS-IS
~ 1s 30 (10) 30 (10) Never
(DIS)
OSPF
~ 1s 40 (120) 40 (120) Never
(NBMA)
RIP/RIPv2 ~ 1s 240 240 Never

Note: Using Cisco Default Values
Routing Protocol Neighbor Behavior
Adjust Hello Timers
R2
R1 R4 BR-W1
R3
R4#show ip bgp vpnv4 vrf cisco neighbor

BGP neighbor is 192.168.101.10, vrf cisco, remote AS 65110, external link
BGP version 4, remote router ID 192.168.201.10
BGP state = Established, up for 1d10h
Last read 00:00:19, hold time is 180, keepalive interval is 60 seconds
BR-W1#
router bgp 65110
R4#show ip bgp vpnv4 vrf cisco neighbor neighbor 192.168.101.9 timers 7 21
BGP neighbor is 192.168.101.10, vrf cisco, remote AS 65110, external link
BGP version 4, remote router ID 192.168.201.10
BGP state = Established, up for 00:01:23
Last read 00:00:03, hold time is 21, keepalive interval is 7 seconds
When Configuring the Holdtime Argument for a Value of Less than
Twenty Seconds, the Following Warning Is Displayed:
%Warning: A Hold Time of Less than 20 Seconds
Increases the Chances of Peer Flapping
Bidirectional Forwarding Detection (BFD)
• Extremely lightweight hello protocol
• IPv4, IPv6, MPLS, P2MP
• 10s of milliseconds (technically, microsecond resolution) forwarding plane
failure detection mechanism.
• Single mechanism, common and standardized
• Multiple modes: Async (echo/non-echo), Demand
• Independent of Routing Protocols
• Levels of security, to match conditions and needs
• Facilitates close alignment with hardware
Drivers for BFD
• Link-layer detection misses some types of outages
• e.g. Control Plane failure
• Control Plane failure detection is very conservative

• 15-180 seconds in default configurations
• Link-layer failure detection is not consistent across media types

• Less than 50ms on APS- protected SONET
• A few seconds on Ethernet
• Several seconds or more on WAN links
• Provides a measure of consistency across routing protocols

• Most current failure detection mechanisms are an order of magnitude too long for
time-sensitive applications
Bidirectional Forwarding Detection
interface GigabitEthernet4
ip address 10.3.255.9 255.255.255.252
bfd interval 50 min_rx 50 multiplier 3
router eigrp 1
network 10.3.0.0 0.0.255.255
bfd all-interfaces
(Gi2)
R1#show bfd neighbors details
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int R1
10.3.255.10 4104/1 Up Up Gi4 (Gi4)
Session state is UP and using echo function with 50 ms interval. interface GigabitEthernet2
Session Host: Software ip address 172.17.2.9 255.255.255.254
OurAddr: 10.3.255.9 bfd interval 333 min_rx 333 multiplier 3
Handle: 2 router bgp 65000
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3 neighbor 172.17.2.8 fall-over bfd
Received MinRxInt: 1000000, Received Multiplier: 3
R2
Holddown (hits): 0(0), Hello (hits): 1000(1371)
Rx Count: 985, Rx Interval (ms) min/max/avg: 34/1978/1226 last: 290 ms ago
Tx Count: 1372, Tx Interval (ms) min/max/avg: 71/1137/879 last: 721 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: EIGRP CEF
Uptime: 00:20:06
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3
My Discr.: 1
- Length: 24
- Your Discr.: 4104
Configured in milliseconds (ms)
Min tx interval: 1000000 - Min rx interval: 1000000 Displayed in microseconds (µs)
Min Echo interval: 50000
Bidirectional Forwarding Detection
interface GigabitEthernet4
ip address 10.3.255.9 255.255.255.252
router eigrp 1
network 10.3.0.0 0.0.255.255
bfd all-interfaces
(Gi2)
<show bfd neighbors details cont’d>
IPv4 Sessions
NeighAddr LD/RD RH/RS State Int R1
172.17.2.8 4102/1 Up Up Gi2 (Gi4)
Session state is UP and using echo function with 333 ms interval. interface GigabitEthernet2
Session Host: Software ip address 172.17.2.9 255.255.255.254
OurAddr: 172.17.2.9 bfd interval 333 min_rx 333 multiplier 3
Handle: 1 router bgp 65000
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3 neighbor 172.17.2.8 fall-over bfd
Received MinRxInt: 1000000, Received Multiplier: 3
R2
Holddown (hits): 0(0), Hello (hits): 1000(6076)
Rx Count: 4977, Rx Interval (ms) min/max/avg: 4/1970/1069 last: 491 ms ago
Tx Count: 6077, Tx Interval (ms) min/max/avg: 754/1180/879 last: 655 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: BGP CEF
Uptime: 01:29:04
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
C bit: 0
Multiplier: 3
My Discr.: 1
- Length: 24
- Your Discr.: 4102
Configured in milliseconds (ms)
Min tx interval: 1000000 - Min rx interval: 1000000 Displayed in microseconds (µs)
Min Echo interval: 333000
Detecting Unreachable Neighbor (Hello Timers vs. BFD)
100% Packet Loss
R1 R2
(Link Up)
EIGRP Default: Elapsed Time Between 10 – 15 Sec
R1#show clock
*09:58:27.716 UTC Sat Jan 27 2018
R1# 12.896
*Jan 27 09:58:40.612: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor
10.3.255.10 (GigabitEthernet4) is down: holding time expired seconds
BFD: Elapsed Time Between 100 - 150 ms with 50ms interval
R1#show clock
*09:35:44.408 UTC Sat Jan 27 2018
R1#
*Jan 27 09:35:45.571: %BFDFSM-6-BFD_SESS_DOWN: BFD-SYSLOG: BFD
session ld:4101 handle:2,is going Down Reason: ECHO FAILURE
*Jan 27 09:35:45.575: %BFD-6-BFD_SESS_DESTROYED: BFD-SYSLOG: 1.172
bfd_session_destroyed, ld:4101 neigh proc:EIGRP, handle:2 act
*Jan 27 09:35:45.580: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor seconds1
10.3.255.10 (GigabitEthernet4) is down: BFD peer down notified
1 injecting 100% loss after hitting show clock in the lab BRKRST-2042 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Agenda
• Introduction
EOT, Static Routing, and DDR
• Enhanced Object Tracking (EOT)
• Static Routing Options
• Floating Static Routes
• Reliable Static Routing (RSR) using EOT
• Dial on Demand Routing (DDR)
• EEM Script
• DMVPN State Tracking
• More information:
• http://cs.co/ddrbackup
• Expands to https://www.cisco.com/c/en/us/support/docs/dial-access/dial-on-demand-routing-ddr/10213-backup-main.html
Enhanced Object Tracking (EOT)
Local Significance
Track Options Syntax
Line-Protocol State track object-number interface type number line-protocol
of Interface track 1 interface serial 2/0 line-protocol
IP-Routing State of track object-number interface type number ip routing
Interface track 2 interface ethernet 1/0 ip routing
IP-Route track object-number ip route IP-Addr/Prefix-len reachability
Reachability track 3 ip route 10.16.0.0/16 reachability
Threshold* of IP- track object-number ip route IP-Addr/Prefix-len metric threshold
Route Metrics track 4 ip route 10.16.0.0/16 metric threshold
Router#show track 100 Router#show track 103

Track 100
Interface Serial2/0 line-protocol
Track 103
IP route 10.16.0.0 255.255.0.0 reachability
IPv6
Line protocol is Up Reachability is Up (EIGRP) Support
1 change, last change 00:00:05 1 change, last change 00:02:04 15.3(3)S
Tracked by:
GLBP FastEthernet0/1 1
First-hop interface is FastEthernet0/0
Tracked by:
15.4(1)T
GLBP FastEthernet0/1 1
* EIGRP, OSPF, BGP, Static Thresholds Are Scaled to Range of (0 – 255)

External Significance

track object-number ip sla type number state
IP SLAs Operation track 5 ip sla 4 state
Reachability of an IP track object-number ip sla type number reachability
SLAs Host track 6 ip sla 4 reachability
Types of IP SLA Probes:

dhcp http path-jitter
dns icmp-echo1 tcp-connect1
ethernet icmp-jitter udp-echo1
frame-relay mpls udp-jitter1
ftp path-echo voip
1Available for IPv6
Compound Operations
track object-number list boolean {and|or}

and - both are up for object to be up
or - one is up for object to be up
list boolean track 5 list boolean or
object 51
object 52 not ! Negates state of object
track object-number list threshold {weight|percentage}
track 6 list threshold weight
object 61 weight 20 ! Twice as important
list threshold object 62 ! Default weight 10
object 63
object 64
threshold weight up 30 down 25
Static Host Route Guarantees probe
Reliable Static Routing destination only reachable via desired
path
Tracking IP SLA track 4 list boolean or
object 400
object 401 Permanent to guarantee probes only utilize
track 400 ip sla 400 reachability desired path. Stay down when down.
track 401 ip sla 401 reachability
ip sla 400
icmp-echo 10.100.100.100 source-ip 10.1.2.120
IP SLA IP SLA
timeout 100
frequency 10
ip sla schedule 400 life forever start-time now
ip sla 401
(.9) (.9) timeout 100
frequency 10
!
ip route 10.100.100.100 255.255.255.255 Ethernet 0/1 192.168.101.9 permanent
ip route 10.100.200.100 255.255.255.255 Ethernet 0/1 192.168.101.9 permanent
ip route 10.100.0.0 255.255.0.0 192.168.101.9 track 4
192.168.101.8/29 192.168.201.8/29
BR-W1 ip route 10.100.0.0 255.255.0.0 192.168.201.9 200
BR-W1#show ip route track-table

ip route 10.100.0.0 255.255.0.0 192.168.101.9 track 4 state is [up]
BR-W1#show ip route 10.100.0.0 255.255.0.0
S 10.100.0.0/16 [1/0] via 192.168.101.9
Reliable Static Routing
Tracking IP SLA
BR-W1#
*Mar 12 03:57:28.367: %TRACKING-5-STATE: 400 ip sla 400 reachability Up->Down Unable to Reach
*Mar 12 03:57:37.374: %TRACKING-5-STATE: 401 ip sla 401 reachability Up->Down
IP SLA
*Mar 12 03:57:38.137: %TRACKING-5-STATE: 4 list boolean or Up->Down IP SLA IP SLA
Responders
(.9) (.9)
192.168.101.8/29 192.168.201.8/29
BR-W1#show ip route track-table

ip route 10.100.0.0 255.255.0.0 192.168.101.9 track 4 state is [down]
BR-W1#show ip route 10.100.0.0 255.255.0.0 longer-prefixes Floating Static
BR-W1
S 10.100.0.0/16 [200/0] via 192.168.201.9 Installed
S 10.100.100.100/32 [1/0] via 192.168.101.9, Ethernet0/1
S 10.100.200.100/32 [1/0] via 192.168.101.9, Ethernet0/1
IPv6 Reliable Static Routing added in 15.4(1)T

EEM Script
IPv6 Static Route Event Tracking
ipv6 route 2001:DB8::12/128 2001:DB8:B::5
ip sla 610
Unable to Reach icmp-echo 2001:DB8::12 source-interface GigabitEthernet0/1.99
threshold 1000
IP SLA IP SLA frequency 10
Responder ip sla schedule 610 life forever start-time now
WAN RTR track 600 list threshold percentage
WAN RTR
<snip additional tracked objects>
object 610
2001:DB8:B::5 threshold percentage down 40 up 60
track 610 ip sla 610
event manager applet DISABLE-STATIC-IPv6 Don’t forget to reenable
event track 600 state down
action 1 cli command "enable"
BR RTR action 2 cli command "configure terminal"
action 3 cli command "no ipv6 route ::/0 2001:DB8:B::5"
action 4 cli command "end"
action 99 syslog msg “DEFAULT IPv6 ROUTE DISABLED"
BR-RTR#
14:22:14: %TRACKING-5-STATE: 610 ip sla 610 state Up->Down
14:22:14: %TRACKING-5-STATE: 600 list threshold percentage Up->Down
14:22:14: %SYS-5-CONFIG_I: Configured from console by on vty0(EEM:DISABLE-STATIC-IPv6)
14:22:14: %HA_EM-6-LOG: DISABLE-STATIC-IPv6: DEFAULT IPv6 ROUTE DISABLED
15.4(1)T added Reliable Static Routing
Black Hole Route Detection ip sla 110
IPSLA with EEM icmp-echo 208.67.222.222 source-interface GigabitEthernet0/0

vrf INET-PUBLIC1 ! fVRF configuration
threshold 1000
frequency 15
Lost connection to ISP but DHCP route ip sla schedule 110 life forever start-time now
ip sla 111
stays in the routing table icmp-echo 208.67.220.220 source-interface GigabitEthernet0/0
vrf INET-PUBLIC1
threshold 1000
frequency 15

track 62 list boolean or
IP SLA object 60
Probes object 61
(config)#ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 dhcp 10 ?

<cr>
event manager applet DISABLE-STATIC-GIG0-0

event track 62 state down
action 2 cli command "configure terminal"
action 3 cli command "no ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 dhcp 10"
Note: This method is compatible with action 99 syslog msg “DEFAULT IP ROUTE via GIG0/0 DISABLED"
dual Internet DHCP design. event manager applet ENABLE-STATIC-GIG0-0
event track 62 state up
action 3 cli command "ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 dhcp 10"
action 99 syslog msg “DEFAULT IP ROUTE via GIG0/0 ENABLED"
Black Hole Route Detection
IPSLA with Recursive Routing
Interface GigabitEthernet0/0
vrf forwarding INET-PUBLIC1
Lost connection to ISP but DHCP route ip address dhcp
stays in the routing table ip sla 110
icmp-echo 208.67.222.222 source-interface GigabitEthernet0/0
vrf INET-PUBLIC1 ! fVRF configuration
threshold 1000
frequency 15
ip sla 111
IP SLA icmp-echo 208.67.220.220 source-interface GigabitEthernet0/0
Probes vrf INET-PUBLIC1
threshold 1000
frequency 15

track 62 list boolean or
Note: This method is compatible with object 60
dual Internet DHCP design. object 61
(config)#ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 dhcp 10 ?
<cr>
ip route 192.0.2.33 255.255.255.255 GigabitEthernet0/0 dhcp 10
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 192.0.2.33 10 track 62
EEM Script
LTE Backup with Event Tracking ip sla 100
icmp-echo 192.168.4.22 source-interface GigabitEthernet0/1
threshold 1000
frequency 15

Don’t forget to disable
event manager applet ACTIVATE-LTE
VPN RTR
WAN RTR event track 60 state down
192.168.4.22
NAS action 3 cli command "interface cellular0/0/0"
action 4 cli command "no shutdown"
(Ce0/0/0) action 99 syslog msg "Activating LTE interface"
14:22:14: %TRACKING-5-STATE: 60 ip sla 100 reachability Up->Down
LTE-RTR 14:22:14: %SYS-5-CONFIG_I: Configured from console by on vty0(EEM:ACTIVATE-LTE)
14:22:14: %HA_EM-6-LOG: ACTIVATE-LTE: Activating LTE interface
14:22:34: %LINK-3-UPDOWN: Interface Cellular0/0/0, changed state to up
14:22:34: %DIALER-6-BIND: Interface Ce0/0/0 bound to profile Di1
14:22:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface Cellular0/0/0, changed state to up
14:22:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel11, changed state to up
14:22:40: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
14:22:42: %DUAL-5-NBRCHANGE: EIGRP-IPv4 201: Neighbor 10.4.36.1 (Tunnel11) is up: new
adjacency
http://www.cisco.com/go/cvd/wan VPN Remote Site over 3G/4G/LTE Technology Design Guide

DMVPN Interface State Control track 2 list boolean or
LTE Backup with DMVPN object 101 not
track 101 interface Tunnel100 line-protocol
interface Tunnel200
if-state track 2
tunnel source Cellular0/0/0
end
#show track 2
Track 2
List boolean or
VPN RTR Boolean OR is Down
WAN RTR
7 changes, last change 00:07:55
object 101 not Up
192.168.4.22 Tracked by:
NAS
IF-State Control 2
17:24:18.682: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel100, changed state to down
(Ce0/0/0)
17:24:18.682: %TRACK-6-STATE: 101 interface Tu100 line-protocol Up -> Down
17:24:18.744: %TRACK-6-STATE: 2 list boolean or Down -> Up
17:24:28.683: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel200, changed state to up
LTE-RTR 17:24:29.276: %BGP-5-ADJCHANGE: neighbor 192.168.200.12 Up
17:24:37.505: %BGP-5-ADJCHANGE: neighbor 192.168.200.22 Up
#show track 2
Track 2
List boolean or
Boolean OR is Up
8 changes, last change 00:00:32
object 101 not Down
Tracked by:
IF-State Control 2
Agenda
• Introduction
First Hop Redundancy Protocols (FHRP)
Failure Protection for the First Hop IP Router
• Hot Standby Router Protocol (HSRP)
• v2 IPv4 and IPv6
BR-W1 BR-W2
• Virtual Router Redundancy Protocol (VRRP)
• RFC5798 (v3 IPv4 and IPv6), RFC3768 (v2 IPv4), RFC2338 (v1)
• Gateway Load Balancing Protocol (GLBP)

• IPv4 and IPv6
Drivers for FHRPs
• Provide routing redundancy for access layer
• How to handle failover when end-hosts have only a single IP default gateway and
cached ARP entry
• Provide routing redundancy for devices that depend on static routing
• Some firewalls do not support dynamic routing
• Independent of routing protocols
• Works with any routing protocol and static routing
• Capable of providing sub-second failover
• Provides load sharing capabilities (GLBP) transparent to end host
Hot Standby Routing Protocol (HSRP)
interface FastEthernet0/0
ip address 10.1.2.2 255.255.255.0 interface FastEthernet0/0
standby version 2 ip address 10.1.2.3 255.255.255.0
standby 4 ip 10.1.2.1 standby version 2
standby 4 priority 110 Active Standby standby 4 ip 10.1.2.1
standby 4 preempt Router Router standby 4 preempt
BR-W1 BR-W2
standby 6 ipv6 autoconfig (.2)
HSRP
(.3)
standby 6 ipv6 autoconfig
standby 6 priority 110 VIP (.1) standby 6 preempt
standby 6 preempt ipv6 address 2001:DB8:5:1::2/64
ipv6 address 2001:DB8:5:1::1/64
Default Gateway: (.1)
DG MAC: MAC VIP
BR-W1#show standby brief
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP
Fa0/0 4 110 P Active local 10.1.2.3 10.1.2.1
Fa0/0 6 110 P Active local FE80::A8BB:CCFF:FE00:3400
FE80::5:73FF:FEA0
:6
Fa0/0 4 100 P Standby 10.1.2.2 local 10.1.2.1
Fa0/0 6 100 P Standby FE80::A8BB:CCFF:FE00:3300
HSRP—Global IPv6 Addresses Available local FE80::5:73FF:FEA0
for Static Deployments :6
Active
Local Router
BR-W1 BR-W2
HSRP
Failures (.2)
(.1)
(.3)
VIP

DG MAC: MAC VIP

|
Fa0/0 4 100 P Active local unknown 10.1.2.1
Fa0/0 6 100 P Active local unknown FE80::5:73FF:FEA0
:6
Complex Failure
Upstream/Remote Requires
“Enhanced Object
Failures Tracking (EOT)”
Active Standby Active

Router Router Router
BR-W1 BR-W2 BR-W1 BR-W2
HSRP HSRP
(.2) (.3) (.2) (.3)
(.1) VIP (.1) VIP
#track 100 interface serial2/0 line-protocol

!
standby version 2
standby 4 priority 110
standby 4 track 100 decrement 20
standby 6 priority 110
standby 6 track 100 decrement 20
BFD interface FastEthernet0/0
R1#show bfd neighbors details

Active <SNIP>
Local Router Registered protocols: HSRP
BR-W1 BR-W2
HSRP <SNIP>
Failures (.2)
(.1)
(.3)
VIP

DG MAC: MAC VIP
standby bfd all-interfaces ! default

!
standby bfd ! Required only when all-interfaces disabled
Gateway Load Balancing Protocol (GLBP) BR-W1#show run int fa0/0
ip address 10.1.2.2 255.255.255.0
AVG = Active Virtual Gateway glbp 4 ip 10.1.2.1
glbp 4 preempt
SVG = Standby Virtual Gateway glbp 4 weighting 110 lower 100
glbp 6 ipv6 autoconfig
AVF = Active Virtual Forwarder glbp 6 preempt
glbp 6 weighting 110 lower 100
ipv6 address 2001:DB8:5:1::1/64
BR-W1#show glbp brief
Interface Grp Fwd Pri State Address Active router Standby router
Fa0/0 4 - 100 Active 10.1.2.1 local 10.1.2.3
Fa0/0 4 1 - Active 0007.b400.0401 local -
AVG SVG Fa0/0 4 2 - Listen 0007.b400.0402 10.1.2.3 -
Fa0/0 6 - 100 Active FE80::7:B4FF:FE00:600
AVF A AVF B local FE80::A8BB:CCF
BR-W1 BR-W2
GLBP F:FE00:3400
(.2) (.3)
Fa0/0 6 1 - Active 0007.b400.0601 local -
VIP (.1) (.1) VIP Fa0/0 6 2 - Listen 0007.b400.0602 FE80::A8BB:CCFF:FE00:3400
-
Fa0/0 4 - 100 Standby 10.1.2.1 10.1.2.2 local
Fa0/0 4 1 - Listen 0007.b400.0401 10.1.2.2 -
Fa0/0 4 2 - Active 0007.b400.0402 local -
Default Gateway: Default Gateway: (.1) Fa0/0 6 - 100 Standby FE80::7:B4FF:FE00:600
(.1) DG MAC: AVF B FE80::A8BB:CCFF:FE00:3300
DG MAC: AVF A local
Fa0/0 6 1 - Listen 0007.b400.0601 FE80::A8BB:CCFF:FE00:3300
-
Fa0/0 6 2 -
BRKRST-2042 Active 0007.b400.0602 local
© 2019 Cisco and/or its affiliates. -Public BRKRST-2042
All rights reserved. Cisco 57
Gateway Load Balancing Protocol (GLBP)
AVG = Active Virtual Gateway
SVG = Standby Virtual Gateway
AVF = Active Virtual Forwarder
BR-W2#
*May 26 19:09:14.260: %GLBP-6-STATECHANGE: FastEth0/0 Grp 4 state Standby -> Act
ive
*May 26 19:09:15.326: %GLBP-6-FWDSTATECHANGE: FastEth0/0 Grp 4 Fwd 1 state Liste
n -> Active
*May 26 19:09:15.826: %GLBP-6-STATECHANGE: FastEth0/0 Grp 6 state Standby -> Act
ive AVG
*May 26 19:09:16.856: %GLBP-6-FWDSTATECHANGE: FastEth0/0 Grp 6 Fwd 1 state Liste
n -> Active AVF A
BR-W1 BR-W2
GLBP AVF B
Local (.2) (.3)
(.1) VIP
Failures

Fa0/0 4 - 100 Active 10.1.2.1 local unknown
Fa0/0 4 1 - Active 0007.b400.0401 local -
Fa0/0 4 2 - Active 0007.b400.0402 local -
Fa0/0 6 - 100 Active FE80::7:B4FF:FE00:600 Default Gateway: (.1) Default Gateway: (.1)
local unknown DG MAC: AVF A DG MAC: AVF B
Fa0/0 6 1 - Active 0007.b400.0601 local -
Fa0/0 6 2 - Active 0007.b400.0602 local -
GLBP with Enhanced Object Tracking
AVG = Active Virtual Gateway
SVG = Standby Virtual Gateway
AVF = Active Virtual Forwarder
Complex Failure
Requires
Upstream/Remote “Enhanced Object
Failures Tracking (EOT)”
Requires “Enhanced
Object Tracking”
AVF A
BR-W1 AVG
AVF A
BR-W2 BR-W1 AVG BR-W2
GLBP AVF B GLBP AVF B
(.2) (.3) (.2) (.3)
(.1) (.1) VIP
VIP
Branch
Tracking IP SLA
ip sla 100
Lo0 Lo0
10.100.100.100 10.100.200.100 timeout 100
frequency 10
IP SLA IP SLA ip sla schedule 100 life forever start-time now
ip sla 200
timeout 100
frequency 10
ip route 10.100.100.100 255.255.255.255 FastEthernet0/1 192.168.101.9 permanent
ip route 10.100.200.100 255.255.255.255 FastEthernet0/1 192.168.101.9 permanent
BR-W1#show ip sla statistics
IPSLA operation id: 100
Latest RTT: 1 milliseconds
AVF A AVF B Latest operation start time: *04:42:11.444 UTC Tue Feb 17 2009
Latest operation return code: OK
BR-W1 BR-W2
GLBP Number of successes: 46
(.2) (.3) Number of failures: 0
VIP (.1) (.1) VIP Operation time to live: Forever
IPSLA operation id: 200
Latest RTT: 1 milliseconds
Latest operation start time: *04:42:11.356 UTC Tue Feb 17 2009
Latest operation return code: OK
Number of successes: 24
Number of failures: 0
Operation time BRKRST-2042
to live: Forever© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Enhanced Object Tracking
Tracking IP SLA
BR-W1#
track 100 ip sla 100 reachability BR-W1#show glbp
track 200 ip sla 200 reachability FastEthernet0/0 – Group 4
track 1 list boolean or State is Active
1 state change, last state change 00:09:59
object 100 IP SLA IP SLA Virtual IP address is 10.1.2.1
object 200 Hello time 3 sec, hold time 10 sec
interface FastEthernet0/0 Next hello sent in 2.336 secs
ip address 10.1.2.2 255.255.255.0 Redirect time 600 sec, forwarder timeout 14400 sec
Preemption enabled, min delay 0 sec
glbp 4 ip 10.1.2.1 Active is local
glbp 4 priority 110 Standby is 10.1.2.3, priority 105 (expires in 7.808 sec)
glbp 4 preempt Priority 110 (configured)
glbp 4 weighting 110 lower 100 Weighting 110 (configured 110), thresholds: lower 100,
glbp 4 load-balancing weighted upper 110
Track object 1 state Up decrement 20
glbp 4 weighting track 1 decrement 20 Load balancing: weighted
Group members:
AVF A AVF B aabb.cc00.0110 (10.1.2.2) local
aabb.cc00.0410 (10.1.2.3)
BR-W1 BR-W2 There are 2 forwarders (1 active)
GLBP
(.2) (.3) Forwarder 1
VIP (.1) (.1) VIP State is Active
<SNIP>
Forwarder 2
State is Listen
<SNIP>
Enhanced Object Tracking
Composite Failure BR-W1#
*Feb 17 05:17:25: %TRACKING-5-STATE: 100 ip sla 100 state Up->Down
*Feb 17 05:17:25: %TRACKING-5-STATE: 200 ip sla 200 state Up->Down
*Feb 17 05:17:26: %TRACKING-5-STATE: 1 list boolean or Up->Down
*Feb 17 05:17:38: %GLBP-6-FWDSTATECHANGE: FastEth0/0 Grp 4 Fwd 1 state
Active -> Listen
BR-W2#show glbp IP SLA IP SLA
FastEthernet0/0 – Group 4
State is Standby
1 state change, last state change 00:28:16
Virtual IP address is 10.1.2.1
BR-W1 Remains Hello time 3 sec, hold time 10 sec
Next hello sent in 1.856 secs
Active Virtual Redirect time 600 sec, forwarder timeout 14400 sec
Unable to Reach
Preemption enabled, min delay 0 sec Either
Gateway (AVG) Active is 10.1.2.2, priority 110 (expires in 10.400 sec)
Standby is local IP SLA
Priority 105 (configured) Responder
Weighting 110 (configured 110), thresholds: lower 100, upper 110
AVF A
Track object 1 state Up decrement 20
AVG
Load balancing: weighted AVF B
BR-W2 Becomes Group members:
BR-W1
GLBP
BR-W2
aabb.cc00.0110 (10.1.2.2) (.2) (.3)
Active Virtual aabb.cc00.0410 (10.1.2.3) local (.1) VIP
Forwarder (AVF) There are 2 forwarders (2 active)

Forwarder 1
for both A and B State is Active
<SNIP>
Forwarder 2
State is Active
<SNIP>
Agenda
• Introduction
Overlay Management Protocol (OMP)
vSmart • TCP based extensible control plane protocol
• Runs between WAN Edge routers and vSmart
controllers and between the vSmart controllers
- Inside TLS/DTLS connections
• Leverages address families to advertise
reachability for TLOCs, unicast/multicast
vSmart vSmart
destinations (statically/dynamically learnt service
side routes), service routes (L4-L7), BFD stats
(TE and H-SDWAN) and Cloud onRamp for SaaS
probe stats (gateway)
- Uses attributes
WAN Edge WAN Edge

• Distributes IPSec encryption keys, and data and
app-aware policies (embedded NETCONF)
Note: WAN Edge routers need not connect to all vSmart Controllers
Bidirectional Forwarding Detection (BFD)
WAN Edge
• Path liveliness and quality measurement detection
protocol
- Up/Down, loss/latency/jitter, IPSec tunnel
MTU
• Runs between all WAN Edge routers in the
WAN Edge WAN Edge
topology
- Inside IPSec tunnels
- Operates in echo mode
- Automatically invoked at IPSec tunnel establishment
- Cannot be disabled
• Uses hello (up/down) interval, poll (app-aware)

WAN Edge WAN Edge interval and multiplier for detection
- Fully customizable per-WAN Edge, per-color
Path Quality and Liveliness Detection
Multiplier (n)
• Each WAN Edge router sends BFD hello
packets for path quality and liveliness
detection
- Packets echoed back by remote site
Hello Interval (ms) • Hello interval and multiplier determine how

Liveliness many BFD packets need to be lost to
Quality declare IPSec tunnel down
App-Route Multiplier (n)
• Number of hello intervals that fit inside poll
interval determines the number of BFD
Poll Interval Poll Interval Poll Interval (ms) packets considered for establishing poll
interval average path quality
• App-route multiplier determines number of
poll intervals for establishing overall
Hello Interval (ms) average path quality
Critical Applications SLA
 WAN Edge Routers vManage App Aware Routing Policy
App A path must have:
continuously perform path
Latency < 150ms
liveliness and quality Loss < 2%
measurements Jitter < 10ms
Internet
Remote Site
MPLS Regional
Path 2 Data Center
LTE
Path1: 10ms, 0% loss, 5ms jitter
Path2: 200ms, 3% loss, 10ms jitter
Path3: 140ms, 1% loss, 10ms jitter IPSec Tunnel
Transport Redundancy - Meshed
 WAN Edge routers are directly connected to all
the transports
- No need for L2 switches front-ending the WAN
MPLS Internet Edge routers
 When transport goes down, WAN Edge routers
detect the condition and bring down the tunnels
built across the failed transport
WAN Edge WAN Edge - BFD times out across tunnels
 Both WAN Edge routers still draw the traffic for
the prefixes available through the SD-WAN
fabric
 If one of the WAN Edge routers fails (dual
failure), second WAN Edge router takes over
forwarding the traffic in and out of site
- Both transport are still available
Transport Redundancy – TLOC Extension
 WAN Edge routers are connected only to their
respective transports
MPLS Internet  WAN Edge routers build IPSec tunnels across
directly connected transports and across the
transports connected to the neighboring WAN
Edge router
WAN Edge WAN Edge - Neighboring WAN Edge router acts as an
underlay router for tunnels initiated from the
other WAN Edge
 If one of the WAN Edge routers fails (dual

failure), second WAN Edge router takes over
forwarding the traffic in and out of site
- Only transport connected to the remaining WAN
Edge router can be used
Path and Remote-End Redundancy
 WAN Edge routers leverage BFD for
Data
Center
detecting tunnel liveliness
• If intermediate network path through the
SD-WAN fabric fails or if the remote-end
WAN Edge router (e.g. data center) fails,
MPLS Internet BFD hellos will time out and remote site
WAN Edge router will bring down its
relevant IPSec tunnels
• Traffic will be rerouted after the failed
condition had been detected
- BFD hello timer and multiplier can be
Remote tweaked for faster detection
Site
SD-WAN Demo
Summary of Convergence Techniques
Excellent Option
R2
R1 R4 SubOptimal Option
R3
Bad Option
Effectiveness of Various Techniques for Different Outage Types
Link Down Link Up Link Up Upstream Upstream
Neighbor Down Loss ~5% Blackhole Brownout
Routing
Protocols
BFD N/A1 N/A1
EOT2
RSR3 using
EOT (w/IP
SLA)
SD-WAN
1BFD Multihop support for Static and BGP routes

2Enhanced Object Tracking
3Reliable Static Routing BRKRST-2042 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Agenda
• Introduction
• MPLS Dual Carrier
• MPLS + Internet
Dual WAN (MPLS—Dual Carrier)
PE-CE Protocol: BGP 10.100.0.0/16 10.1.2.0/24
• Default behavior: 1-way 10.1.1.0/24
load sharing A-R1 MPLS - SP A A-R4
HQ- HQ-W1
• Load is shared from HQ to CORE1 192.168.101.8/29

BR-W1
Branch
192.168.201.8/29
HQ-W2
HQ- B-R1 MPLS - SP B B-R4

CORE2
EIGRP eBGP eBGP
HQ-CORE1#show ip route
D EX 10.1.2.0/24 [170/258816] via 10.1.1.110, 02:24:22, Vlan10
[170/258816] via 10.1.1.210, 02:24:22, Vlan10
• Only one link used Branch to

HQ BR-W1#show iproute
B 10.100.0.0/16 [20/0] via 192.168.101.9, 00:34:00
PE-CE Protocol: BGP Layer 3 Campus Locations
• IGP (EIGRP examples) 10.100.0.0/16 10.1.2.0/24
• Routes redistributed from BGP into 10.1.1.0/24
IGP (match & tag) A-R1 MPLS - SP A A-R4
• BGP routes are treated as IGP external HQ- HQ-W1

CORE1 192.168.101.8/29
BR-W1
• BGP 192.168.201.8/29
HQ-W2
• No iBGP required between HQ-W1 & HQ- MPLS - SP B B-R4

B-R1
HQ-W2 (CE routers) CORE2
• Routes redistributed from IGP into

BGP except those tagged as originally EIGRP eBGP eBGP
sourced from BGP
Mutual Route Redistribution Detail
Routes into EIGRP
HQ-W1#
router eigrp networkers
address-family ipv4 unicast autonomous-system 65110
topology base
10.1.1.0/2
redistribute bgp 65110 metric 45000 100 255 1 1500
4 address-family ipv6 unicast autonomous-system 65110
topology base
redistribute bgp 65110 metric 45000 100 255 1 1500
BR
HQ-W1
AS 65100 Routes into BGP
HQ-CORE1
eBGP HQ-W1#
EIGRP
iBGP
10.1.2.0/24
10.1.1.0/2
router bgp 65110
address-family ipv4
eBGP
redistribute eigrp 65110 route-map BLOCK-TAGGED-ROUTES
AS 65200 address-family ipv6
HQ-W2
BR redistribute eigrp 65110 route-map BLOCK-TAGGED-ROUTES
HQ-CORE2
!
BGP Redistribution to
route-map BLOCK-TAGGED-ROUTES deny 10
10.100.0.0/16
IGP automatically tags match tag 65100 65200
routes with neighbor route-map BLOCK-TAGGED-ROUTES permit 20
AS Number !
PE-CE Protocol: BGP Layer 2 Single Router Branch
10.100.0.0/16 10.1.2.0/24
• Is it possible to load share 10.1.1.0/24
from Branch to HQ? A-R1 MPLS - SP A A-R4
• BGP Multipath HQ-

CORE1
HQ-W1
192.168.101.8/29
• Allows installation of multiple 192.168.201.8/29

BR-W1
BGP paths to same destination HQ-W2
• Requirements (all must be equal) HQ-

CORE2
B-R1 MPLS - SP B B-R4
• Neighbor AS or
AS-PATH EIGRP eBGP eBGP
• Weight
BR-W1#show ip bgp
• Local Preference
• AS-PATH length Network Next Hop Metric LocPrf Weight Path
• Origin
* 10.100.0.0/16 192.168.201.9 0 65200 65200 ?
• Med
*> 192.168.101.9 0 65100 65100 ?
BR-W1#show ip route
B 10.100.0.0/16 [20/0] via 192.168.101.9, 00:34:00
PE-CE Protocol: BGP Layer 2 Single Router Branch
10.100.0.0/16 10.1.2.0/24
• Is it possible to load share from 10.1.1.0/24
Branch to HQ? A-R1 MPLS - SP A A-R4
• maximum-paths 2 HQ-
CORE1
HQ-W1
192.168.101.8/29
BR-W1
Requires hidden command:

192.168.201.8/29
• HQ-W2
• bgp bestpath as-path multipath- HQ-

CORE2
B-R1 MPLS - SP B B-R4
relax
EIGRP eBGP eBGP
router bgp 65110

bgp bestpath as-path multipath-relax
address-family ipv4
maximum-paths 2
address-family ipv6
maximum-paths 2
BR-W1#show ip route
B 10.100.0.0/16 [20/0] via 192.168.201.9, 00:03:44
[20/0] via 192.168.101.9, 00:03:44
Agenda
• Introduction
• MPLS Dual Carrier
• MPLS + Internet
DUAL WAN (MPLS + Internet)
PE-CE Protocol: BGP, Tunnel Protocol: EIGRP
• Headquarters WAN Edge
EIGRP BGP BGP
• W1 learns Branch route via eBGP 10.100.0.0/16 10.1.2.0/24
• W2 learns Branch route via EIGRP 10.1.1.0/24
Headquarters Core
A-R1 MPLS - SP A A-R4
•
HQ-CORE1 HQ-W1 BR-W1
• W1 redistributes eBGP into EIGRP, 192.168.101.8/29
EIGRP
HSRP
results in EIGRP external Internet
• W2 does not require redistribution, HQ-CORE2 HQ-W2

VPN Tunnel
BR-W2
results in EIGRP internal 10.0.1.0/29
Core1, Core2 install Branch route via

EIGRP
•
W2 HQ-W1#show ip route
B 10.1.2.0/24 [20/0] via 192.168.101.2, 05:24:01
HQ-W2#show ip route
HQ to Branch Traffic Flows D 10.1.2.0/24 [90/26882560] via 10.0.1.2, 00:00:04, Tunnel1
Across Tunnel HQ-CORE1#show ip route
D 10.1.2.0/24 [90/26882816] via 10.1.1.210, 00:02:32, Vlan10
• Single Router Branch WAN Edge
• W1 learns HQ route via eBGP and EIGRP Internal
• eBGP Administrative Distance preferred
EIGRP BGP BGP
10.100.0.0/16 10.1.2.0/24
10.1.1.0/24
A-R1 MPLS - SP A A-R4
HQ-W1
HQ-CORE1
192.168.101.8/2
9 BR-W1
Branch to HQ Traffic HQ-W2

Internet
VPN Tunnel
Flows Across MPLS

HQ-CORE2
10.0.1.0/29
EIGRP
BR-W1#show ip route
B 10.100.100.0/24 [20/0] via 192.168.101.9, 04:48:58
B 10.100.200.0/24 [20/0] via 192.168.101.9, 03:44:06
Dual Router Branch WAN Edge
EIGRP BGP BGP
• 10.100.0.0/16 10.1.2.0/24
10.1.1.0/24
• W1 learns HQ route via eBGP A-R1 MPLS - SP A A-R4
• W2 learns HQ route via EIGRP

HQ-W1 BR-W1
HQ-CORE1
192.168.101.8/2
9
HSRP
EIGRP
• No redistribution configured
Internet
HQ-W2 VPN Tunnel

BR-W2
HQ-CORE2
• HSRP Primary is on W1
10.0.1.0/29
EIGRP
BR-W1#show ip route
B 10.100.100.0/24 [20/0] via 192.168.101.9, 04:48:58
B 10.100.200.0/24 [20/0] via 192.168.101.9, 03:44:06
BR-W2#show ip route
Branch to HQ Traffic D 10.100.100.0/24 [90/26882816] via 10.0.1.1, 00:10:56, Tunnel1
D 10.100.200.0/24 [90/26882816] via 10.0.1.1, 00:10:57, Tunnel1
Flows Across MPLS BR-W1#show standby brief
|
Fa0/1 1 110 P Active local 10.1.2.220 10.1.2.1
• How to force HQ to Branch traffic across MPLS (primary)?
• Adjust administrative distance EIGRP BGP BGP
• For EIGRP routes learned via tunnel 10.100.0.0/16 10.1.2.0/24
• Ensure administrative distance is 10.1.1.0/24
higher than that of EIGRP external (170) A-R1 MPLS - SP A A-R4

HQ-W2#
router eigrp 65110
Only change is on hub
192.168.101.8/29
network 10.0.1.0 0.0.0.7
EIGRP
HSRP
distance 195 10.0.1.0 0.0.0.7 Internet
• Redistribute between two EIGRP Processes HQ-CORE2 HQ-W2

VPN Tunnel
BR-W2
Forcing External as done between BGP and 10.0.1.0/29
Campus EIGRP EIGRP

HQ-W2#
Router eigrp 65100
network 10.0.1.0 0.0.0.7 Requires additional changes
router eigrp 65110
redistribute eigrp 65100
or Proper Pre-Planning
HQ-W1#show ip route
Now: B 10.1.2.0/24 [20/0] via 192.168.101.2, 05:24:01
HQ to Branch Traffic Flows HQ-W2#show ip route
D EX 10.1.2.0/24 [170/261120] via 10.1.1.110, 00:07:25, GigE0/0
Across MPLS HQ-CORE1#show ip route
D EX 10.1.2.0/24 BRKRST-2042
[170/258816]
BRKRST-2042 via 10.1.1.110,
© 2019 Cisco 00:08:44,
and/or its affiliates. All rights reserved. Cisco Public Vlan10
87
MPLS Failure
• Failure within MPLS cloud EIGRP BGP BGP
• Dependent on provider 10.100.0.0/16 10.1.2.0/24
Worst Case
10.1.1.0/24
•
MPLS - SP A
Link up neighbor down

A-R1 A-R4
•
• Primary dependency BGP timers
192.168.101.8/29
EIGRP
HSRP
• End to end convergence time as Internet
long as BGP Holdtime HQ-CORE2 HQ-W2

VPN Tunnel
BR-W2
• Configuration options 10.0.1.0/29
• BFD for almost immediate notification EIGRP
• End-to-end Application Restoration as

HQ Route Tables
fast as SD-WAN detects
After Failure: HQ-W2#show ip route
D 10.1.2.0/24 [195/26882560] via 10.0.1.2, 00:06:46, Tunnel1
HQ to Branch Traffic
HQ-CORE1#show ip route
Flows Across Tunnel D 10.1.2.0/24 [90/26882816] via 10.1.1.210, 00:09:18, Vlan10
MPLS Failure EIGRP BGP BGP
Failure within MPLS cloud

10.100.0.0/16 10.1.2.0/24
• 10.1.1.0/24
• Suboptimal routing at Branch A-R1 MPLS - SP A A-R4
HSRP primary remains HQ-CORE1 HQ-W1 BR-W1

• 192.168.101.8/29
unchanged at BR-W1
EIGRP
HSRP
Internet
• Use EOT and move HSRP HQ-CORE2 HQ-W2

VPN Tunnel
BR-W2
primary to BR-W2 10.0.1.0/29
EIGRP
Branch Route Tables

BR-W1#show ip route
D 10.100.100.0/24
After Failure: [90/26885376] via 10.1.2.220, 00:22:42, FastEthernet0/1
Branch to HQ D 10.100.200.0/24
[90/26885376] via 10.1.2.220, 00:22:42, FastEthernet0/1
Traffic Flows
Across Tunnel BR-W2#show ip route
D 10.100.100.0/24 [90/26882816] via 10.0.1.1, 01:08:44, Tunnel1
D 10.100.200.0/24 [90/26882816] via 10.0.1.1, 01:08:45, Tunnel1
Agenda
• Introduction
• Final Wrap Up
• Key Takeaways
Key Takeaways
• Outages can manifest in many different ways. Network design should be
based on application requirements to survive various outages.
• Cisco IOS has inherent load sharing capabilities. Analyze your network
topology and use these to your advantage.
• End-to-end convergence time is a critical metric. Understand how
localized topology changes affect end-to-end resiliency.
• Multiple links/paths not only increase network reliability but can improve
application performance.
Key Takeaways
• IP SLA based monitoring can detect outage types that are virtually
undetectable by traditional “hello based” techniques.
• BFD is a lightweight tool for speeding convergence of all protocols.
• Cisco SD-WAN permits full utilization of available bandwidth and path
selection based on current real time characteristics.
• Most effective network designs incorporate a combination of convergence
techniques
• Cisco SD-WAN utilizes these features, while simplifying deployment and
management, and increasing application availability.
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKRST-2042
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations
Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com
Continue Your Education
Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings
Thank you

High Availability

Uploaded by

Copyright:

Available Formats

High Availability

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

High Availability

Uploaded by

Copyright:

Available Formats

BRKRST-2042

David Prall, Principal Systems Engineer

• How does outage manifest?

• System Availability: a ratio of the 98.000000% 7.3 Days

ISR ISR ISR

99.999% 99.999% 99.999%

p 10.0.0.0/8 is variably subnetted, 14 subnets, 5 masks

More Specific OSPF Override EIGRP

Note: BGP default value for maximum-paths = 1

1Not available in IOS-XE based images

The Traffic Share Count Is Critical to

• Now that we have load balancing

#show ip cef exact-route 1.1.1.1 2.2.2.2

Keepalive (B) Holdtime (B,E,I)

RIP/RIPv2 30 180 180 240

Note: Cisco Default Values

Recovery Times by Protocol

Link Down Link Up Link Up Link Up

BGP ~1s 180 180 Never

RIP/RIPv2 ~ 1s 240 240 Never

R4#show ip bgp vpnv4 vrf cisco neighbor

• Control Plane failure detection is very conservative

• Link-layer failure detection is not consistent across media types

• Provides a measure of consistency across routing protocols

Router#show track 100 Router#show track 103

* EIGRP, OSPF, BGP, Static Thresholds Are Scaled to Range of (0 – 255)

Track Options Syntax

Types of IP SLA Probes:

Track Options Syntax

track object-number list boolean {and|or}

BR-W1#show ip route track-table

BR-W1#show ip route track-table

IPv6 Reliable Static Routing added in 15.4(1)T

IPSLA with EEM icmp-echo 208.67.222.222 source-interface GigabitEthernet0/0

track 60 ip sla 110 reachability

(config)#ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 dhcp 10 ?

event manager applet DISABLE-STATIC-GIG0-0

track 60 ip sla 110 reachability

track 60 ip sla 100 reachability

http://www.cisco.com/go/cvd/wan VPN Remote Site over 3G/4G/LTE Technology Design Guide

• Gateway Load Balancing Protocol (GLBP)

Default Gateway: (.1)

BR-W2#show standby brief

Active Standby Active

#track 100 interface serial2/0 line-protocol

R1#show bfd neighbors details

Default Gateway: (.1)

standby bfd all-interfaces ! default

BR-W2#show glbp brief

Forwarder (AVF) There are 2 forwarders (2 active)

WAN Edge WAN Edge

• Uses hello (up/down) interval, poll (app-aware)

Hello Interval (ms) • Hello interval and multiplier determine how

 If one of the WAN Edge routers fails (dual

BFD N/A1 N/A1

1BFD Multihop support for Static and BGP routes

• Default behavior: 1-way 10.1.1.0/24

load sharing A-R1 MPLS - SP A A-R4

• Load is shared from HQ to CORE1 192.168.101.8/29