3.DGTL-TSCSPG-603 - EVPN Easy Tshoot
3.DGTL-TSCSPG-603 - EVPN Easy Tshoot
3.DGTL-TSCSPG-603 - EVPN Easy Tshoot
Cisco Live
EVPN Easy Troubleshoot
Diego Zorrilla, XR Tech Lead
DGTL-TSCSPG-603
#CiscoLive
Agenda
• EVPN Basic Recap.
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
EVPN Basic Recap.
EVPN Basic Recap.
• EVPN Instance (EVI) BGP
• Its our VPN PE
• Ethernet Segment
• Null
• Auto or Set PE
DF
• Topology CORE
MPLS / SR
• Single Home PE
• Dual Home A/S CE-1
CE-2
• Dual Home A/A
• BGP
• AFI = L2VPN (25) EVPN (70) Non-DF
• Route Type = 1 to 5
• Route Attributes = RD + ESI + MAC + Label + RT
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Troubleshooting EVPN
Minimum Requeriments for Troubleshooting
• Topology
PE PE
• Single Home Non-DF DF
• Dual Home Active / Standby
BE
• Dual Home Active / Active
• MACs + IPs
CE-1 BE DF Non-DF CE-2
• Source & Destination MAC-A MAC-B
IP-A IP-B
PE PE
• Flow Type
• Unicast
• BUM (Multicast / Broadcast / Unknown Unicast)
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Case Study 1
Broken Connectivity between hosts in same DC
Broken Connectivity between hosts in same DC
Problem Description CE-1
MAC A
VLAN 1000
• Service Provider A had an up and running EVPN
network.
CE-2
MAC B
VLAN 2000
PE-2
CE-3
MAC C
VLAN 3000
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Broken Connectivity between hosts in same DC
Troubleshooting CE-1
MAC A
VLAN 1000
• Are we learning the MACs in L2VPN BD?
AC and L2VPN config for PE-1 and PE-2 AC and L2VPN config for PE-3 BE1
! !
interface Bundle-Ether2.1000 l2transport interface Bundle-Ether1.2000 l2transport
encapsulation dot1q 1000 encapsulation dot1q 2000
CE-2
rewrite ingress tag pop 1 symmetric rewrite ingress tag pop 1 symmetric
MAC B
! !
VLAN 2000
interface Bundle-Ether2.3000 l2transport ! PE-2
encapsulation dot1q 3000 l2vpn
rewrite ingress tag pop 1 symmetric bridge group EVPN_BG_1234
! bridge-domain EVPN_BD_1234
l2vpn interface Bundle-Ether1.2000
bridge group EVPN_BG_1234 !!
bridge-domain EVPN_BD_1234 evi 1234
interface Bundle-Ether2.1000 !
!
CE-3
interface Bundle-Ether2.3000 MAC C
! VLAN 3000
evi 1234
!
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Case Study 1 : Broken Connectivity between hosts in same DC
Troubleshooting Outputs
CE-1
CE-2 MAC A
VLAN 1000
show arp vrf 2000
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.20.30.10 6 aaaa.aaaa.1000 ARPA Port-channel1.2000. <<<<< CE-1
Internet 10.20.30.30 6 cccc.cccc.3000 ARPA Port-channel1.2000. <<<<< CE-3
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Case Study 1 : Broken Connectivity between hosts in same DC
Troubleshooting Outputs
CE-1
CE-1 MAC A
show arp vrf 1000 VLAN 1000
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.20.30.10 - aaaa.aaaa.1000 ARPA Vlan1000
Internet 10.20.30.20 0 b0b0.b0b0.2000 ARPA Vlan1000
Internet 10.20.30.30 0 Incomplete ARPA <<<<<<<<<<< CE-3
ping vrf 1000 10.20.30.30 <<< Connectivity Failure Between CE-1 and CE-3 BE2
Sending 5, 100-byte ICMP Echos to 10.20.30.30, timeout is 2 seconds:
..... PE-3
XR_PE-1 BE1
show l2vpn forwarding bridge-domain EVPN_BG_1234:EVPN_BD_1234 mac-address location 0/0/cpu0
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Case Study 1 : Broken Connectivity between hosts in same DC
Troubleshooting Outputs
CE-1
show arp vrf 1000
Internet 10.20.30.30 0 Incomplete ARPA <<<<< CE-3
XR_PE-2
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Case Study 1 : Broken Connectivity between hosts in same DC
Troubleshooting Outputs
XR_PE-1 XR_PE-2
show evpn ethernet-segment interface bundle-Ether 2 carving detail show evpn ethernet-segment interface bundle-Ether 2 carving detail
192.168.0.2 192.168.0.2
Main port : Main port :
Interface name : Bundle-Ether2 Interface name : Bundle-Ether2
State : Up State : Up
Redundancy : Active Redundancy : Active
ESI type : 0 ESI type : 0
Value : 11.2211.2211.2211.2211 Value : 11.2211.2211.2211.2211
Topology : Topology :
Operational : MH, All-active Operational : MH, All-active
Service Carving : Manual Service Carving : Manual
Primary : 1 Primary : 1
Pri : 1234 Pri : 4000
Secondary : 1 Secondary : 1
Sec : 4000 Sec : 1234
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Broken Connectivity between hosts in same DC
ARP From
Root Cause CE-1 to CE-3
in VLAN 1000
• PE-2 will forward the traffic to PE-1 with Multicast Label and Split PE-1
Horizon Label. DF
PE-2
Non-DF
CE-3
MAC C
VLAN 3000
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Broken Connectivity between hosts in same DC
Solution CE-1
MAC A BE1
PE-1
Non-DF
VLAN 1000
1.- Having 2 Bundles between LAN and PEs.
In this case if BUM traffic from AC-1 in BE-X reaches Non-DF the
packet will be forwarded to PE-2 and it will forward to AC-2 since CE-3 PE-2
MAC C BE2
the SHL is for BE-X and not BE-Y. DF
VLAN 3000
CE-1
2.- Using an Active / Standby Solution. MAC A
PE-1
VLAN 1000
CE-3 PE-2
MAC C
VLAN 3000
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Case Study 2
Duplicate Multicast Traffic being Received
•
Duplicate Multicast Traffic being Received
Problem Description
• CU X had requirement to run Multicast in existing L2VPN,
and because of this, it enhances the network from Single
Multicast
Home to Dual Home. Traffic
CE-1
MAC A CE-2
VLAN 4000 MAC B
VLAN 4000
PE-2 PE-4
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Duplicate Multicast Traffic being Received
Troubleshooting
• Who is the Source?
Multicast
Traffic
• What PEs are part of the EVI?
• What are the labels for Multicast and Split Horizon Label
PE-1 PE-3
for those PEs in the EVI?
CE-1
MAC A CE-2
VLAN 4000 MAC B
VLAN 4000
PE-2 PE-4
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Case Study 2 : Duplicate Multicast Traffic being Received
Troubleshooting Outputs
PE-1 = 192.168.0.1 CE-1 XR_PE-2
PE-2 = 192.168.0.2 ping 232.11.22.33 source 192.168.40.2 repeat 3333 tim 0 show evpn evi vpn-id 4000 inclusive-multicast detail
! Sending 3333, 100-byte ICMP Echos to 232.11.22.33
interface Bundle-Ether2.4000 l2transport ...................................................................... VPN-ID Encap EtherTag Originating IP
...................................................................... ---------- ------ ---------- -------------------------
encapsulation dot1q 4000
4000 MPLS 0 192.168.0.1
rewrite ingress tag pop 1 symmetric Nexthop: 192.168.0.1
XR_PE-2
! show interface bundle-ether 2.4000 <<< AC to CE-1 Label : 24008
l2vpn Bundle-Ether2.4000 is up, line protocol is up Source : Remote
bridge group EVI_4000 3334 packets input, 393370 bytes
4000 MPLS 0 192.168.0.2
bridge-domain EVI_4000 Nexthop: ::
interface Bundle-Ether2.4000 show interface gig 0/0/0/0 <<< CORE Label : 24018
! GigabitEthernet0/0/0/1 is up, line protocol is up Source : Local
10020 packets output, 1375179 bytes,
evi 4000
4000 MPLS 0 192.168.0.8
! show mpls forwarding Nexthop: 192.168.0.8
evpn Local Outgoing Prefix Outgoing Next Hop Bytes Label : 24122
evi 4000 Label Label or ID Interface Switched Source : Remote
------ ----------- ------------------ ------------ ------------
control-word-disable 16001 16001 SR Pfx (idx 1) Gi0/0/0/0 20.20.20.5 383670 4000 MPLS 0 192.168.0.9
advertise-mac 16008 16008 SR Pfx (idx 8) Gi0/0/0/0 20.20.20.5 383670 Nexthop: 192.168.0.9
! 16009 16009 SR Pfx (idx 9) Gi0/0/0/0 20.20.20.5 383670
Label : 24120
interface Bundle-Ether2 Source : Remote
ethernet-segment
identifier type 0 11.22.11.22.11.22.11.22.11
!
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Case Study 2 : Duplicate Multicast Traffic being Received
Troubleshooting Outputs
P Router
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Case Study 2 : Duplicate Multicast Traffic being Received
Troubleshooting Outputs
XR_PE-1
GigabitEthernet0/0/0/1 <<< CORE Interface
3349 packets input, 454601 bytes, 1 total input drops
PE-1 PE-3
RSV_EGR_L2_EVPN_ESI_DROP 3336
XR_PE-4
GigabitEthernet0/0/0/0 <<< CORE Interface
3394 packets input, 444923 bytes
CE-2
6797 packets input, 795577 bytes <<< Duplicate Multicast Packets
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Case Study 2 : Duplicate Multicast Traffic being Received
Troubleshooting Outputs
PE-3 = 192.168.0.8 PE-3 PE-4
!
l2vpn show evpn ethernet-segment interface bundle-Ether 1 carving show evpn ethernet-segment interface bundle-Ether 1 carving
bridge group EVI_4000 detail detail
bridge-domain EVI_4000
Ethernet Segment Id Interface Nexthops Ethernet Segment Id Interface Nexthops
interface Bundle-Ether1.4000
! ------------------------ ---------------------------------- ------------------------ ----------------------------------
evi 4000 0088.9988.9988.9988.9988 BE1 192.168.0.8 0089.8989.8989.8989.8989 BE1 192.168.0.9
! Main port : Main port :
evpn Interface name : Bundle-Ether1 Interface name : Bundle-Ether1
! Interface MAC : bc5a.561d.8cdf Interface MAC : bc5a.561d.68de
interface Bundle-Ether1 ESI type :0 IfHandle : 0x08004024
ethernet-segment Value : 88.9988.9988.9988.9988 ESI type :0
identifier type 0 88.99.88.99.88.99.88.99.88 ES Import RT : 8899.8899.8899 (from ESI) Value : 89.8989.8989.8989.8989
! Topology : ES Import RT : 8989.8989.8989 (from ESI)
Operational : SH Topology :
Configured : All-active (AApF) (default) Operational : SH
Service Carving : Auto-selection Configured : All-active (AApF) (default)
PE-4 = 192.168.0.9 Peering Details : Service Carving : Auto-selection
!! 192.168.0.8 [MOD:P:00] Peering Details :
l2vpn Service Carving Results: 192.168.0.9 [MOD:P:00]
bridge group EVI_4000 Forwarders : 2 Service Carving Results:
bridge-domain EVI_4000 Elected :2 Forwarders : 2
interface Bundle-Ether1.4000 EVI E : 1000, 4000 Elected :2
! Local SHG label : 24022 EVI E : 1000, 4000
evi 4000 Local SHG label : 24017
!
evpn
!
interface Bundle-Ether1
ethernet-segment
identifier type 0 89.89.89.89.89.89.89.89.89
!
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Duplicate Multicast Traffic being Received
Root Cause and Solution
• ”Remote” PEs will recibe the multicast, and only Designated Forwarder (DF) will flood the Multicast.
• If two PEs working in Dual Home Active / Active do not have same ESI, both will be DF for the ESI and both will forward
multicast frames to the LAN, duplicated traffic will happen.
Multicast
Traffic
PE-3
PE-1 DF
CE-1
MAC A CE-2
VLAN 4000 MAC B
VLAN 4000
PE-2 PE-4
Non-DF
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Appendix
Swiss Army Knife
EVPN Basic Configuration for Single Home
router bgp 65500
!
address-family l2vpn evpn
BGP !
neighbor-group Route_Reflector
address-family l2vpn evpn
!
interface Bundle-Ether1.1000 l2transport
encapsulation dot1q 1000
AC rewrite ingress tag pop 1 symmetric
!
evpn
evi 1000
EVPN control-word-disable
advertise-mac
!
l2vpn
bridge group EVPN_BG
bridge-domain EVPN_BD
L2VPN interface Bundle-Ether1.1000
!
evi 1000
!
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
EVPN Basic Configuration for Dual Home A/A (1 of 2)
PE-1 PE-2
router bgp 65500 router bgp 65500
! !
address-family l2vpn evpn address-family l2vpn evpn
BGP ! !
neighbor-group Route_Reflector neighbor-group Route_Reflector
address-family l2vpn evpn address-family l2vpn evpn
! !
interface Bundle-Ether1.1000 l2transport interface Bundle-Ether1.1000 l2transport
encapsulation dot1q 1000 encapsulation dot1q 1000
AC rewrite ingress tag pop 1 symmetric rewrite ingress tag pop 1 symmetric
! !
evpn evpn
evi 1000 evi 1000
EVPN control-word-disable control-word-disable
advertise-mac advertise-mac
! !
l2vpn l2vpn
bridge group EVPN_BG bridge group EVPN_BG
bridge-domain EVPN_BD bridge-domain EVPN_BD
L2VPN interface Bundle-Ether1.1000 interface Bundle-Ether1.1000
! !
evi 1000 evi 1000
! !
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
EVPN Basic Configuration for Dual Home A/A (2 of 2)
PE-1 PE-2
redundancy redundancy
iccp iccp
group 12 group 12 <<< Same Values for P-1 and P-2
mlacp node 1 mlacp node 2 <<< Dif Values in P-1 and P-2
mlacp system mac 0111.0111.0111 mlacp system mac 0111.0111.0111 <<< Same Values for P-1 and P-2
mlacp system priority 1 mlacp system priority 1
mode singleton mode singleton
MLACP backbone backbone
interface xxxx interface xxxx <<< CORE Link(s)
! !
interface Bundle-Ether1 interface Bundle-Ether1
lacp switchover suppress-flaps 300 lacp switchover suppress-flaps 300
mlacp iccp-group 12 mlacp iccp-group 12
mlacp port-priority 1 mlacp port-priority 1
! !
evpn evpn
interface Bundle-Ether1 interface Bundle-Ether1
EVPN ethernet-segment ethernet-segment
identifier type 0 12.12.12.12.12.12.12.12.12 identifier type 0 12.12.12.12.12.12.12.12.12 <<< Same Values for P-1 and P-2
! !
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
EVPN Optional Configs.
Configuration Action Default
evpn
evi 1000
advertise-mac Advertise local mac addr in BGP Default is NOT to advertise.
unknown-unicast-suppression Avoid flooding of unknown unicast frames through EVPN. Default is to flood.
control-word-disable Add CW on packets. Default is CW enable.
load-balancing flow-label static Add Extra label to allow load balancing on core links. Default is no Flow Label.
EVPN !
interface bundle-ether1
ethernet-segment
load-balancing-mode single-active Only one PE handles the traffic, per EVI. Default is Active / Active.
service-carving manual Controls DF and Non-DF.
Manual load balancing per EVI Default is Even on PE-X and Odd on
primary 1-10 secondary 11-20
PE-Y
!
group 1234
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Swiss Army Knife
EVPN: BGP:
#CiscoLive DGTL-TSCSPG-603 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Thank you
#CiscoLive
#CiscoLive