Brkaci 2641
Brkaci 2641
Brkaci 2641
Endpoints
Andy Gossett, DCBU ACI Escalation
@agccie
Presented By Joseph Ristaino, DCBU ACI Escalation
@jristain
BRKACI-2641
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Acronyms/Definitions
Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure HAL Hardware Abstraction Layer
ACL Access Control List MDT Multicast Distribution Tree
➔ Reference Slide
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Endpoint Learning
What is an ACI Endpoint
Depends on who’s counting…
10.0.0.10 21.215.190.9
fvCEp
<epg-dn>/cep-00:00:00:00:0a
coop
fvIp
db Spine Two hardware
<epg-dn>/cep-00:00:00:00:0a/ip-[10.0.0.10]
mac: 00:00:00:00:0a entries
count: 1
ip0 : 10.0.0.10
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
What is an ACI Endpoint
Why the count matters
#Mac w/ one
#Mac + #IP
or more IPs
450K max
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Classical Learning
Encap + Interface => VLAN
VLAN => VRF
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC
L2 Forwarding:
L3 Forwarding (Longest Prefix Match)
(VLAN, DMAC) Miss => Flood
(VRF, DIP) Miss => Drop
(VLAN, DMAC) Gateway MAC => Route
(VRF, DIP) Hit=> Adjacency
(VLAN, DMAC) Hit => Destination Port
config on destination port + VLAN Might be Glean or packet rewrite (SMAC, DMAC,
determines egress encap VLAN, etc…), may include destination port in
(tagged or untagged) adjacency or require second L2 lookup on new DMAC
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
ARP Packet
Classical Learning DMAC
SMAC
LPM Routes
Eth: 0x0806
• Connected/direct routes manually Route Adj
configured 10.1.1.101/32 … Hdr/Opcode
• Static/dynamic routing protocols to 20.1.1.101/32
10.1.1.0/24 …
Glean Sender MAC
learn prefixes
20.1.1.0/24 Glean
Sender IP
Host Routes (IP Endpoints)
ARP Target MAC
• Glean adjacency for connected P
routes to punt frame and generate ARP Target IP
ARP request
• ARP/ND used to create MAC to IP
binding and install host route into 10.1.1.101/24 20.1.1.101/24
routing table
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
ACI Learning (Physical Local - PL)
Encap + Interface => EPG EPGs and L3
EPG => BD Learning
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC BD => VRF
ARP L3 Forwarding
L3 forwarding based on ARP target IP field
(VRF, ARP Target IP) Miss => Proxy
with miss sent to spine proxy ☺
(VRF, ARP Target IP) Hit=> Adjacency
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
ACI Learning (Virtual Local - VL) Fabric TEP
Host VTEP
Inner Header VXLAN Outer Header
Infra VLAN
Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID Rsvd DIP SIP 802.1Q SMAC DMAC
UDP
AVS/AVE/OVS
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
iVXLAN Header
OUTER INNER
MAC Header 802.1Q IPv4 Header UDP Header iVXLAN Header
VXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
D S D
Flags E Source Group
L P P
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
flags Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID DIP SIP 802.1Q SMAC DMAC
EPG UDP
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
ACI Learning
Learning Exceptions
• No IP EP learning if routing is
disabled on the BD
• No IP EP learning on external BD’s
(Layer-3 Outside interfaces)
• No IP EP learning on Infra VLAN
VXLAN/Opflex traffic
VXLAN Tunnel
• No IP learning of shared service
prefixes outside of our VRF between host and
fabric on Infra VLAN
LPM Routes (Same as Classical)
• Pervasive SVI Routes (BD Subnets) Static/Dynamic WAN/
Routing on L3Out Internet
• Static and dynamic routing protocols
on L3Out AVS/AVE/OVS
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
ACI Learning Leaf Endpoint Database
Remote IP Entries Endpoint Entry
VRF (VRF, IP) - EPG (pcTag)
Frame Forwarding Learn - Interface/Tunnel
Operation - Control flags
Non- Bridged MAC
IP/IP BD
Remote MAC Entries
(VRF, BD, MAC)
ARP - MAC (sender-HW),
IP (sender-IP)
IPv4 Unicast MAC, IP Local MAC and IP Entries
Routed Encap (VRF, BD, VLAN/VXLAN, MAC)
(VRF, BD, VLAN/VXLAN, IP)
IPv6 Unicast MAC, IP
Routed
IP
IPv6 Neighbor MAC, IP IP
Mac Entry
Discovery IP
Entry
Entry IP
Entry
Relationship to Entry
multiple IPs
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
ACI Learning (COOP and EP Sync)
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
ACI Learning: Review
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Spines
Moves and Bounce Addr Interface Detail
A tun1001 leaf101/102 vTEP
B tun4 leaf104 TEP
Leaf101/102
Addr Interface Detail
A vpc1 local vpc
Leaf 103
Addr Interface Detail
A B
- - -
- - -
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Spines
Moves and Bounce Addr Interface Detail
A tun1001
tun3 leaf101/102
leaf103 TEP vTEP
Spines receive
3 event and updates B tun4 leaf104 TEP
leaf101/102 Leaf101/102
Addr Interface Detail
Detail
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Spines
Moves and Bounce Addr Interface Detail
A tun3 leaf103 TEP
leaf101/102
2 bounce to leaf103
B tun4 leaf104 TEP
Leaf101/102
Addr Interface Detail
A tun3, XR -> leaf103 with
bounce bounce bit set
leaf101 leaf102 leaf103 leaf104 B tun4 XR -> leaf104
Leaf 103
leaf103 learns
Addr Interface
Interface 3
Detail
Detail host B to leaf104
A B
A eth1/1
eth1/1 local
locallearn
learn
-
B tun4
- XR
- -> leaf104
host B sends
1 packet to host A Leaf 104
Addr Interface Detail
A tun1001 XR -> leaf101/102 VIP
B eth1/1 local learn
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Spines
Moves and Bounce Addr Interface Detail
A tun3 leaf103 TEP
B tun4 leaf104 TEP
Leaf101/102
Addr Interface Detail
A tun3, XR -> leaf103 with
bounce bounce bit set
leaf101 leaf102 leaf103 leaf104 B tun4 XR -> leaf104
Leaf 103
Addr Interface Detail
A B
A eth1/1 local learn
B tun4 XR -> leaf104
host A sends
4 packet to host B Leaf 104
Addr Interface Detail
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Aging Addr Time-left Reset-count Hit
A 15 second
900 second 225
224 Yes
No
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
VPC Aging
B No peer-attached B No local
A B
• For vpc, both leaves in the vpc domain have to age out the entry before it
is removed. This applies to remote and local entries
• For orphan ports, as soon as the local leaf ages it out it is deleted from
both switches.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
VPC Aging
Peer-aged flag set indicating that peer When vpc endpoint is aged,
2 has aged the entry. Will be deleted 1 set local-aged flag and send
once local leaf ages out it as well. update to peer
vpc host
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
VPC Aging
vpc host
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
VPC Aging
B No peer-attached B No local
A B local-aged
Orphan host
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Configuration Options
Nerd Knobs
Timers – Endpoint Retention Policy
Move 256/sec - -
• If moves/sec exceed rate then learning is disabled on BD for the hold time
as a protection mechanism for software components (epm/epmc/coop)
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Timers – Endpoint Retention Policy
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Timers – Endpoint Retention Policy
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Issue #1
Switch independent NIC team and load/spreading (Misconfigured Host)
ARP on eth2-1
1 with mac A, IP C
eth2-1 eth2-2
mac: A mac: B
• Each routed IP frame triggers a new IP learn within the fabric and endpoint
is rapidly moving between mac A and mac B
• Possibly no perceived impact on dataplane traffic, however possible high
CPU on leaf. If NIC is between two leaves, then may see coop process
high on spine as well.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Issue #1 Available in 3.2(1)
Fix – Enable Rogue Endpoint Detection
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Issue #1
Fix – Enable Rogue Endpoint Detection
Example Fault
• Fault is raised under the
node and also be seen
under System faults.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Issue #1
Fix – Enable Rogue Endpoint Detection
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Issue #1
What about EP Loop Protection? Not RECOMMENDED
• Action is potentially
disruptive to other stable
endpoints.
• BD Learn disable prevents
new learns on the entire
BD
• Port disable may impact a
critical port such as fabric-
interconnect or DCI link.
No mechanism to prioritize
a host port.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Issue #2
Old IP never times out after new IP is assigned to host
System -> System Settings -> Endpoint Controls -> • For aging, an endpoint is a
IP Aging
MAC with one or more IP
addresses. If the MAC is
active then all IPs learned
on the MAC will remain
active.
• IP Aging policy performs
aging on each IP
individually
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Issue #3
Misconfigured host/L4-L7 service triggers unexpected learn
Border Leaf (BL)
L3 Addr Interface Detail
Out A tun1 XR -> Service Leaf
IP: X
B tun1 XR -> Service Leaf
C eth1/1 local learn
service border
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Issue #3
Misconfigured host/L4-L7 service triggers unexpected learn
Host-A sends pkt Border Leaf (BL)
1 with source-IP X L3 Addr Interface Detail
Out A tun1 XR -> Service Leaf
dmac Triggers a learn
XR 3
IP: X
smac
B tun1 -> Service Leaf leaf
on border
C eth1/1 local learn
SIP-X service border
X tun1 XR -> Service Leaf
DIP-C
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Issue #3
Misconfigured host/L4-L7 service triggers unexpected learn
Packet incorrectly sent Border Leaf (BL)
3 to SL instead of L3Out L3 Addr Interface Detail
Out A tun1 XR -> Service Leaf
IP: X
B tun1 BLService
has learned IP
2XR -> Leaf
X toward SL
dmac C eth1/1 local learn
service border
smac X tun1 XR -> Service Leaf
SIP-C
A C Service Leaf (SL)
B DIP-X Addr Interface Detail
A eth1/1 local learn
Same problem if Host-B Host-C sends pktB eth1/2 local learn
tries to send packet to IP X. 1 with source-IP XC tun6 XR -> Border Leaf
All connectivity to this IP is
broken X eth1/1 local learn
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Issue #3 Available in 1.1(1)
Fix: Limit IP Learning to Subnet
Tenant -> Networking -> Bridge Domain
• Default setting for new
BDs created in 2.3(1e)
and 3.0(1k) and
above.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Issue #3
Fix: Limit IP Learning to Subnet (Partial Fix)
Local off-subnet Border Leaf (BL)
Packet is still
1 2
learn is ignored L3forwarded toAddrBL Interface Detail
Out A tun1 XR -> Service Leaf
dmac Triggers a learn
XR 3
IP: X
smac
B tun1 -> Service Leaf leaf
on border
C eth1/1 local learn
SIP-X service border
X tun1 XR -> Service Leaf
DIP-C
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Available in
Issue #3 2.2(2) and 3.0(2)
Fix: Enforce Subnet Check
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Available in
Issue #3 2.2(2) and 3.0(2)
Fix: Enforce Subnet Check
Local off-subnet
1 learn is ignored L3 • This feature is available only for Gen2
dmac
Out switches and above
IP: X
smac • This implicitly enables local subnet
SIP-X service border check whether it is enabled or not
XR off-subnet for all
DIP-C 2 BDs in VRF is ignored
enabled on the BD (i.e., Limit Ip
A B C Learning to Subnet on the BD is no
longer required).
• For remote learns, the IP is only
learned if the IP belongs to at least
BD in the VRF.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Issue #4 Leaf101
Addr Interface Detail
Stale Endpoint on Border Leaf
A tun3, XR -> leaf103 with
Traffic from L3out destined to Host-A bounce bounce bit set
is bounced through leaf101
L3
Out
Leaf 103
Addr Interface Detail
A eth1/1 local learn
leaf101 leaf103 border
Border Leaf
Addr Interface Detail
A A B A tun1 XR -> leaf101 TEP
• In initial state, Host-A has triggered an XR learn on the border leaf. Let’s
assume in this example that Host-A was communicating with Host-B.
• Host-A then moves to leaf103. It no longer sends any frames to Host-B but
continues sending frames out the L3out toward the border leaf.
• Leaf101 maintains a bounce-entry for Host-A
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Issue #4 Leaf101
Addr Interface Detail
Stale Endpoint on Border Leaf
A -
tun3, Bounce entry timed
XR -> leaf103 with out
bounce bounce bit set
Eventually bounce
L3
Leaf 103 entry times out
Out
Addr Interface Detail
A eth1/1 local learn
HIT bit set, but move
leaf101 leaf103 border
Border Leaf ignored due to DL bit
Addr Interface Detail Hit
A A tun1 XR -> leaf101 TEP No
Yes
• Leaf103 is a Gen1 leaf and the VRF is in ingress enforcement. Due to hardware
restriction on Gen1, traffic sent to the L3Out has the DL (don’t-learn) bit set in the
iVXLAN header.
• When the border leaf receives the frame, it updates aging hit bit but does not update
the learn entry since DL bit is set.
• Eventually, the bounce entry on leaf101 will timeout but border leaf will still have XR
entry point to leaf-101. Any traffic destined to host-A will be dropped
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Issue #4 Leaf101
Stale Endpoint on Border Leaf Addr Interface Detail
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Available in
Issue #4 2.2(2) and 3.0(1)
Fix: Disable Remote Endpoint Learning on Border Leaf
System -> System Settings -> Fabric Wide Settings
• Let’s consider the same scenario as Issue#4. Host-A moved from leaf101 to
leaf103, a bounce entry is present on Host-A, and some flow is resetting the
XR hit-bit on the border leaf toward leaf101
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Stale Endpoint Software Fix
Feature: EP Announce on Bounce Delete
Leaf101
Addr Interface Detail
L3
Out A tun3
- XR -> leaf103
Bounce entry timed-out
Border Leaf
leaf101 leaf103 border Addr Interface Detail
Interface Detail
A -
tun1 XRDeleted by announce
-> leaf101 TEP
Bounce timer expires, A
Send EP Announce Delete Triggers XR delete on any
leaf still pointing to leaf101
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Issue #5
I have no control over the devices connected to the network…
• Some environments must support
VM with multiple NICs that
perform their own routing OR
allow users to spin up their own
Users routing
virtual routers, load-balancers, or
through their own firewalls
virtual firewalls
• There are supported design
recommendations to address
each scenario, however it is too
Servers IP
load-sharing
difficult or not possible to address
Dynamic load-
Virtual
balancers
each in the current network
routers
• Can we just do traditional IP
learning?
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Issue #5 Available in 4.0(1)
Fix: Disable IP Dataplane Learning on the VRF
Tenant -> Networking -> VRFs • Local MAC learning still occurs via
dataplane
• Remote MAC learning still occurs
via dataplane for Gen2
• BD L2 hardware proxy is required
to support Gen1 since remote MAC
learning will not occur
• IP Aging must be enabled
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Endpoint Control Best Practices
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Endpoint Learning
Troubleshooting Tips
Packet Walk Checklist Always Check Faults!
Problem: Host-A cannot ping the gateway
• Start with the basics:
❑ Verify EPG/BD/VRF basic config
❑ What leaf/port is the host connected?
❑ Is the vlan-encap deployed to the leaf?
❑ Is the port a member of the vlan?
❑ Is the SVI present with gateway config?
A ❑ Is the endpoint learned?
10.1.1.101
0000.0000.000A
If we were learning the endpoint in the
EPG: e1
fabric, we could quickly tell which leaf/port
BD: bd1 it was connected and, most likely, it would
VRF: v1 be able to ping its gateway…
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Packet Walk Checklist ❑ Is the endpoint learned?
Problem: Host-A cannot ping the gateway Skip to the last step first, since it
can validate all other steps
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Packet Walk Checklist
Problem: Host-A cannot ping the gateway
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Packet Walk Checklist
Problem: Host-A cannot ping the gateway
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk Checklist ❑ Is the vlan-encap deployed?
Problem: Host-A cannot ping the gateway ❑ Is the port a member of the vlan?
fab4-leaf101# vsh_lc -c 'show system internal eltmc info vlan access_encap_vlan 101' | egrep "vlan_id"
vlan_id: 3028 ::: hw_vlan_id: 3009
vlan_id: 3028 ::: isEpg: 1
bd_vlan_id: 3027 ::: hwEpgId: 12766
Get the PI vlan for the encap
fab4-leaf101# show vlan id 3028 extended
(FD) and the BD vlans
VLAN Name Encap Ports
---- -------------------------------- ---------------- ------------------------
3028 ag:app:e1 vlan-101 Eth1/3, Eth1/4, Eth1/6,
Po3, Po4 Verify my interface is
fab4-leaf101# show vlan id 3027 extended
forwarding for both EPG
VLAN Name Encap Ports and BD vlans
---- -------------------------------- ---------------- ------------------------
3027 ag:bd1 vxlan-15958069 Eth1/3, Eth1/4, Eth1/6,
Po3, Po4
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Packet Walk Checklist ❑ Is the SVI present with gateway
config?
Problem: Host-A cannot ping the gateway
❑ Is the endpoint learned?
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Packet Walk Checklist ❑ Is the endpoint learned?
❑ Is the correct subnet pushed?
Problem: Host-A cannot ping the gateway
❑ Is learning enabled?
fab4-leaf101# show system internal epm vlan 3027 detail | egrep "Learn|fwd_mode|BD Subnet"
Valid : Yes ::: Incomplete : No ::: Learn Enable : Yes
fwd_mode : route,bridge ::: fwd_ctrl : mdst-flood,ip-lrn-pfx-check,
BD Subnet ip_pfx-1 : 10.1.1.1/24
fab4-leaf101# vsh_lc -c 'show system internal epmc vlan 3027 detail' | egrep "Learn|fwd_mode|BD Subnet"
fwd_mode : route,bridge ::: fwd_ctrl : mdst-flood,ip-lrn-pfx-check, ::: bridge_mode: mac ::: unk_mac_ucast:
proxy
Learning disabled :no
BD Subnet ip_pfx-1 : 10.1.1.1/24 Both epm (sup component) and epmc (LC
component) have routing enabled on the BD
and learning is enabled.
Gen2 only, ensure that learning Also BD subnet list contains our prefix
is globally enabled in Hal
fab4-leaf101# vsh_lc -c 'show system internal epmc global-info' | egrep "Hal Learn"
Hal Learn Disabled : No
fab4-leaf101# vsh_lc -c 'show platform internal hal learn learn' | egrep status
status : Enabled
status_reason : None
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Packet Walk Checklist ❑ Is the endpoint learned?
❑ Is the correct subnet pushed?
Problem: Host-A cannot ping the gateway
❑ Is learning enabled?
Move 256/sec - -
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Packet Walk Checklist ❑ Is the endpoint learned?
❑ Are we receiving the frame?
Problem: Host-A cannot ping the gateway
What tools do we have to help?
SPAN, ELAM (ELAM-Assistant App)
• We got lucky that the vlan-encap the host was sending in was configured on
the leaf, else the frame would have been dropped and no MAC learn
triggered Limit IP Learning to Subnet enabled by
default, vlan-102 in a different BD or
• Why wasn’t the IP learned? unicast routing disabled on that BD
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Packet Walk Checklist ❑ Is the endpoint learned?
Fixed: Host-A can ping the gateway
Fixed the host config and now
we’re learning the IP!
fab4-leaf101# show endpoint ip 10.1.1.101
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged L - local
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
3028 vlan-101 0000.0000.000a LV po3
ag:v1 vlan-101 10.1.1.101 LV po3
IP address : 10.1.1.101
Vrf : 2555909 Verify endpoint in coop using
Flags : 0 VRF vnid and IP address
EP bd vnid : 15958069
EP mac : 00:00:00:00:00:0A
Publisher Id : 10.0.128.93 Mac and BD VNID
Record timestamp : 06 09 2019 13:32:53 827717825
Publish timestamp : 06 09 2019 13:32:53 828777370
Seq No: 0
Remote publish timestamp: 12 31 1969 19:00:00 0
URIB Tunnel Info
Num tunnels : 1
Tunnel address : 10.0.128.95
Tunnel ref count : 1:::: pTEP/vTEP/eTEP of leaf/pod/site
• Endpoint must be in coop in order for proxy lookups to work. This is critical
for XR miss for both intra/inter-pod and intra/inter-site. You should see the
same state on all spines.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Packet Walk Checklist ❑ Does coop have the endpoint?
Fixed: Host-A can ping the gateway
Bonus validation
fab4-spine201# show coop internal info repo ep key 15958069 00:00:00:00:00:0A | egrep "^Vrf|^Tunnel nh|^EP|num
of active|^Real"
EP bd vnid : 15958069
EP mac : 00:00:00:00:00:0A Verify endpoint is in coop using
Vrf vnid : 2555909 Tunnel next-hop BD VNID and mac address
Tunnel nh : 10.0.128.95
num of active ipv4 addresses : 4
num of active ipv6 addresses : 1
Real IPv4 EP : 10.1.1.101 IPv4/IPv6 addressed
Real IPv4 EP : 10.1.1.102
Real IPv4 EP : 10.1.1.103 tied to this MAC
Real IPv4 EP : 10.1.1.104
Real IPv6 EP : 2001:0000:0000:0000:0000:0000:0000:0065
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Endpoint Learning Troubleshooting Review
❑ Verify logical config (EPG/BD/VRF and contracts)
❑ Verify no network faults under the EPG that would prevent the encap from being
deployed
❑ Verify that the leaf has the encap deployed
❑ Verify that the port is a member of the vlan
❑ Verify that the SVI is present on the leaf with the proper subnets
❑ Verify that local leaf is learning the endpoint
❑ Verify learning is enabled on the BD
❑ Verify software components have the correct BD prefixes programmed
❑ Verify the leaf is receiving the frame on expected interface and encapsulation
❑ Verify that endpoint is present in coop and coop has correct tunnel address
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Recommend Troubleshooting Apps
https://aciappcenter.cisco.com/
The EnhancedEndpointTracker is a Cisco ACI ELAM is a built-in tool that captures a single packet at
application that maintains a database of endpoint the ASIC level to check forwarding decision details.
events on a per-node basis allowing for unique fabric- It is typically used by Cisco TAC as it requires a deep
wide analysis. The application can be knowledge of each ACI ASIC to both perform and
configured to analyze, notify, and automatically correctly understand the resulting output.
remediate various endpoint events. This gives
ACI fabric operators better visibility and control over This app wraps the differences between each ACI ASIC
the endpoints in the fabric. and provides a UI to perform an ELAM capture for
those who don't have access to ASIC level information.
It then decodes this results of the ELAM capture in a
user friendly format.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Enhanced Endpoint Tracker
Active endpoint count
and fast search
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Enhanced Endpoint Tracker
Fast search for IP or MAC
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Enhanced Endpoint Tracker
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Enhanced Endpoint Tracker
Full details of current state of endpoint within
the fabric including local and XR learns
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Enhanced Endpoint Tracker
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Continue your education
Demos in the
Walk-in labs
Cisco campus
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Thank you
Appendix
Packet Walk Checklist Subtle but important. If bridged then
we need to check MAC endpoints, if
Problem: Host-A cannot ping Host-B routed we need to check IP…
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Packet Walk Checklist ❑ Am I learning Host-A and Host-B
IPs in the fabric?
Problem: Host-A cannot ping Host-B
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Packet Walk Checklist ❑ Do we have a remote learn for
Host-B on ingress leaf or are we
Problem: Host-A cannot ping Host-B using proxy-path?
fab4-leaf101# show endpoint ip 10.1.2.102
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged L - local
Leaf-101 (ingress leaf) does not
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLANhave an XR IPlearn for Host-B IP Info
Address
+-----------------------------------+---------------+-----------------+--------------+-------------+
<none>
fab4-leaf101# show isis dteps vrf overlay-1 | grep 10.0.208.64 Next-hop IP is spine anycast
10.0.208.64 SPINE N/A PHYSICAL,PROXY-ACAST-V4 IPv4 Proxy
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Packet Walk Checklist ❑ Do the spines have Host-B entry
programmed to handle proxy?
Problem: Host-A cannot ping Host-B
We can get it vrf vnid First, we need the VNID for the
from the leaf VRF to validate routed flow.
fab4-spine201# show coop internal info ip-db key 2555909 10.1.2.102 The tunnel address can by
IP address : 10.1.2.102
one of several different
Vrf : 2555909 type of TEPs:
Flags : 0x2 Spine has the entry in coop
EP bd vnid : 16187409
EP mac : 00:00:00:00:00:0B
(should validate each spine) • Physical TEP within same
Publisher Id : 10.4.0.2 pod
Record timestamp : 12 31 1969 19:00:00 0
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Packet Walk Checklist ❑ For the leaf that is performing
policy enforcement, do I have the
Problem: Host-A cannot ping Host-B appropriate contract?
❑ VRF VNID
❑ Source EPG pcTag (Host-A)
❑ Destination EPG pcTag (Host-B)
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Packet Walk Checklist ❑ For the leaf that is performing
policy enforcement, do I have the
Problem: Host-A cannot ping Host-B appropriate contract?
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Packet Walk Checklist ❑ For the leaf that is performing
policy enforcement, do I have the
Problem: Host-A cannot ping Host-B appropriate contract?
fab4-leaf103# show zoning-rule scope 2555909
Rule ID SrcEPG DstEPG FilterID operSt Scope Action
======= ====== ====== ======== ====== ===== ======
4419 0 0 implicit enabled 2555909 deny,log
4420 0 0 implarp enabled 2555909 permit
4421 0 15 implicit enabled 2555909 deny,log
4535 0 49154 implicit enabled 2555909 permit
fab4-leaf103# show logging ip access-list internal packet-log deny | egrep 10.1.2.102 | head
[ Fri May 17 04:02:02 2019 634490 usecs]: CName: ag:v1(VXLAN: 2555909), VlanType: Unknown, Vlan-Id: 0, SMac:
0x000c0c0c0c0c, DMac:0x000c0c0c0c0c, SIP: 10.1.1.101, DIP: 10.1.2.102, SPort: 0, DPort: 0, Src Intf: Tunnel14,
Proto: 1, PktLen: 98
<snip>
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Packet Walk Checklist ❑ For the leaf that is performing
policy enforcement, do I have the
Problem: Host-A cannot ping Host-B appropriate contract?
In this instance the contract was missing. Add the proper consumer/provider and/or
VzAny/preferred group updates to allow communication between the two EPGs
fab4-leaf101# show zoning-rule scope 2555909 | egrep "Rule|===|16389"
Rule ID SrcEPG DstEPG FilterID operSt Scope Action
======= ====== ====== ======== ====== ===== ======
4735 49155 16389 7 enabled 2555909 permit
4700
4736
49155
16389
16389
49155
default
default
Traffic from
enabled
enabled
Host-A (pcTag
2555909
2555909
permit
permit
6137 16389 49155 6 49155)
enabled to Host-B (pcTag 16389)
2555909 permit
BRKACI-2641 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Thank you