ACI 5.

0 release
PBR Source MAC rewrite
Minako Higuchi
Technical Marketing Engineer, INSBU
Last update: 03/09/2020
Overview
• ACI PBR rewrites destination MAC to make traffic go to a service
device but doesn’t change source MAC. Thus the service node sees
the source MAC address of the source endpoint, which could cause a
problem if the service node uses “source MAC based forwarding”
instead of IP based forwarding.
• This is the feature enhancement to provide an option to change
source MAC to service BD SVI MAC.
• Consideration: When enabling source MAC rewrite feature, it requires that
service BD uses the reserved default BD MAC.

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Source MAC based forwarding
• This is typically used to avoid asymmetric path or to simplify routing
configuration in the service device as it doesn’t have to rely on
routing.
• Each vendor has different terminology for the feature
• F5: Auto Last Hop
• Citrix: MAC-Based Forwarding (MBF)

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Design and configuration considerations
• When enabling source MAC rewrite feature, it requires that service
BD uses the reserved default BD MAC (00:22:bd:f8:19:ff). Custom
MAC can’t be used in the service BD.
• APIC has validation to make sure the service BD using default BD MAC. If
not, service graph rendering is failed and APIC raises a fault.
• This feature is supported on 2nd gen leaf onward, not on 1st gen.
• Source MAC rewrite option is at PBR policy level. If same PBR
destination IP is used in different PBR policies with inconsistent
source MAC rewrite configuration: one with disabled and the other
with enabled, Source MAC rewrite is enabled.
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
 Default setting

Traffic flow example without

source MAC rewrite

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
 In this case, source MAC rewrite is NOT enabled

Traffic flow (incoming) 2: policy applied (PBR)

Src IP: 192.168.1.1 5: Routing to provider ToR
1: traffic from Web to App Src MAC: Web-MAC DL (Don’t Learn Bit): 1
Src class: Web Dest IP: 192.168.1.2
Dest Class: App Dest MAC: PBR-MAC
Segment ID: Service-BD 6: traffic to App
Src IP: 192.168.1.1 Src IP: 192.168.1.1
Src MAC: Web-MAC Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.2 Dest IP: 192.168.1.2
Dest MAC: App-MAC Dest MAC: App-MAC

BD1 (192.168.1.254/24) Service-BD BD1 (192.168.1.254/24)

192.168.100.254/24
4: traffic comes back
Src class: PBR-node
Dest Class: App
EPG EPG
PBR device Src IP: 192.168.1.1
Web App
Src MAC: PBR-MAC
Dest IP: 192.168.1.2
192.168.1.1/24 192.168.100.100/24 Dest MAC: Leaf BD-MAC 192.168.1.2/24
MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC
3: traffic to Service node

Src IP: 192.168.1.1 *Source MAC is NOT rewritten

Src MAC: Web-MAC
Dest IP: 192.168.1.2
Dest MAC: PBR-MAC

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
VRF1
 In this case, source MAC rewrite is NOT enabled

Traffic flow (return traffic)

2: policy applied (PBR)
5: Routing to consumer ToR Src IP: 192.168..1.2
DL (Don’t Learn Bit): 1 Src MAC: App-MAC
Dest IP: 192.168.1.1 1: traffic from App to Web
6: traffic to Web Dest MAC: PBR-MAC Src class: App
Src IP: 192.168.1.2 Segment ID: Service-BD Dest Class: Web
Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.1 Src IP: 192.168.1.2
Dest MAC: Web-MAC Src MAC: App-MAC
Dest IP: 192.168.1.1
Dest MAC: Web-MAC

BD1 (192.168.1.254/24) Service-BD BD1 (192.168.1.254/24)

192.168.100.254/24

3: traffic to Service node

4: traffic comes back
EPG Src class: PBR-node Src IP: 192.168.1.2 EPG
PBR device
Web Dest Class: Web Src MAC: Leaf BD-MAC App
Dest IP: 192.168.1.1
192.168.1.1/24 Src IP: 192.168.1.2 192.168.100.100/24 Dest MAC: PBR-MAC 192.168.1.2/24
MAC: Web-MAC Src MAC: PBR-MAC MAC: App-MAC
MAC: PBR-MAC
Dest IP: 192.168.1.1
Dest MAC: Leaf BD-MAC
If it’s forwarding based on routing information on the PBR device,
destination MAC is Leaf BD-MAC, assuming the PBR device has route
to 192.168.1.0/24 via 192.168.100.254.
If source MAC based forwarding is enabled on the PBR device
destination MAC is Web-MAC and the leaf drops traffic.
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
VRF1
 New option

Traffic flow example with

source MAC rewrite
(one-arm design)

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Topology

BD1 (192.168.1.254/24) Service-BD BD1 (192.168.1.254/24)

192.168.100.254/24

EPG EPG
PBR device
Web App

192.168.1.1/24 192.168.100.100/24 192.168.1.2/24

MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
VRF1
 In this case, source MAC rewrite is enabled

Traffic flow (incoming) 2: policy applied (PBR)

5: Routing to provider ToR
DL (Don’t Learn Bit): 1
Src IP: 192.168.1.1
1: traffic from Web to App Src MAC: Leaf BD-MAC
Src class: Web Dest IP: 192.168.1.2
Dest Class: App Dest MAC: PBR-MAC
Segment ID: Service-BD 6: traffic to App
Src IP: 192.168.1.1 Src IP: 192.168.1.1
Src MAC: Web-MAC Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.2 Dest IP: 192.168.1.2
Dest MAC: App-MAC Dest MAC: App-MAC

BD1 (192.168.1.254/24) Service-BD BD1 (192.168.1.254/24)

192.168.100.254/24
4: traffic comes back
Src class: PBR-node
Dest Class: App
EPG EPG
PBR device Src IP: 192.168.1.1
Web App
Src MAC: PBR-MAC
Dest IP: 192.168.1.2
192.168.1.1/24 192.168.100.100/24 Dest MAC: Leaf BD-MAC 192.168.1.2/24
MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC
3: traffic to Service node

Src IP: 192.168.1.1 *Source MAC for the flow is Leaf BD-MAC
Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.2
Dest MAC: PBR-MAC

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
VRF1
Traffic flow (return traffic) Source MAC rewrite is not
mandatory for this direction
in this example.
2: policy applied (PBR)
5: Routing to consumer ToR Src IP: 192.168..1.2
DL (Don’t Learn Bit): 1 Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.1 1: traffic from App to Web
6: traffic to Web Dest MAC: PBR-MAC Src class: App
Src IP: 192.168.1.2 Segment ID: Service-BD Dest Class: Web
Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.1 Src IP: 192.168.1.2
Dest MAC: Web-MAC Src MAC: App-MAC
Dest IP: 192.168.1.1
Dest MAC: Web-MAC

BD1 (192.168.1.254/24) Service-BD BD1 (192.168.1.254/24)

192.168.100.254/24

3: traffic to Service node

4: traffic comes back
EPG Src class: PBR-node Src IP: 192.168.1.2 EPG
PBR device
Web Dest Class: Web Src MAC: Leaf BD-MAC App
Dest IP: 192.168.1.1
192.168.1.1/24 Src IP: 192.168.1.2 Dest MAC: PBR-MAC 192.168.1.2/24
192.168.100.100/24
Src MAC: PBR-MAC
MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC
Dest IP: 192.168.1.1
Dest MAC: Leaf BD-MAC

*Source MAC based forwarding

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
VRF1
 New option

Traffic flow example with

source MAC rewrite
(two-arm design)

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
Topology

BD1 (192.168.1.254/24) BD2 (192.168.2.254/24)

Svc-Con-BD Svc-Prov-BD
(192.168.100.254/24) (192.168.200.254/24)

192.168.100.100 192.168.200.100
EPG EPG
PBR-MAC-con PBR-MAC-prov
Web DB
Shadow EPG Shadow EPG
192.168.1.1/24 Svc-consumer Svc-provider 192.168.2.1/24
MAC: Web-MAC MAC: DB-MAC

PBR Node (L3)

VRF1

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
 In this case, source MAC rewrite is enabled

Traffic flow (incoming) 2: policy applied (PBR)

Src IP: 192.168.1.1 5: Routing to provider ToR
1: traffic from Web to DB Src MAC: Leaf BD-MAC DL (Don’t Learn Bit): 1
Src class: Web Dest IP: 192.168.2.1
Dest Class: DB Dest MAC: PBR-MAC-con
Segment ID: Svc-Con-BD 6: traffic to DB
Src IP: 192.168.1.1 Src IP: 192.168.1.1
Src MAC: Web-MAC Src MAC: Leaf BD-MAC
Dest IP: 192.168.2.1 Dest IP: 192.168.2.1
Dest MAC: Leaf BD-MAC Dest MAC: DB-MAC

BD1 (192.168.1.254/24) BD2 (192.168.2.254/24)

Svc-Con-BD Svc-Prov-BD
(192.168.100.254/24) (192.168.200.254/24)
4: traffic comes back
Src class: Svc-provider
Dest Class: DB
192.168.100.100 192.168.200.100
EPG EPG
PBR-MAC-con PBR-MAC-prov
Src IP: 192.168.1.1
Web DB
Src MAC: PBR-MAC-prov
Shadow EPG Shadow EPG Dest IP: 192.168.2.1
192.168.1.1/24 Svc-consumer Svc-provider Dest MAC: Leaf BD-MAC 192.168.2.1/24
MAC: Web-MAC MAC: DB-MAC
3: traffic to Service node
PBR Node (L3)
Src IP: 192.168.1.1 *Source MAC for the flow is Leaf BD-MAC
Src MAC: Leaf BD-MAC VRF1
Dest IP: 192.168.2.1
Dest MAC: PBR-MAC-con

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Traffic flow (return traffic) Source MAC rewrite is not
mandatory for this direction
in this example.

5: Routing to consumer ToR 2: policy applied (PBR)

DL (Don’t Learn Bit): 1 Src IP: 192.168.2.1
Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.1 1: traffic from DB to Web
6: traffic to Web Dest MAC: PBR-MAC-prov Src class: DB
Src IP: 192.168.2.1 Segment ID: Svc-Prov-BD Dest Class: Web
Src MAC: Leaf BD-MAC
Dest IP: 192.168.1.1 Src IP: 192.168.2.1
Dest MAC: Web-MAC Src MAC: DB-MAC
Dest IP: 192.168.1.1
Dest MAC: Leaf BD-MAC

BD1 (192.168.1.254/24) BD2 (192.168.2.254/24)

Svc-Con-BD Svc-Prov-BD
(192.168.100.254/24) (192.168.200.254/24)
3: traffic to Service node
192.168.100.100 192.168.200.100
EPG Src IP: 192.168.2.1 EPG
PBR-MAC-con
4: traffic comes back PBR-MAC-prov
Web Src MAC: Leaf BD-MAC DB
Src class: Svc-consumer
Dest Class: Web Shadow EPG Shadow EPG Dest IP: 192.168.1.1
192.168.1.1/24 Svc-consumer Svc-provider Dest MAC: PBR-MAC-prov 192.168.2.1/24
MAC: Web-MAC Src IP: 192.168.2.1 MAC: App-MAC
Src MAC: PBR-MAC-con
Dest IP: 192.168.1.1 PBR Node (L3)
Dest MAC: Leaf BD-MAC

*Source MAC based forwarding VRF1

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
 In this case, source MAC rewrite is not required because
source (Web) and PBR device are in same BD

One-arm design
Traffic flow example
One BD for consumer, provider and service

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
 In this case, source MAC rewrite is not required because
source (Web) and PBR device are in same BD
Topology

BD1 (192.168.1.254/24)

EPG EPG
PBR device
Web App

192.168.1.1/24 192.168.1.100/24 192.168.1.2/24

MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
VRF1
 In this case, source MAC rewrite is NOT enabled

Traffic flow (incoming) 2: policy applied (PBR)

5: Routing to provider ToR
DL (Don’t Learn Bit): 1
Src IP: 192.168.1.1
1: traffic from Web to App Src MAC: Web-MAC
Src class: Web Dest IP: 192.168.1.2
Dest Class: App Dest MAC: PBR-MAC
Segment ID: BD1 6: traffic to App
Src IP: 192.168.1.1 Src IP: 192.168.1.1
Src MAC: Web-MAC Src MAC: PBR-MAC
Dest IP: 192.168.1.2 Dest IP: 192.168.1.2
Dest MAC: App-MAC Dest MAC: App-MAC

BD1 (192.168.1.254/24)

4: traffic comes back

Src class: PBR-node
Dest Class: App
EPG EPG
PBR device Src IP: 192.168.1.1
Web App
Src MAC: PBR-MAC
Dest IP: 192.168.1.2
192.168.1.1/24 192.168.1.100/24 Dest MAC: App-MAC 192.168.1.2/24
MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC
3: traffic to Service node

Src IP: 192.168.1.1

Src MAC: Web-MAC
Dest IP: 192.168.1.2
Dest MAC: PBR-MAC

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
VRF1
 In this case, source MAC rewrite is NOT enabled

Traffic flow (return traffic)

2: traffic redirected
5: Routing to consumer ToR Src IP: 192.168..1.2
DL (Don’t Learn Bit): 1 Src MAC: App-MAC
Dest IP: 192.168.1.1 1: traffic from App to Web
6: traffic to Web Dest MAC: PBR-MAC Src class: App
Src IP: 192.168.1.2 Segment ID: BD1 Dest Class: Web
Src MAC: PBR-MAC
Dest IP: 192.168.1.1 Src IP: 192.168.1.2
Dest MAC: Web-MAC Src MAC: App-MAC
Dest IP: 192.168.1.1
Dest MAC: Web-MAC

BD1 (192.168.1.254/24)

3: traffic to Service node

4: traffic comes back
EPG Src class: PBR-node Src IP: 192.168.1.2 EPG
PBR device
Web Dest Class: Web Src MAC: App-MAC App
Dest IP: 192.168.1.1
192.168.1.1/24 Src IP: 192.168.1.2 192.168.1.100/24 Dest MAC: PBR-MAC 192.168.1.2/24
Src MAC: PBR-MAC
MAC: Web-MAC MAC: PBR-MAC MAC: App-MAC
Dest IP: 192.168.1.1
Dest MAC: Web-MAC

*Source MAC based forwarding

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
VRF1
Configuration

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
ACI configuration

It’s disabled by default

In this traffic direction example, Rewrite source MAC option should be

enabled on consumer side. It’s not mandatory to enable it on provider
side in this example though enabling both shouldn’t break traffic
paths.

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Sample F5 configuration for test

Just for
forwarding.
All VLANs
No SNAT

Auto Last Hop Enable: use source

MAC based forwarding

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22