Brkaci 2125 PDF

ACI Multi-Site
Architecture and
Deployment
Max Ardica – Principal Engineer

BRKACI-2125
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
Cisco Spark spaces will be cs.co/clus17/#BRKACI-2125

available until July 3, 2017.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
At the end of the session, the participants should be able to:

 Articulate the different deployment options to interconnect
Cisco ACI networks (Multi-Pod vs. Multi-Site)
 Understand the functionalities and specific design
considerations associated to the ACI Multi-Site architecture
Initial assumption:
 The audience already has a good knowledge of ACI main
concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)
BRKACI-2125 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• ACI Network and Policy Domain
Evolution
• ACI Multi-Site Deep Dive
Overview and Use Cases
Introducing ACI Multi-Zone
Inter-Site Connectivity Deployment
Considerations
Control and Data Planes
Connecting to the External Layer 3 Domain
CloudSec and VMM Integration
Migration Scenarios
• Conclusions and Q&A
ACI Network and Policy Domain
Evolution
Introducing: Application Centric Infrastructure (ACI)
Web App DB
Outside QoS QoS QoS
(Tenant
Filter Service Filter
VRF)
APIC
Application Policy
ACI Fabric Infrastructure Controller
Integrated GBP VXLAN Overlay
Cisco ACI
Fabric and Policy Domain Evolution
ACI Single Pod Fabric ACI Stretched Fabric ACI Multi-Pod Fabric ACI Multi-Site
IPN IP
Pod ‘A’ Pod ‘n’ Fabric ‘A’ Fabric ‘n’
DC1 APIC Cluster DC2 MP-BGP - EVPN MP-BGP - EVPN
… …
ACI
APIC Cluster Multi-Zone
ACI 1.1 ACI 2.0 - Multiple ACI 3.0 - Multiple …more to

ACI 1.0 Leaf/Spine
Geographically Networks (Pods) in a Availability Zones come!
Single Pod Fabric
Stretch a single single Availability Zone (Fabrics) in a Single
fabric (Fabric) Region ’and’ Multi-
Region Policy
Management
Fabric and Policy Domain Evolution
Deployment Options
Single APIC Cluster/Single Fabric Multiple APIC Clusters/Multiple Fabrics
Stretched Fabric Multi-Fabric (with L2 and L3 DCI)
ACI Fabric Fabric ‘A’ Fabric ‘n’
DC1 APIC Cluster DC2
Inter-Site
App
L2/L3
DCI
Multi-Pod (from 2.0 Release) Multi-Site (3.0 Release, Q3CY17)

IPN
Pod ‘A’ Pod ‘n’ Fabric ‘A’ IP Fabric ‘n’
MP-BGP - EVPN MP-BGP - EVPN
… …
ACI
APIC Cluster Multi-Zone
Regions and Availability Zones
OpenStack and AWS Definitions
OpenStack
 Regions - Each Region has its own full OpenStack

deployment, including its own API endpoints, networks and
compute resources
 Availability Zones - Inside a Region, compute nodes can be
logically grouped into Availability Zones, when launching new
VM instance, we can specify AZ or even a specific node in a
AZ to run the VM instance
Amazon Web Services

 Regions – Separate large geographical areas, each
composed of multiple, isolated locations known as
Availability Zones
 Availability Zones - Distinct locations within a region that
are engineered to be isolated from failures in other
Availability Zones and provide inexpensive, low latency
network connectivity to other Availability Zones in the same
region
Terminology
 Pod – A Leaf/Spine network sharing a common control plane (ISIS, BGP,

COOP, …)
Pod == Network Fault Domain
 Fabric – Scope of an APIC Cluster, it can be one or more Pods
Fabric == Availability Zone (AZ) or Tenant Change Domain
 Multi-Pod – Single APIC Cluster with multiple leaf spine networks
Multi-Pod == Multiple Networks within a Single Availability Zone (Fabric)
 Multi-Fabric – Multiple APIC Clusters + associated Pods (you can have
Multi-Pod with Multi-Fabric)*
Multi-Fabric == Multi-Site == a DC infrastructure Region with multiple AZs
* Available from ACI release 3.1

ACI Multi-Pod For More Information on
ACI Multi-Pod:
Overview BRKACI-2003
VXLAN
Inter-Pod Network
Pod ‘A’ Pod ‘n’
MP-BGP - EVPN
APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP
Availability Zone
 Multiple ACI Pods connected by an IP Inter-Pod  Forwarding control plane (IS-IS, COOP)
L3 network, each Pod consists of leaf and spine fault isolation
nodes  Data Plane VXLAN encapsulation between
 Managed by a single APIC Cluster Pods
 Single Management and Policy Domain  End-to-end
BRKACI-2125
BRKACI-2003© 2017policy
Cisco and/or enforcement
its affiliates. All rights reserved. Cisco Public 12
Single Availability Zone with Maintenance & Configuration Zones
Scoping ‘Network Device’ Changes
Maintenance Zones – Groups of
switches managed as an “upgrade”
group Inter-Pod Network
ACI Multi-Pod
Fabric
APIC Cluster
Configuration Zone ‘A’ Configuration Zone ‘B’

 Configuration Zones can span any required set of switches, simplest approach may be to map a
configuration zone to an availability zone, applies to infrastructure configuration and policy only
Reducing the Impact of Configuration Errors
Introducing Configuration Zones
 Three different zone deployment modes:

 Enabled (default): updates are immediately sent
to all nodes part of the zone
Note: a node not part of any zone is equivalent Change the deployment
to a node part of a zone set to enabled. mode
Select entire Pod
 Disabled: updates are postponed until the zone
deployment mode is changed (or a node is Select specific Leaf Switches
removed from the zone)
 Triggered: send postponed updates to the nodes
part of the zone
Show the changes not applied yet
 The deployment mode can be configured for to a Disabled zone
an entire Pod or for a specified set of leaf
switches
Single Availability Zone with Tenant Isolation
Isolation for ‘Virtual Network Zone and Application’ Changes
Inter-Pod Network
ACI Multi-Pod
Fabric
APIC Cluster
Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain
 The ACI ‘Tenant’ construct provide a domain of application and associated virtual
network policy change
 Domain of operational change for an application (e.g. production vs. test)
ACI Multi-Pod
Supported Topologies
Intra-DC Two DC sites directly connected
10G/40G/100G
10G*/40G/100G 10G*/40G/100G
POD 1 10G*/40G/100G 10G*/40G/100G
POD n POD 1 Dark fiber/DWDM POD 2
(up to 50 msec RTT**)
…
APIC Cluster APIC Cluster
3 (or more) DC Sites directly connected Multiple sites interconnected by a

10G/40G/100G
generic L3 network
10G*/40G/100G
POD 1 10G*/40G/100G POD 2
Dark fiber/DWDM 10G*/40G/100G 10G*/40G/100G
(up to 50 msec RTT**)
L3
10G*/40G/100G
10G*/40G/100G (up to 50msec RTT**)
10G*/40G/100G
POD 3 ** ©50 msec

Ciscosupport addedAllin SW release 2.3(1)
* 10G only with QSA adapters on EX spines BRKACI-2125 2017 and/or its affiliates. rights reserved. Cisco Public 16
ACI Multi-Site
Overview IP Network
MP-BGP - EVPN
ACI
Multi-Zone
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
Region ‘C’
 Separate ACI Fabrics with independent APIC  MP-BGP EVPN control plane between sites
clusters  Data Plane VXLAN encapsulation across
 ACI Multi-Zone (AMZ) pushes cross-fabric sites
configuration to multiple APIC clusters  End-to-end policy definition and enforcement
providing scoping of all configuration changes
Typical Requirement
Creation of Two Independent Fabrics/AZs
Fabric ‘A’ (AZ 1)
Fabric ‘B’ (AZ 2)
Deployment of Two (or More) Pods per Fabric/AZ

‘Classic’ Active/Active
Pod ‘1.A’ Pod ‘2.A’
ACI
Multi-Zone
Pod ‘1.B’ Pod ‘2.B’
Use Case 1: Fabrics/AZs Map to Physical DCs
DC ‘One’

ACI
Multi-Zone
DC ‘Two’ BRKACI-2125 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Use Case 2: Fabric/AZ Deployed across Physical DCs

DC ‘One’ ACI
Multi-Zone
DC ‘Two’
ACI Multi-Site Deep Dive
Overview and Use Cases
ACI Multi-Site ACI 3.0 Release
(Q3CY17)
Overview IP Network
MP-BGP - EVPN
ACI
Multi-Zone
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
Region ‘C’
 Separate ACI Fabrics with independent APIC clusters  MP-BGP EVPN control plane between sites
 ACI Multi-Zone pushes cross-fabric configuration to  Data Plane VXLAN encapsulation across sites
multiple APIC clusters providing scoping of all  End-to-end policy definition and enforcement
configuration changes
ACI Multi-Site
Network and Identity Extended between Fabrics
Network information carried across Identity information carried across

Fabrics (Availability Zones) Fabrics (Availability Zones)
VTEP IP VNID Class-ID Tenant Packet No Multicast Requirement in

Backbone, Head-End
Replication (HER) for any
IP Network Layer 2 BUM traffic)
MP-BGP - EVPN
ACI
Multi-Zone
ACI Multi-Site
Namespace Normalization
Translation of Class-ID, VNID
Translation of Source IP Network (scoping of name spaces)
VTEP address
MP-BGP - EVPN
…
ACI
Multi-Zone
Site 1 Site to Site VTEP traffic (VTEPs, VNID
Site n
and Class-ID are mapped on spine)
Leaf to Leaf VTEP, Class-ID is local to the Fabric
Leaf to Leaf VTEP, Class-ID is local to the Fabric
VTEP
VNID Class-ID Tenant Packet
VTEP IP
VNID Class-ID Tenant Packet VTEP
IP VNID Class-ID Tenant Packet
IP
 Maintain separate name spaces with ID translation performed on the spine nodes
 Requires specific HW on the spine to support for this functionality
ACI Multi-Site
Hardware Requirements
 Support all ACI leaf switches (1st

Generation, -EX and -FX) Can have only a subset
of spines connecting to
IP Network
 Only -EX spine nodes (or newer) to connect the IP network
to the inter-site network
1st Gen 1st Gen -EX -EX
 New FX non modular spine (9364C,
64x40G/100G ports) will be supported for
Multi-Site in Q1CY18 timeframe
 1st generation spines (including 9336PQ)
not supported
Can still leverage those for intra-site leaf to leaf ACI
Multi-Zone
communication
ACI Multi-Site Networking Options
Per Bridge Domain Behavior
Layer 3 only across sites IP Mobility without L2

flooding
L3 L3
Site Site Site Site 2
Site
1 2 1 2
ACI ACI
Multi-Zone Multi-Zone
 Bridge Domains and  Same IP subnet defined in

subnets not extended separate Sites
across Sites  Support for IP Mobility
 Layer 3 Intra-VRF or Inter- (‘cold’ VM migration) and
VRF communication only intra-subnet communication
across sites
 No Layer 2 flooding
across sites
IP Mobility without Layer 2 Flooding
Traditional L3Out on the Border Leaf Nodes
IP
WAN
 Ingress traffic may continue to be steered to Site 1

even after the IP Mobility event
 ACI Multi-Site discovers the new location of the
endpoint and redirects the traffic to Site 2
IP Mobility without Layer 2 Flooding
Traditional L3Out on the Border Leaf Nodes GOLF L3Out
IP WAN
GOLF Routers GOLF Routers
MP-BGP MP-BGP
EVPN EVPN
WAN
 Ingress traffic may continue to be steered to Site 1  Host routes must be advertised from each site
even after the IP Mobility event toward the GOLF routers
 ACI Multi-Site discovers the new location of the  Ingress traffic is optimally steered toward the
endpoint and redirects the traffic to Site 2 new endpoint location
Per Bridge Domain Behavior
Layer 3 only across sites IP Mobility without L2 Full Layer 2 and Layer
flooding 3 Extension
L3 L3 L3
Site Site Site Site 2
Site Site Site
1 2 1 2 1 2
ACI ACI ACI
Multi-Zone Multi-Zone Multi-Zone
 Bridge Domains and  Same IP subnet defined in  Interconnecting separate

subnets not extended separate Sites sites for fault containment
across Sites  Support for IP Mobility and scalability reasons
 Layer 3 Intra-VRF or Inter- (‘cold’ VM migration) and  Layer 2 domains stretched
VRF communication only intra-subnet communication across Sites (Support for
across sites ‘hot’ VM migration)
 No Layer 2 flooding  Layer 2 flooding across
across sites sites
ACI Multi-Site
Scalability Values Supported in 3.0 Release
AMZ Scale AMZ Scale
Scale Parameter
Stretched Objects Not Stretched Objects
Sites 4 N/A
Leaf scale 200 per site ACI 2.3 parity
Tenants 100 ACI 2.3 parity AMZ Scale
VRFs 400 ACI 2.3 parity
Subnet 2,000 ACI 2.3 parity
BD 800 ACI 2.3 parity
EPGs 800 ACI 2.3 parity
Endpoints 180,000 ACI 2.3 parity
Contracts 1,000 ACI 2.3 parity APIC Domain 2

APIC Domain 1
Scale Scale
L3Outs External EPGs 500 (prefixes) ACI 2.3 parity
Translation Entries 32,000 N/A
IGMP Snooping 8,000 8,000

Introducing ACI Multi-Zone
ACI Multi-Zone
Multi-Site Policy Manager  Micro-services architecture
• Multiple VMs are created and run concurrently
(active/active)
• vSphere only support at FCS (KVM and physical
appliance support scoped for future releases)
REST
GUI  OOB Mgmt connectivity to the APIC clusters
API
deployed in separate sites
• Support for 500 msec to 1 sec RTT
ACI Multi-Zone
 Main functions offered by ACI Multi-Zone:
VM VM VM • Monitoring the health-state of the different ACI Sites
• Provisioning of day-0 configuration to establish
Hypervisor inter-site EVPN control plane
• Defining and provisioning policies across sites
(scope of changes)
….. • Inter-site troubleshooting (post-3.0 release)
Site 1 Site 2 Site n
 Recommended to deploy ACI Multi-Zone for a
single ACI site to plan for a future Multi-Site
deployment
ACI Multi-Zone
Deployment Considerations
Intra-DC Deployment Interconnecting DCs over WAN
New York
Site3
IP Network
WAN
Milan Rome
Hypervisor Hypervisor Hypervisor Site1 Site2
VM VM VM
ACI Multi-Zone
Hypervisor Hypervisor
ACI
VM VM Multi-Zone VM
 Hypervisors can be connected directly to the DC OOB network  Moderate latency supported between ACI Multi-Zone nodes
 Each ACI Multi-Zone VM has a unique routable IP  Higher latency (500 msec to 1 sec RTT) between ACI Multi-Zone
 Async calls from ACI Multi-Zone to APIC nodes and remote APIC clusters
 If possible deploy a node in each site for availability purposes
(network partition scenarios)
ACI Multi-Zone
Dashboard
 Health/Faults for all managed

sites
 Easily way to identify
stretched policies across
sites
 Quickly search for any
deployed inter-site policy
 Provide direct access to the
APIC GUIs in different sites
ACI Multi-Zone
MP-BGP/EVPN Infra
 Configure Day-0 infra

policies
 Select spines establishing
MP-BGP EVPN peering
with remote sites
 Site/Pod Data Plan TEPs
(D-ETEPs)
 Spine Control Plane TEPs
(CP-ETEPs)
ACI Multi-Zone
Users
 Local Users to ACI Multi-

Zone
 Authentication and
Authorization
 RBAC
ACI Multi-Zone
Tenants
 APIC Tenants managed by

ACI Multi-Zone
 Associated to Users
(RBAC)
 Can be imported from
APIC(s)
ACI Multi-Zone
Templates and Profiles
Profile
 Template = APIC policy definition POLICY
(App & Network)

Template
Template DEFINITION
EP1 EP2
 Template is the scope/granularity

C
EPG EPG
of what can be pushed to sites

 Template is associated to all SITE
LOCAL
managed sites or a subset of
sites
 Profile = Group of Templates Site 1 Site 2
sharing a common use-case
EFFECTIVE EFFECTIVE
 Scope of change: policies can be POLICY POLICY
pushed to separate sites at

different times
ACI Multi-Zone
Templates and Profiles
 Template = APIC policy definition

(App & Network)
 Template is the scope/granularity
of what can be pushed to sites
 Template is associated to all
managed sites or a subset of
sites
 Profile = Group of Templates
sharing a common use-case
 Scope of change: policies can be
pushed to separate sites at
different times
ACI Multi-Zone
Scope of Changes
Tenant-A Tenant-B  Each Site has assigned a unique ID

Scope: Site 1 and 2 Scope: Site 3
 Configuration created on ACI Multi-
Zone (Template) has associated a
Site 1 Site 2 Site 3 scope
 The scope identifies the list of sites
where the configuration is pushed
 Configuration changes are staged
on the ACI Multi-Zone before being
ACI pushed to the sites
Multi-Zone
APIC vs. ACI Multi-Zone Functions
ACI
Multi-Zone
 Central point of management and  Complementary to APIC

configuration for the Fabric  Provisioning and managing of “Inter-Site
 Responsible for all Fabric local functions Tenant and Networking Policies”
Fabric discovery and bring up  Scope of changes
Fabric access policies Granularly propagate policies to multiple APIC
Service graphs clusters
Domains creation (VMM, Physical, etc.)
…  Can import and merge configuration from
 Integration with third party services different APIC cluster domains
 Maintains runtime data (VTEP address, VNID,  End-to-end visibility and troubleshooting
Class_ID, GIPo, etc.)  No run time data, configuration repository
 No participation in the fabric control and data  No participation in the fabric control and data
planes planes
Inter-Site Connectivity
Deployment Considerations
ACI Multi-Site
Inter-Site IP Network Requirements
Site ‘A’ IP Site ‘n’
MP-BGP EVPN
ACI
Multi-Zone
 Not managed by APIC, must be separately configured (day-0 configuration)

 IP topology can be arbitrary, not mandatory to connect to all spine nodes, can extend long distance
(across the World)
 Main requirements:
 OSPF on the first hop routers to peer with the spine nodes and exchange site specific E-TEP reachability
 Increased MTU support to allow site-to-site VXLAN traffic
Control and Data Planes
ACI Multi-Site
BGP Inter-Site Peers
 Spines connected to the IP Network perform
IP two main functions:
Network
1. Establishing MP-BGP EVPN peerings with
BGP Inter-Site
Peers
remote sites
• One dedicated Control Plane ETEP address (CP-
ETEP) is assigned to each spine running MP-BGP
1st Gen 1st Gen
DP-ETEP EVPN  must be a globally routable address
CP-ETEP
1
CP-ETEP
2
• Full mesh MP-BGP EVPN peerings with BGP Inter-Site
Peers in remote sites
• Received EVPN information is synced with the other
local spines that are not BGP Inter-Site Peers
2. Forward inter-sites data-plane traffic
• One Anycast Data Plane ETEP address (DP-ETEP) is
assigned to all the spines externally connected 
uniquely identifies the site and must be globally
routable in the inter-site network
ACI Multi-Site IP Network Routing Table
Exchanging TEP Information across Sites
DP-ETEP A
CP-ETEP S3-S4
DP-ETEP B
CP-ETEP S5-S8
 OSPF between spines and Inter-Site
network (only supported option in 3.0)
 Exchange of External Spine TEP OSPF IP
OSPF
addresses (DP-ETEPs and CP-ETEPs) Network
across sites
DP-ETEP A DP-ETEP B
TEP Pool information not advertised to the S1 S2 S3 S4 S5 S6 S7 S8
CP-ETEPs S3-S4 IS-IS to OSPF CP-ETEPs S5-S8
Inter-Site network mutual redistribution
Recommended to use not overlapping TEP TEP Pool 1 TEP Pool 2
Pools if possible ACI
Multi-Zone
 Multicast support not required in the
Inter-Site Network Site 2
Site 1
Head-End Replication (HER) for L2 BUM traffic
(only for stretched BDs) Leaf Routing Table Leaf Routing Table
IP Prefix Next-Hop IP Prefix Next-Hop
DP E-TEP B Pod1-S3, Pod1-S4 DP E-TEP A Pod2-S1, Pod2-S2,
Pod2-S3, Pod2-S4
ACI Multi-Site
Inter-Site MP-BGP EVPN Control Plane
S3-S4 Table S5-S8 Table

 MP-BGP EVPN used to
EP1 Leaf 1 EP2 Leaf 4
communicate Endpoint (EP) MP-BGP EVPN
EP2 DP-ETEP B EP1 DP-ETEP A
information across Sites
EP3 Leaf 4 EP4 Leaf 6
MP-iBGP or MP-EBGP peering
supported across sites IP
Network
Remote host route entries (EVPN Type-
2) are associated to the remote site DP-ETEP A DP-ETEP B
S2 S3 S4 S5 S6 S7 S8
Anycast DP-ETEP address S1
 Automatic filtering of endpoint COOP COOP

information across Sites ACI
Multi-Zone
Host routes are exchanged only if there
EP2 EP4
is a cross-site contract requiring EP1 EP3
Site 1 Site 2
communication between endpoints
Define and push inter-site policy
EP1 EP2
EPG
C EPG
ACI Multi-Site
Inter-Site Policies
IP
Network
DP-ETEP A DP-ETEP B
 Inter-Site policies defined on S1 S2 S3 S4 S5 S6 S7 S8
the ACI Multi-Zone are
pushed to the respective
ACI
APIC domains Multi-Zone
 Policies are enforced at the EP1 EP2

Site 1 Site 2
ingress leaf node, once it has EP1
EP1 C EP2 EPG
learned on the data plane EPG
C EP2 EPG EPG
info for remote endpoint

EP1 EP2
EPG
C EPG
ACI Multi-Site
Inter-Sites Unicast Data Plane Policy information (EP1’s
Class-ID) carried across Pods
S2 has remote info for EP2
VTEP IP VNID Class-ID Tenant Packet
and encapsulates traffic to
remote DP ETEP B
EP1 AddressLeaf 4
S6 translates the VNID
EP2 Leaf 4
EP2 DP ETEP B S4 rewrites the S-VTEP and Class-ID to local DP ETEP A
IP values and sends traffic to
EP1
to be DP ETEP A
the local leaf
4 4
Site 1 3 5 Site 2
DP-ETEP A DP-ETEP B
S1 S2Proxy AS3 S4 S5 S6Proxy BS7 S8

EP2 e1/1
EP1 e1/3 EP1 DP ETEP A
5 * Proxy B
* Proxy A ACI
Multi-Zone
Leaf learns remote Site
EP2 unknown, traffic is 2 location info for EP1
EP1 EP2
encapsulated to the local Proxy
A Spine VTEP (adding S_Class 1 6
information) EP1 sends traffic destined If policy allows it, EP2
to remote EP2 receives the packet
EP1 EP2
EPG
C EPG = VXLAN Encap/Decap
ACI Multi-Site
Inter-Site Data Plane (2) Policy information (EP1’s
S_Class) carried across Pods
VTEP IP VNID Class-ID Tenant Packet
S3 translates the VNID

and S_Class to local EP1 Leaf 4
values and sends traffic to EP2 DP ETEP A
IP S6 rewrites the S-VTEP
the local leaf to be DP ETEP B
10 9
Site 1 DP-ETEP B Site 2
DP-ETEP A
S1 S2 S3 S4 S5 S6 S7 S8
EP1 e1/3
EP2 DP ETEP B EP1 DP ETEP A
** Proxy A
8 * Proxy B
11 ACI
Leaf encapsulates traffic to
Leaf learns remote Site Multi-Zone
remote DP ETEP address
location info for EP2 EP1 EP2
12 7
EP1 receives the packet EP2 sends traffic back to
remote EP1
EP1 EP2
EPG
C EPG = VXLAN Encap/Decap
ACI Multi-Site
Inter-Site Data Plane (3) = VXLAN Encap/Decap
From this point EP1 to EP2 communication is encapsulated Leaf to Remote Spine DP ETEPs in both directions
IP
Site 1 DP-ETEP B Site 2

DP-ETEP A
S1 S2 S3 S4 S5 S6 S7 S8
**
ACI
Multi-Zone
EP1 EP2
EP1 EP2
EPG
C EPG
ACI Multi-Site
Layer 2 BUM Traffic Data Plane
S3 is elected as Multi-Site forwarder for GIPo 1 S7 translates the VNID and the
BUM traffic  it creates an unicast VXLAN GIPo values to locally significant
packet with DP-ETEP A as S_VTEP and ones and associates the frame to
Multicast HER-ETEP B* as D_VTEP IP an FTAG tree
3 4
DP-ETEP A HER-ETEP B
S1 S2 S3 S4 S5 S6 S7 S8
BUM frame is flooded along the
tree associated to GIPo. VTEP
2 5
* learns VM1 remote location
*
EP1 DP-ETEP A
BUM frame is associated to ACI
Multi-Zone
GIPo1 and flooded intra-site via Proxy B
*
the corresponding FTAG tree EP1 EP2
1 6
GIPo1 = Multicast Group EP1 generates a BUM EP2 receives the BUM
associated to EP1’s BD frame
frame
*This is a different ETEP address than the one used for inter-site L3 unicast communication
Connecting to the External
Layer 3 Domain
Connecting ACI to Layer 3 Domain
‘Traditional’ L3Out on the BL Nodes
Client
PE
PE
WAN
PE
L3Out PE
 Connecting to WAN Edge devices at

Border Leaf nodes
Definition of a L3Out logical construct
 VRF-lite hand-off for extending L3 multi-
Border Leafs
tenancy outside the ACI fabric
Each tenant defines one (or more) L3Out with
a set of Logical Nodes, Logical Interfaces,
peering protocol
Multi-Site and Traditional L3Out BD1 L3Out-1 BD2
ACI EPG Web1 L3Out-2 EPG Web2

Basic assumption: every site defines
Multi-Zone 3
its local L3Out connection C1 C2
2 4
IP Network
2 ExtEPG-1 ExtEPG-2
Site 1 Site 2
1 L3Out-1 L3Out-2 1
BL Nodes BL Nodes
Routing Protocol Routing Protocol
Route policy Route policy
ExtEPG-1 ExtEPG-1
EPG Web1 C1 ExtEPG-1 EPG Web2 C2 ExtEPG-2
5 5
Multi-Site and Traditional L3Out BD L3Out-1
Stretched BD
L3Out-2
Basic assumptions: every site ACI EPG Web
Multi-Zone
defines its local L3Out connection C1
ExtEPG-1 ExtEPG-2
IP Network
Site 1 Site 2
EPG Web C1 ExtEPG-1 EPG Web C1 ExtEPG-2
Connecting ACI to Layer 3 Domain For More Information on
GOLF Deployment:
‘GOLF’ Design LABACI-2101
= VXLAN Encap/Decap Client

PE
PE
WAN
PE
PE
GOLF Routers (ASR 9000, ASR

DCI 1000, Nexus 7000)
OTV/VPLS
 Direct or indirect connection from spines to WAN Edge
routers
 Better scalability, one protocol session for all VRFs, no longer
constraint by border leaf HW table
 VXLAN handoff with MP-BGP EVPN
 Simplified tenant L3Out configuration
 Support for host routes advertisement out of the ACI Fabric
 VRF configuration automation on GOLF router through
OpFlex exchange
GOLF and Multi-Site Integration
Centralized and Distributed Models
Centralized GOLF Devices* Distributed GOLF Devices
WAN
GOLF Routers WAN
GOLF Routers GOLF Routers
MP-BGP MP-BGP MP-BGP

MP-BGP
EVPN EVPN EVPN
EVPN
ACI ACI
 Common when ‘sites’ represent rooms/halls in the  ‘Sites’ represent separate physical DCs
same physical DC
 Local only MP-BGP EVPN peering between
 MP-BGP EVPN peering required from spines in spines and GOLF router
each fabric and the centralized WAN Edge devices
*Supported post-FCS BRKACI-2125 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Inter-DC Scenario with Stretched BD
WAN Edge devices inject host

routes into the WAN or register
Host routes for endpoint belonging them in the LISP database
Host routes for endpoint belonging
to public BD subnets in Pod ‘A’
to public BD subnets in Pod ‘B’
MP-BGP EVPN Control Plane

MP-BGP EVPN Control Plane
Site ‘A’ IPN Site ‘B’
Inter-DC Scenario with Stretched BD (2) Granular inbound path
Remote Router Table
10.10.10.10/32 optimization( host route
G1,G2
advertisement into the WAN or
10.10.10.11/32 G3,G4
integration with LISP)
G1,G2 Routing Table

10.10.10.0/24 A WAN G3,G4 Routing Table
10.10.10.10/32 A 10.10.10.0/24 B
10.10.10.11/32 B
IPN
Proxy A Proxy B
10.10.10.10 ACI 10.10.10.11

Multi-Zone
CloudSec and VMM
Integration
ACI Multi-Site
CloudSec Encryption for VXLAN Traffic
Encrypted Fabric to Fabric Traffic
[ GCM-AES-128 (32-bit PN), GCM--AES-256 (32-bit
PN), GCM-AES-128-XPN (64-bit PN), GCM-AES-256-
VTEP Information Clear Text XPN (64-bit PN)])
VTEP IP MACSEC VXLAN Tenant Packet
IP Network
MP-BGP - EVPN
ACI
Multi-Zone
Support planned for 3.1 release for FX line cards and 9364C platform
ACI Multi-Site and L3Out 3 EPG Web
VMM Integration ACI
Multi-Zone
VMM DC1 VMM DC2
2 4
IP Network
2
Site 1 VMM Domain VMM Domain Site 2

DC1 DC2
1 1
vCenter vCenter
Server-1 Server-2
5 5
ESX ESX ESX ESX
Live migration with vSphere 6 and above
Port-Group Web Port-Group Web
6 VDS1 6 VDS2
VMM Domain: DC1 and DC2
Migration Scenarios
ACI Multi-Zone Interactions with APIC Clusters Supported at
FCS
Supported at FCS
1
ACI ACI
2
1
2
1 2
1 1 3
1
Site 1 Site 2 Site 1 Site 2

Green Field or Green Field or Brown Field Green Field
Brown Field Brown Field
1. Model inter-site policies on the 1. Import existing App1 from site 1 to

ACI Multi-Zone and apply the ACI Multi-Zone template
template to the sites
2. Apply App1 template to site 2
2. Push policies to the green field (or
3. Push App1 template to site 2
brown field) sites
ACI Multi-Zone Interactions With APIC Clusters Future
Roadmap
Roadmap Options
2
ACI ACI
1
Diff &
2
1 Merge
1 3
1 1 1
3
1
Site 1 Site 2 Site 1 Site 2

Brown Field Brown Field Brown Field Brown Field
Assumption: Tenant1 is defined in site1 only 1. Import existing Tenant1, App1 definition from
both sites
1. Import existing Tenant1, App1 definition
from site 1 APIC 2. Diff/merge template policies and apply the
template to both sites
2. Apply App1 template to site 2
3. Push template policies to both sites
3. Push App1 template to site 2
ACI Multi-Site
Migration Paths
Fabric 1
‘Brownfield’ ACI Fabric to Site 1 Site 2
Multi-Site
APIC
Multi-Zone
Pod ‘A’ Pod ‘B’ Multi-Pod to Pod ‘A’ Pod ‘B’

‘Hierarchical Multi-Site’ Site 2
APIC Cluster
APIC Cluster
APIC
Multi-Pod Planned for Q1CY18 Site 1 Multi-Zone
Fabric 1 Fabric 2
Multi-Fabric Design to
Inter-Site Site 1 Site 2
App Multi-Site
L2/L3
DCI APIC
Multi-Fabric Scoped for the future BRKACI-2125 Multi-Zone

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Conclusions and Q&A
Conclusions
 Cisco ACI offers different multi-fabric
options that can be deployed today
 There is a solid roadmap to evolve
those options in the short and mid term
 Multi-Pod represents the natural
evolution of the existing Stretched
Fabric design
 Multi-Site will replace the Dual-Fabric
MP-BGP EVPN MP-BGP EVPN
approach
 Cisco will offer migration options to

drive the adoption of those new
solutions
ACI Multi-Pod and ACI Multi-Site
Summary of Main Differences
Multi-Pod Multi-Site
Pod ‘A’ IPN Pod ‘n’ Site ‘A’ IP Site ‘n’
MP-BGP EVPN MP-BGP EVPN
… …
APIC Cluster
ACI
Multi-Zone
Operational Feature Richness Change Domain Fabric Nodes

Simplicity across Pods Isolation Scale
High Latency No Multicast

Lower Number of Single VMM required in the
across Sites
APIC Nodes across Pods IP Network
Where to Go for More Information
 ACI Stretched Fabric White Paper

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-aci-
stretched-fabric.html#concept_524263C54D8749F2AD248FAEBA7DAD78
 ACI Multi-Pod White Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-
centric-infrastructure/white-paper-c11-737855.html?cachemode=refresh
 ACI Dual Fabric Design Guide
http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737077.html?cachemode=refresh
 ACI and GOLF High Level Integration Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/application-centric-infrastructure/white-paper-c11-736899.html
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

Brkaci 2125 PDF

Uploaded by

Copyright:

Available Formats

Brkaci 2125 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkaci 2125 PDF

Uploaded by

Copyright:

Available Formats

ACI Multi-Site

Max Ardica – Principal Engineer

Cisco Spark spaces will be cs.co/clus17/#BRKACI-2125

At the end of the session, the participants should be able to:

DC1 APIC Cluster DC2 MP-BGP - EVPN MP-BGP - EVPN

ACI 1.1 ACI 2.0 - Multiple ACI 3.0 - Multiple …more to

Multi-Pod (from 2.0 Release) Multi-Site (3.0 Release, Q3CY17)

MP-BGP - EVPN MP-BGP - EVPN

 Regions - Each Region has its own full OpenStack

Amazon Web Services

 Pod – A Leaf/Spine network sharing a common control plane (ISIS, BGP,

* Available from ACI release 3.1

Configuration Zone ‘A’ Configuration Zone ‘B’

 Three different zone deployment modes:

Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain

APIC Cluster APIC Cluster

3 (or more) DC Sites directly connected Multiple sites interconnected by a

POD 3 ** ©50 msec

Fabric ‘A’ (AZ 1)

Fabric ‘B’ (AZ 2)

Fabric ‘A’ (AZ 1)

Pod ‘1.A’ Pod ‘2.A’

Fabric ‘B’ (AZ 2)

Pod ‘1.B’ Pod ‘2.B’

Fabric ‘A’ (AZ 1)

Pod ‘1.A’ Pod ‘2.A’

Fabric ‘B’ (AZ 2)

Pod ‘1.B’ Pod ‘2.B’

Fabric ‘A’ (AZ 1)

Pod ‘1.A’ Pod ‘2.A’

Fabric ‘B’ (AZ 2)

Pod ‘1.B’ Pod ‘2.B’

Network information carried across Identity information carried across

VTEP IP VNID Class-ID Tenant Packet No Multicast Requirement in

 Support all ACI leaf switches (1st

Layer 3 only across sites IP Mobility without L2

 Bridge Domains and  Same IP subnet defined in

 Ingress traffic may continue to be steered to Site 1

 Bridge Domains and  Same IP subnet defined in  Interconnecting separate

Leaf scale 200 per site ACI 2.3 parity

Tenants 100 ACI 2.3 parity AMZ Scale

VRFs 400 ACI 2.3 parity

Subnet 2,000 ACI 2.3 parity

BD 800 ACI 2.3 parity

EPGs 800 ACI 2.3 parity

Endpoints 180,000 ACI 2.3 parity

Contracts 1,000 ACI 2.3 parity APIC Domain 2

Translation Entries 32,000 N/A

IGMP Snooping 8,000 8,000

 Health/Faults for all managed

 Configure Day-0 infra

 Local Users to ACI Multi-

 APIC Tenants managed by

(App & Network)

 Template is the scope/granularity

of what can be pushed to sites

 Scope of change: policies can be POLICY POLICY

pushed to separate sites at

 Template = APIC policy definition