Cisco Nexus 9000 Series User manual

Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release
7.x
First Published: 2015-01-27
Last Modified: 2017-02-17
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH
THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,
CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version
of the UNIX operating system. All rights reserved. Copyright ©1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.
CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network
topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional
and coincidental.
This product includes cryptographic software written by Eric Young ([email protected]).
This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (https://www.openssl.org/)
This product includes software written by Tim Hudson ([email protected]).
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: https:/
/www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1110R)
©2016-2017 Cisco Systems, Inc. All rights reserved.

CONTENTS
Preface Preface ix
Audience ix
Document Conventions ix
Related Documentation for Cisco Nexus 9000 Series Switches x
Documentation Feedback x
Obtaining Documentation and Submitting a Service Request xi
CHAPTER 1 New and Changed Information 1
New and Changed Information 1
CHAPTER 2 Overview 5
VXLAN Overview 5
VXLAN Encapsulation and Packet Format 6
VXLAN Tunnel Endpoint 6
VXLAN Packet Forwarding Flow 6
Cisco Nexus 9000 as Hardware-Based VXLAN Gateway 6
vPC Consistency Check for vPC VTEPs 7
Static Ingress Replication 8
Bud Node Topology 9
VXLAN BGP EVPN Control Plane 9
CHAPTER 3 Configuring VXLAN 11
Information About VXLAN 11
Guidelines and Limitations for VXLAN 11
Considerations for VXLAN Deployment 14
VPC Considerations for VXLAN Deployment 15
Network Considerations for VXLAN Deployments 18
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
iii

Considerations for the Transport Network 19
Configuring VXLAN 19
Enabling VXLANs 19
Mapping VLAN to VXLAN VNI 20
Configuring Port VLAN Mapping on a Trunk Port 20
Configuring Inner VLAN and Outer VLAN Mapping on a Trunk Port 23
Creating and Configuring an NVE Interface and Associate VNIs 25
Configuring Static MAC for VXLAN VTEP 25
Disabling VXLANs 26
Configuring BGP EVPN Ingress Replication 27
Configuring Static Ingress Replication 27
Configuring Q-in-VNI 28
Configuring Selective Q-in-VNI 30
Configuring Q-in-VNI with LACP Tunneling 32
Configuring FHRP Over VXLAN 35
Overview for FHRP Over VXLAN 35
Guidelines and Limitations for FHRP Over VXLAN 35
Only Supported Deployments for FHRP Over VXLAN 36
New Supported Topology for Configuring FHRP Over VXLAN 38
Configuring IGMP Snooping Over VXLAN 39
Overview of IGMP Snooping Over VXLAN 39
Guidelines and Limitations for IGMP Snooping Over VXLAN 39
Configuring IGMP Snooping Over VXLAN 40
Verifying the VXLAN Configuration 40
Example of VXLAN Bridging Configuration 43
CHAPTER 4 Configuring VXLAN BGP EVPN 51
Information About VXLAN BGP EVPN 51
Guidelines and Limitations for VXLAN BGP EVPN 51
Notes for EVPN Convergence 53
Considerations for VXLAN BGP EVPN Deployment 54
VPC Considerations for VXLAN BGP EVPN Deployment 55
Network Considerations for VXLAN Deployments 57
Considerations for the Transport Network 58
BGP EVPN Considerations for VXLAN Deployment 59
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
iv
Contents

Commands for BGP EVPN 59
Configuring VXLAN BGP EVPN 60
Enabling VXLAN 60
Configuring VLAN and VXLAN VNI 61
Configuring VRF for VXLAN Routing 61
Configuring SVI for Hosts for VXLAN Routing 62
Configuring VRF Overlay VLAN for VXLAN Routing 62
Configuring VNI Under VRF for VXLAN Routing 62
Configuring Anycast Gateway for VXLAN Routing 63
Configuring the NVE Interface and VNIs 63
Configuring BGP on the VTEP 64
Configuring RD and Route Targets for VXLAN Bridging 64
Configuring VXLAN EVPN Ingress Replication 65
Configuring BGP for EVPN on the Spine 66
Suppressing ARP 67
Disabling VXLANs 68
Duplicate Detection for IP and MAC Addresses 68
Verifying the VXLAN BGP EVPN Configuration 70
Example of VXLAN BGP EVPN (EBGP) 71
Example of VXLAN BGP EVPN (IBGP) 80
Example Show Commands 87
CHAPTER 5 Configuring VXLAN OAM 91
VXLAN OAM Overview 91
Loopback (Ping) Message 92
Traceroute or Pathtrace Message 93
Configuring VXLAN OAM 95
Configuring NGOAM Profile 98
NGOAM Authentication 99
CHAPTER 6 Configuring VXLAN EVPN Multihoming 101
VXLAN EVPN Multihoming Overview 101
Introduction to Multihoming 101
BGP EVPN Multihoming 101
BGP EVPN Multihoming Terminology 102
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
v
Contents

EVPN Multihoming Implementation 102
EVPN Multihoming Redundancy Group 103
Ethernet Segment Identifier 103
LACP Bundling 103
Guidelines and Limitations for VXLAN EVPN Multihoming 104
Configuring VXLAN EVPN Multihoming 104
Enabling EVPN Multihoming 104
VXLAN EVPN Multihoming Configuration Examples 105
Configuring Layer 2 Gateway STP 107
Layer 2 Gateway STP Overview 107
Guidelines for Moving to Layer 2 Gateway STP 107
Enabling Layer 2 Gateway STP on a Switch 108
Configuring VXLAN EVPN Multihoming Traffic Flows 111
EVPN Multihoming Local Traffic Flows 111
EVPN Multihoming Remote Traffic Flows 116
EVPN Multihoming BUM Flows 121
Configuring VLAN Consistency Checking 124
Overview of VLAN Consistency Checking 124
VLAN Consistency Checking Guidelines and Limitations 124
Configuring VLAN Consistency Checking 125
Displaying Show command Output for VLAN Consistency Checking 125
Configuring ESI ARP Suppression 126
Overview of ESI ARP Suppression 126
Limitations for ESI ARP Suppression 127
Configuring ESI ARP Suppression 127
Displaying Show Commands for ESI ARP Suppression 127
APPENDIX A VXLAN Bud Node Over VPC 129
VXLAN Bud Node Over VPC Overview 130
VXLAN Bud Node Over VPC Topology Example 131
APPENDIX B DHCP Relay in VXLAN BGP EVPN 135
DHCP Relay in VXLAN BGP EVPN Overview 135
DHCP Relay in VXLAN BGP EVPN Example 136
Basic VXLAN BGP EVPN Configuration 137
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
vi
Contents

DHCP Relay on VTEPs 139
Client on Tenant VRF and Server on Layer 3 Default VRF 139
Client on Tenant VRF (SVI X) and Server on the Same Tenant VRF (SVI Y) 142
Client on Tenant VRF (VRF X) and Server on Different Tenant VRF (VRF Y) 146
Client on Tenant VRF and Server on Non-Default Non-VXLAN VRF 148
Configuring VPC Peers Example 150
vPC VTEP DHCP Relay Configuration Example 152
APPENDIX C EVPN with Transparent Firewall Insertion 155
Overview of EVPN with Transparent Firewall Insertion 155
EVPN with Transparent Firewall Insertion Example 157
Show Command Examples 160
APPENDIX D IPv6 Across a VXLAN EVPN Fabric 163
Overview of IPv6 Across a VXLAN EVPN Fabric 163
Configuring IPv6 Across a VXLAN EVPN Fabric Example 163
Show Command Examples 166
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
vii
Contents

Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
viii
Contents

Preface
This preface includes the following sections:
•Audience, page ix
•Document Conventions, page ix
•Related Documentation for Cisco Nexus 9000 Series Switches, page x
•Documentation Feedback, page x
•Obtaining Documentation and Submitting a Service Request, page xi
Audience
This publication is for network administrators who install, configure, and maintain Cisco Nexus switches.
Document Conventions
Command descriptions use the following conventions:
DescriptionConvention
Bold text indicates the commands and keywords that you enter literally
as shown.
bold
Italic text indicates arguments for which the user supplies the values.
Italic
Square brackets enclose an optional element (keyword or argument).[x]
Square brackets enclosing keywords or arguments separated by a vertical
bar indicate an optional choice.
[x | y]
Braces enclosing keywords or arguments separated by a vertical bar
indicate a required choice.
{x | y}
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
ix

DescriptionConvention
Nested set of square brackets or braces indicate optional or required
choices within optional or required elements. Braces and a vertical bar
within square brackets indicate a required choice within an optional
element.
[x {y | z}]
Indicates a variable for which you supply values, in context where italics
cannot be used.
variable
A nonquoted set of characters. Do not use quotation marks around the
string or the string will include the quotation marks.
string
Examples use the following conventions:
DescriptionConvention
Terminal sessions and information the switch displays are in screen font.screen font
Information you must enter is in boldface screen font.boldface screen font
Arguments for which you supply values are in italic screen font.
italic screen font
Nonprinting characters, such as passwords, are in angle brackets.< >
Default responses to system prompts are in square brackets.[ ]
An exclamation point (!) or a pound sign (#) at the beginning of a line
of code indicates a comment line.
!, #
Related Documentation for Cisco Nexus 9000 Series Switches
The entire Cisco Nexus 9000 Series switch documentation set is available at the following URL:
http://www.cisco.com/en/US/products/ps13386/tsd_products_support_series_home.html
Documentation Feedback
To provide technical feedback on this document, or to report an error or omission, please send your comments
to [email protected]. We appreciate your feedback.
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
x
Preface
Related Documentation for Cisco Nexus 9000 Series Switches

Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service
request, and gathering additional information, see What's New in Cisco Product Documentation at: http://
www.cisco.com/c/en/us/td/docs/general/whatsnew/whatsnew.html.
Subscribe to What's New in Cisco Product Documentation, which lists all new and revised Cisco technical
documentation as an RSS feed and delivers content directly to your desktop using a reader application. The
RSS feeds are a free service.
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
xi
Preface
Obtaining Documentation and Submitting a Service Request

Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
xii
Preface
Obtaining Documentation and Submitting a Service Request

CHAPTER 1
New and Changed Information
This chapter provides release-specific information for each new and changed feature in the Cisco Nexus
9000 Series NX-OS VXLAN Configuration Guide.
•New and Changed Information, page 1
New and Changed Information
This table summarizes the new and changed features for the Cisco Nexus 9000 Series NX-OS VXLAN
Configuration Guide and where they are documented.
Table 1: New and Changed Features
Where DocumentedChanged
in
Release
DescriptionFeature
Guidelines and Limitations for
VXLAN, on page 11
7.0(3)I6(1)Added support for configuring
CLI command lacp
vpc-convergence for better
convergence of Layer 2 EVPN
VXLAN.
Configuring new CLI
command lacp
vpc-convergence
Guidelines and Limitations for
VXLAN, on page 11
7.0(3)I6(1)Added support for configuring
VXLAN on Cisco Nexus
3232C and 3264Q switches.
Configuring VXLAN on Cisco
Nexus 3232C and 3264Q
switches
NGOAM Authentication, on
page 99
7.0(3)I6(1)Added support for configuring
NGOAM authentication.
Configuring NGOAM
Authentication
Introduction to Multihoming,
on page 101,Overview of
VLAN Consistency Checking,
on page 124, and Overview of
ESI ARP Suppression, on page
126
7.0(3)I5(2)Added support for VXLAN
EVPN Multihoming, VLAN
Consistency Checking, and ESI
ARP suppression.
Configuring VXLAN EVPN
Multihoming, VLAN
Consistency Checking, and ESI
ARP Suppression
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
1

Where DocumentedChanged
in
Release
DescriptionFeature
Configuring Selective
Q-in-VNI, on page 30
7.0(3)I5(2)Added support for configuring
selective Q-in-VNI.
Configuring Selective
Q-in-VNI
Guidelines and Limitations for
FHRP Over VXLAN, on page
35
7.0(3)I5(2)Added support for configuring
FHRP over VXLAN on the
Cisco Nexus 9200, 9300, and
9300-EX Series switches.
Configuring FHRP over
VXLAN
Guidelines and Limitations for
IGMP Snooping Over
VXLAN, on page 39
7.0(3)I5(2)Added support for configuring
VXLAN IGMP snooping on
Cisco Nexus 9300 Series
switches and Cisco Nexus 9500
Series switches with
N9K-X9732C-EX line cards.
Configuring VXLAN IGMP
Snooping
Configuring FHRP Over
VXLAN, on page 35
7.0(3)I5(1)Added support for configuring
VRRP (FHRP) over VXLAN.
Configuring VRRP (FHRP)
over VXLAN
VXLAN OAM Overview, on
page 91
7.0(3)I5(1)Added support for configuring
VXLAN OAM.
Configuring VXLAN OAM
Configuring IGMP Snooping
Over VXLAN, on page 39
7.0(3)I5(1)Added support for configuring
IGMP Snooping over VXLAN.
Configuring IGMP Snooping
over VXLAN
Guidelines and Limitations for
VXLAN, on page 11
7.0(3)I4(2)Added support for the Cisco
Nexus 93108TC-EX and
93180YC-EX switches and the
X9732C-EX line card.
VXLAN
Commands for BGP EVPN7.0(3)I4(1)Added support for suppress
mac-route command.
Support for suppress
mac-route command
Guidelines and Limitations for
VXLAN
Guidelines and Limitations for
VXLAN BGP EVPN
7.0(3)I4(1)Added support for In Service
Software Upgrade (ISSU).
Support for In Service Software
Upgrade (ISSU)
Notes for EVPN Convergence7.0(3)I3(1)Added improved support for
EVPN convergence
Improved EVPN convergence
support
Verifying the VXLAN
Configuration
7.0(3)I2(2)Added support for displaying
tracking route information.
Tracking route support
Configuring Q-in-VNI with
LACP Tunneling
7.0(3)I2(2)Added support for VXLAN
with LACP tunneling.
LACP tunneling support for
VXLAN
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
2
New and Changed Information
New and Changed Information

Where DocumentedChanged
in
Release
DescriptionFeature
Guidelines and Limitations for
VXLAN
Guidelines and Limitations for
VXLAN BGP EVPN
7.0(3)I2(1)Added VXLAN support for
FEX host interface port.
VXLAN FEX HIF support
Considerations for VXLAN
Deployment
7.0(3)I2(1)Added recommendation for
centralized gateway
Flood and Learn and
centralized gateway support
Configuring Port VLAN
Mapping on a Trunk Port
7.0(3)I2(1)Added support for PV routing.PV Routing support
VXLAN Bud Node Over VPC
Overview
7.0(3)I2(1)Example configuration of
VXLAN bud node over VPC.
VXLAN Bud Node Over VPC
DHCP Relay in VXLAN BGP
EVPN Overview
7.0(3)I2(1)Enables DHCP relay support in
BGP EVPN environment.
BGP EVPN DHCP Relay
support
Configuring Q-in-VNI7.0(3)I2(1)Added support for Q-in-VNI.Q-in-VNI support
Configuring Inner VLAN and
Outer VLAN Mapping on a
Trunk Port
7.0(3)I2(1)Added support for inner VLAN
and outer VLAN mapping on
a trunk port.
Inner VLAN and Outer VLAN
Mapping on a Trunk Port
support
Configuring VXLAN EVPN
Ingress Replication
7.0(3)I1(2)Replicates BUM traffic to
remote VTEP peers that are
learned through the BGP EVPN
control plane on Cisco Nexus
9300 Series switches.
VXLAN EVPN ingress
replication
Configuring Port VLAN
Mapping on a Trunk Port
7.0(3)I1(2)Enables VLAN translation
between the ingress VLAN and
a local VLAN on a port on
Cisco Nexus 9300 Series
switches.
Port VLAN mapping on a trunk
port
vPC Consistency Check for
vPC VTEPs
7.0(3)I1(2)Enables two switches
configured as a vPC pair to
exchange and verify their
configuration compatibility.
vPC Consistency Check for
vPC VTEPs
Configuring Static MAC for
VXLAN VTEP
7.0(3)I1(2)Enables the configuration of
static MAC addresses behind a
peer VTEP on Cisco Nexus
9300 Series switches.
Static MAC for VXLAN VTEP
support
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
3
New and Changed Information
New and Changed Information

Where DocumentedChanged
in
Release
DescriptionFeature
Configuring VXLAN BGP
EVPN
7.0(3)I1(2)Added support for Cisco Nexus
9500 Series switches.
VXLAN BGP EVPN support
Configuring Static Ingress
Replication
7.0(3)I1(1)Enables the replication of BUM
traffic to remote peers.
Ingress replication support
Bud Node Topology7.0(3)I1(1)Enables a VXLAN VTEP
device to also be an IP transit
device.
Bud node topology support
Configuring VXLAN BGP
EVPN
7.0(3)I1(1)Enables the learning of remote
VTEPs, overlay MACs, and
routes through the BGP EVPN
control plane protocol on Cisco
Nexus 9300 Series switches.
VXLAN BGP EVPN support
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
4
New and Changed Information
New and Changed Information

CHAPTER 2
Overview
This chapter contains the following sections:
•VXLAN Overview, page 5
VXLAN Overview
Cisco Nexus 9000 switches are designed for hardware-based VXLAN function. It provides Layer 2 connectivity
extension across the Layer 3 boundary and integrates between VXLAN and non-VXLAN infrastructures.
This can enable virtualized and multitenant data center designs over a shared common physical infrastructure.
VXLAN provides a way to extend Layer 2 networks across Layer 3 infrastructure using MAC-in-UDP
encapsulation and tunneling. VXLAN enables flexible workload placements using the Layer 2 extension. It
can also be an approach to building a multitenant data center by decoupling tenant Layer 2 segments from
the shared transport network.
When deployed as a VXLAN gateway, Cisco Nexus 9000 switches can connect VXLAN and classic VLAN
segments to create a common forwarding domain so that tenant devices can reside in both environments.
VXLAN has the following benefits:
•Flexible placement of multitenant segments throughout the data center.
It provides a way to extend Layer 2 segments over the underlying shared network infrastructure so that
tenant workloads can be placed across physical pods in the data center.
•Higher scalability to address more Layer 2 segments.
VXLAN uses a 24-bit segment ID, the VXLAN network identifier (VNID). This allows a maximum of
16 million VXLAN segments to coexist in the same administrative domain. (In comparison, traditional
VLANs use a 12-bit segment ID that can support a maximum of 4096 VLANs.)
•Utilization of available network paths in the underlying infrastructure.
VXLAN packets are transferred through the underlying network based on its Layer 3 header. It uses
equal-cost multipath (ECMP) routing and link aggregation protocols to use all available paths.
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
5

VXLAN Encapsulation and Packet Format
VXLAN is a Layer 2 overlay scheme over a Layer 3 network. It uses MAC Address-in-User Datagram Protocol
(MAC-in-UDP) encapsulation to provide a means to extend Layer 2 segments across the data center network.
VXLAN is a solution to support a flexible, large-scale multitenant environment over a shared common physical
infrastructure. The transport protocol over the physical data center network is IP plus UDP.
VXLAN defines a MAC-in-UDP encapsulation scheme where the original Layer 2 frame has a VXLAN
header added and is then placed in a UDP-IP packet. With this MAC-in-UDP encapsulation, VXLAN tunnels
Layer 2 network over Layer 3 network.
VXLAN uses an 8-byte VXLAN header that consists of a 24-bit VNID and a few reserved bits. The VXLAN
header together with the original Ethernet frame goes in the UDP payload. The 24-bit VNID is used to identify
Layer 2 segments and to maintain Layer 2 isolation between the segments. With all 24 bits in VNID, VXLAN
can support 16 million LAN segments.
VXLAN Tunnel Endpoint
VXLAN uses VXLAN tunnel endpoint (VTEP) devices to map tenants’end devices to VXLAN segments
and to perform VXLAN encapsulation and de-encapsulation. Each VTEP function has two interfaces: One
is a switch interface on the local LAN segment to support local endpoint communication through bridging,
and the other is an IP interface to the transport IP network.
The IP interface has a unique IP address that identifies the VTEP device on the transport IP network known
as the infrastructure VLAN. The VTEP device uses this IP address to encapsulate Ethernet frames and transmits
the encapsulated packets to the transport network through the IP interface. A VTEP device also discovers the
remote VTEPs for its VXLAN segments and learns remote MAC Address-to-VTEP mappings through its IP
interface.
The VXLAN segments are independent of the underlying network topology; conversely, the underlying IP
network between VTEPs is independent of the VXLAN overlay. It routes the encapsulated packets based on
the outer IP address header, which has the initiating VTEP as the source IP address and the terminating VTEP
as the destination IP address.
VXLAN Packet Forwarding Flow
VXLAN uses stateless tunnels between VTEPs to transmit traffic of the overlay Layer 2 network through the
Layer 3 transport network.
Cisco Nexus 9000 as Hardware-Based VXLAN Gateway
VXLAN is a new technology for virtual data center overlays and is being adopted in data center networks
more and more, especially for virtual networking in the hypervisor for virtual machine-to-virtual machine
communication. However, data centers are likely to contain devices that are not capable of supporting VXLAN,
such as legacy hypervisors, physical servers, and network services appliances, such as physical firewalls and
load balancers, and storage devices, etc. Those devices need to continue to reside on classic VLAN segments.
It is not uncommon that virtual machines in a VXLAN segment need to access services provided by devices
in a classic VLAN segment. This type of VXLAN-to-VLAN connectivity is enabled by using a VXLAN
gateway.
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
6
Overview
VXLAN Encapsulation and Packet Format

A VXLAN gateway is a VTEP device that combines a VXLAN segment and a classic VLAN segment into
one common Layer 2 domain.
A Cisco Nexus 9000 Series Switch can function as a hardware-based VXLAN gateway. It seamlessly connects
VXLAN and VLAN segments as one forwarding domain across the Layer 3 boundary without sacrificing
forwarding performance. The Cisco Nexus 9000 Series eliminates the need for an additional physical or virtual
device to be the gateway. The hardware-based encapsulation and de-encapsulation provides line-rate
performance for all frame sizes.
vPC Consistency Check for vPC VTEPs
The vPC consistency check is a mechanism used by the two switches configured as a vPC pair to exchange
and verify their configuration compatibility. Consistency checks are performed to ensure that NVE
configurations and VN-Segment configurations are identical across vPC peers. This check is essential for the
correct operation of vPC functions.
DescriptionvPC Check TypeParameter
Brings down the affected VLANs on vPC ports on both sides.Type-1-nongracefulVLAN-VNI mapping
Member VNIs must be the same on both nodes. VNIs that are
not common bring down the corresponding VLANs on vPC ports
on both sides. (The attributes considered are mcast group address,
suppress-arp, and Layer 3 VRF VNI.)
Type-1-nongracefulVTEP-Member-VNI
If an emulated IP address is not the same on both nodes, all
gateway vPC ports on one side (secondary) are brought down.
Alternatively, one side of all vPC ports is brought down.
The VTEP source loopback on the vPC secondary is also brought
down if the emulated IP address is not the same on both sides.
Type-1-nongracefulVTEP-emulated IP
The NVE needs to be in the oper UP state on both sides for the
vPC consistency check.
If both VTEPs are not in the OPER_UP state, the secondary leg
is brought down along with the VTEP source loopback on the
vPC secondary.
Type-1-nongracefulNVE Oper State
The vPC on both sides must be configured with the same
host-reachability protocol. Otherwise, the secondary leg is
brought down along with the VTEP source loopback on the vPC
secondary.
Type-1-nongracefulNVE
Host-Reachability
Protocol
VLAN-to-VXLAN VN-segment mapping is a type-1 consistency check parameter. The two VTEP switches
are required to have identical mappings. VLANs that have mismatched VN-segment mappings will be
suspended. When the graceful consistency check is disabled and problematic VLANs arise, the primary vPC
switch and the secondary vPC switch will suspend the VLANs.
The following situations are detected as inconsistencies:
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
7
Overview
vPC Consistency Check for vPC VTEPs

•One switch has a VLAN mapped to a VN-segment (VXLAN VNI), and the other switch does not have
a mapping for the same VLAN.
•The two switches have a VLAN mapped to different VN-segments.
Beginning with 7.0(3)I1(2), each VXLAN VNI must have the same configuration. However, when
configuring with ingress-replication protocol static, the list of static peer IP addresses are not checked
as part of the consistency check.
Note
The following is an example of displaying vPC information:
sys06-tor3# sh vpc consistency-parameters global
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
Vlan to Vn-segment Map 1 1024 Relevant Map(s) 1024 Relevant Map(s)
STP Mode 1 MST MST
STP Disabled 1 None None
STP MST Region Name 1 "" ""
STP MST Region Revision 1 0 0
STP MST Region Instance to 1
VLAN Mapping
STP Loopguard 1 Disabled Disabled
STP Bridge Assurance 1 Enabled Enabled
STP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,
BPDUFilter, Edge BPDUGuard Disabled Disabled
STP MST Simulate PVST 1 Enabled Enabled
Nve Oper State, Secondary 1 Up, 4.4.4.4 Up, 4.4.4.4
IP
Nve Vni Configuration 1 10002-11025 10002-11025
Allowed VLANs - 1-1025 1-1025
Local suspended VLANs - - -
Static Ingress Replication
VXLAN uses flooding and dynamic MAC address learning to transport broadcast, unknown unicast, and
multicast traffic. VXLAN forwards these traffic types using a multicast forwarding tree or ingress replication.
With static ingress replication:
•Remote peers are statically configured.
•Multi-destination packets are unicast encapsulated and delivered to each of the statically configured
remote peers.
Cisco NX-OS supports multiple remote peers in one segment and also allows the same remote peer in
multiple segments.
Note
Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x
8
Overview
Static Ingress Replication