Cisco Nexus 1000V User manual

Cisco Systems, Inc.
www.cisco.com
Cisco has more than 200 offices worldwide.
Addresses, phone numbers, and fax numbers
are listed on the Cisco website at
www.cisco.com/go/offices.
Cisco Nexus 1000V Troubleshooting
Guide, Release 5.2(1)SV3(1.1)
May 16, 2016
Text Part Number: OL-31393-01

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this
URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1110R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the
document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
© 2016 Cisco Systems, Inc. All rights reserved.

iii
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
CONTENTS
New and Changed Information ii-xiii
Preface xvii
Audience xvii
Document Conventions xvii
Related Documentation xviii
Obtaining Documentation and Submitting a Service Request xx
CHAPTER 1Overview 1-1
Troubleshooting Process 1-1
Best Practices 1-1
Troubleshooting Basics 1-2
Troubleshooting Guidelines 1-2
Gathering Information 1-2
Verifying Ports 1-3
Verifying Layer 2 Connectivity 1-3
Verifying Layer 3 Connectivity 1-3
Overview of Symptoms 1-4
System Messages 1-4
System Message Text 1-4
syslog Server Implementation 1-5
Troubleshooting with Logs 1-6
Viewing Logs 1-6
Cisco Support Communities 1-7
Contacting Cisco or VMware Customer Support 1-7
CHAPTER 2Troubleshooting Tools 2-1
Commands 2-1
Ping 2-1
Traceroute 2-2
Monitoring Processes and CPUs 2-2
Identifying the Running Processes and their States 2-2

Contents
iv
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
Displaying CPU Utilization 2-3
Displaying CPU and Memory Information 2-4
RADIUS 2-4
Syslog 2-5
CHAPTER 3Installation 3-1
Isolating Installation Problems 3-1
Verifying Your VMware License Version 3-1
Host is Not Visible from the Distributed Virtual Switch 3-2
Refreshing the vCenter Server Connection 3-3
Improving Performance on the ESX and VM 3-4
Verifying the Domain Configuration 3-4
Verifying the Port Group Assignments for a VSM VM Virtual Interface 3-4
Verifying VSM and vCenter Server Connectivity 3-5
Troubleshooting Connections to vCenter Server 3-5
Recovering the Network Administrator Password 3-6
Managing Extension Keys 3-6
Known Extension Problems and Resolutions 3-7
Resolving a Plug-In Conflict 3-7
Finding the Extension Key on the Cisco Nexus 1000V 3-7
Finding the Extension Key Tied to a Specific DVS 3-8
Verifying Extension Keys 3-8
Recreating the Cisco Nexus 1000V Installation 3-10
Removing Hosts from the Cisco Nexus 1000V DVS 3-11
Removing the Cisco Nexus 1000V from the vCenter Server 3-11
Unregistering the Extension Key in the vCenter Server 3-12
Problems with the Cisco Nexus 1000V Installation Management Center 3-14
CHAPTER 4Licenses 4-1
Information About Licenses 4-1
Contents of the License File 4-2
Prerequisites to License Troubleshooting 4-2
Problems with Licenses 4-3
License Troubleshooting Commands 4-4
CHAPTER 5Upgrades 5-1
Information About Upgrades 5-1

Contents
v
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
Problems with the In-Service Software Upgrade 5-1
Problems with the VEM Upgrade 5-5
Problems with the GUI Upgrade 5-6
Recovering a Secondary VSM with Active Primary 5-7
Recovering a Primary VSM with Active Secondary 5-12
Problems with VSM-VEM Layer 2 to 3 Conversion Tool 5-16
Upgrade Troubleshooting Commands 5-16
CHAPTER 6High Availability 6-1
Information About High Availability 6-1
System-Level High Availability 6-2
Network-Level High Availability 6-2
Problems with High Availability 6-2
High Availability Troubleshooting Commands 6-5
CHAPTER 7VSM and VEM Modules 7-1
Information About Modules 7-1
Troubleshooting a Module Not Coming Up on the VSM 7-1
Guidelines for Troubleshooting Modules 7-2
Flowchart for Troubleshooting Modules 7-3
Problems with the VSM 7-4
Verifying the VSM Is Connected to vCenter Server 7-6
Verifying the VSM Is Configured Correctly 7-7
Checking the vCenter Server Configuration 7-10
Checking Network Connectivity Between the VSM and the VEM 7-10
Recovering Management and Control Connectivity of a Host When a VSM is Running on a VEM 7-12
Checking the VEM Configuration 7-14
Collecting Logs 7-16
VSM and VEM Troubleshooting Commands 7-17
CHAPTER 8L3Sec 8-1
Troubleshooting L3Sec 8-1
CHAPTER 9Ports 9-1
Information About Ports 9-1
Information About Interface Characteristics 9-1
Information About Interface Counters 9-2
Information About Link Flapping 9-2

Contents
vi
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
Information About Port Security 9-2
Port Diagnostic Checklist 9-3
Problems with Ports 9-3
Cannot Enable an Interface 9-4
Port Link Failure or Port Not Connected 9-4
Link Flapping 9-5
Port ErrDisabled 9-6
VM Cannot Ping a Secured Port 9-7
Port Security Violations 9-8
Port State is Blocked on a VEM 9-9
Port Troubleshooting Commands 9-9
CHAPTER 10 Port Profiles 10-1
Information About Port Profiles 10-1
Problems with Port Profiles 10-2
Recovering a Quarantined Offline Interface 10-4
Port Profile Logs 10-5
Port Profile Troubleshooting Commands 10-5
CHAPTER 11 Port Channels and Trunking 11-1
Information About Port Channels and Trunking 11-1
Port Channel Overview 11-1
Port Channel Restriction 11-2
Trunking Overview 11-2
Initial Troubleshooting Checklist 11-2
Troubleshooting Asymmetric Port Channels 11-3
Cannot Create Port Channel 11-4
Newly Added Interface Does Not Come Online In a Port Channel 11-4
Forcing Port Channel Characteristics onto an Interface 11-4
Verifying a Port Channel Configuration 11-5
VLAN Traffic Does Not Traverse Trunk 11-5
CHAPTER 12 Layer 2 Switching 12-1
Information About Layer 2 Ethernet Switching 12-1
Port Model 12-1
Viewing Ports from the VEM 12-2
Viewing Ports from the VSM 12-3
Port Types 12-4

Contents
vii
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
Layer 2 Switching Problems 12-4
Verifying a Connection Between VEM Ports 12-4
Verifying a Connection Between VEMs 12-5
Isolating Traffic Interruptions 12-6
Layer 2 Switching Troubleshooting Commands 12-7
Troubleshooting Microsoft NLB Unicast Mode 12-12
Limitations and Restrictions 12-12
Disabling Automatic Static MAC Learning on a vEthernet Interface 12-12
Checking Status on a VSM 12-12
Checking the Status on a VEM 12-13
Configuring MS NLB for Multiple VM NICs in the Same Subnet 12-13
Troubleshooting BPDU Guard 12-14
BPDU Guard Troubleshooting Commands 12-14
CHAPTER 13 VLANs 13-1
Information About VLANs 13-1
Initial Troubleshooting Checklist 13-2
Cannot Create a VLAN 13-3
CHAPTER 14 Private VLANs 14-1
Information About Private VLANs 14-1
Private VLAN Domains 14-1
Spanning Multiple Switches 14-1
Private VLAN Ports 14-2
Troubleshooting Guidelines 14-2
Private VLAN Troubleshooting Commands 14-2
CHAPTER 15 NetFlow 15-1
Information About NetFlow 15-1
NetFlow Troubleshooting Commands 15-1
Common NetFlow Problems 15-2
Debugging a Policy Verification Error 15-3
Debugging Statistics Export 15-3
CHAPTER 16 ACLs 16-1
Information About Access Control Lists 16-1
ACL Configuration Limits 16-1

Contents
viii
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
ACL Restrictions 16-2
ACL Troubleshooting Commands 16-2
Displaying ACL Policies on the VEM 16-2
Debugging Policy Verification Issues 16-3
Troubleshooting ACL Logging 16-3
Using the CLI to Troubleshoot ACL Logging on a VEM 16-4
ACL Logging Troubleshooting Scenarios 16-5
CHAPTER 17 Quality of Service 17-1
Information About Quality of Service 17-1
QoS Configuration Limits 17-1
QoS Troubleshooting Commands 17-2
Troubleshooting the VEM 17-3
Debugging Policy Configuration Errors 17-4
Debugging Policy Verification Failures 17-5
Debugging Policing Configuration Errors 17-6
CHAPTER 18 SPAN 18-1
Information About SPAN 18-1
SPAN Session Guidelines 18-1
Problems with SPAN 18-2
SPAN Troubleshooting Commands 18-3
CHAPTER 19 Multicast IGMP 19-1
Information About Multicast 19-1
Multicast IGMP Snooping 19-1
Problems with Multicast IGMP Snooping 19-2
Troubleshooting Guidelines 19-2
IGMP Snooping Debugging Commands 19-2
Multicast IGMP Snooping Troubleshooting Commands 19-6
Problems with Multicast IGMP Snooping 19-9
CHAPTER 20 DHCP, DAI, and IPSG 20-1
Information About DHCP Snooping 20-1
Information About Dynamic ARP Inspection 20-2
Information About IP Source Guard 20-2
Guidelines and Limitations for Troubleshooting 20-2

Contents
ix
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
Problems with DHCP Snooping 20-3
Troubleshooting Dropped ARP Responses 20-4
Problems with IP Source Guard 20-5
Collecting and Evaluating Logs 20-5
VSM Logging 20-5
Host Logging 20-6
DHCP, DAI, and IPSG Troubleshooting Commands 20-6
CHAPTER 21 Storm Control 21-1
Information About Storm Control 21-1
Troubleshooting Storm Control 21-1
Troubleshooting VSM Commands 21-1
Troubleshooting VEM Commands 21-1
Debugging Storm Control on a VEM 21-2
CHAPTER 22 System 22-1
Information About the System 22-1
General Restrictions for vCenter Server 22-2
Extension Key 22-2
Recovering a DVS 22-2
Recovering a DVS With a Saved Copy of the VSM 22-3
Recovering a DVS Without a Saved Copy of the VSM 22-4
Problems Related to VSM and vCenter Server Connectivity 22-5
Connection Failure After ESX Reboot 22-6
Setting the System MTU 22-7
Recovering Lost Connectivity Due to MTU Mismatch 22-8
VSM Creation 22-9
Port Profiles 22-9
Problems with Port Profiles 22-10
Problems with Hosts 22-10
Problems with VM Traffic 22-10
VEM Troubleshooting Commands 22-11
VEM Log Commands 22-12
Error Messages 22-12
CHAPTER 23 Before Contacting Technical Support 23-1
Cisco Support Communities 23-1

Contents
x
Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)
OL-31593-01
Gathering Information for Technical Support 23-1
Obtaining a File of Core Memory Information 23-2
Copying Files 23-3
CHAPTER 24 Network Segmentation Manager 24-1
Information About Network Segmentation Manager 24-1
Problems with Network Segmentation Manager 24-2
Network Segmentation Manager Troubleshooting Commands 24-7
CHAPTER 25 VXLANs 25-1
Information About VXLANs 25-1
Overview 25-1
VXLAN Tunnel EndPoint 25-2
VXLAN Gateway 25-2
VXLAN Trunks 25-3
VXLAN Border Gateway Protocol Control Plane 25-3
Multi-MAC Capability 25-8
Fragmentation 25-8
Scalability 25-8
Supported Features 25-9
VXLAN Troubleshooting Commands 25-9
VSM Commands 25-9
VXLAN Gateway Commands 25-11
VEM Commands 25-14
VEM Packet Path Debugging 25-16
VEM Multicast Debugging 25-17
VXLAN Data Path Debugging 25-18
vemlog Debugging 25-18
Vempkt 25-19
Statistics 25-20
show Commands 25-20
CHAPTER 26 VSI Discovery and Configuration Protocol 26-1
Information About VDP 26-1
Problems with VDP 26-2
VDP Troubleshooting Commands 26-2
VSM Commands 26-2
VEM Commands 26-4