Cisco Nexus 1000V User manual

Send document comments to nexus1k-docfeedbac[email protected].
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco Nexus 1000V Troubleshooting
Guide, Release 4.2(1)SV2(2.1)
April 13, 2015
Text Part Number: OL-28795-01

Send document comments to nexus1k-docfeedbac[email protected].
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this
URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1110R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the
document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
© 2013 Cisco Systems, Inc. All rights reserved.

Send document comments to nexus1k-docfeedbac[email protected].
iii
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
CONTENTS
New and Changed Information xiii
Preface xv
Audience xv
Document Conventions xv
Related Documentation xvi
Obtaining Documentation and Submitting a Service Request xviii
xviii
Overview of Troubleshooting 1-1
Overview of the Troubleshooting Process 1-1
Overview of Best Practices 1-1
Troubleshooting Basics 1-2
Troubleshooting Guidelines 1-2
Gathering Information 1-2
Verifying Ports 1-3
Verifying Layer 2 Connectivity 1-3
Verifying Layer 3 Connectivity 1-3
Overview of Symptoms 1-4
System Messages 1-4
System Message Text 1-4
Syslog Server Implementation 1-5
Troubleshooting with Logs 1-6
Viewing Logs 1-6
Cisco Support Communities 1-7
Contacting Cisco or VMware Customer Support 1-7
Tools Used in Troubleshooting 2-1
Commands 2-1
Ping 2-1
Traceroute 2-2
Monitoring Processes and CPUs 2-2
Identifying the Processes Running and their States 2-2
Displaying CPU Utilization 2-3

Send document comments to nexus1k-docfeedbac[email protected].
Contents
iv
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
Displaying CPU and Memory Information 2-4
RADIUS 2-4
Syslog 2-5
Logging Levels 2-5
Enabling Logging for Telnet or SSH 2-6
Installation 3-1
Isolating Installation Problems 3-1
Verifying Your VMware License Version 3-1
Host is Not Visible from Distributed Virtual Switch 3-2
Refreshing the vCenter Server Connection 3-3
Improving Performance 3-4
Verifying the Domain Configuration 3-4
Verifying the Port Group Assignments for a VSM VM Virtual Interface 3-4
Verifying VSM and vCenter Server Connectivity 3-5
Troubleshooting Connections to a vCenter Server 3-5
Recovering the Network Administrator Password 3-6
Managing Extension Keys 3-6
Known Extension Problems and Resolutions 3-7
Resolving a Plug-In Conflict 3-7
Finding the Extension Key on the Cisco Nexus 1000V 3-7
Finding the Extension Key Tied to a Specific DVS 3-8
Verifying Extension Keys 3-9
Recreating the Cisco Nexus 1000V Installation 3-10
Removing the Hosts from the Cisco Nexus 1000V DVS 3-11
Removing the Cisco Nexus 1000V From the vCenter Server 3-11
Unregister the Extension Key in the vCenter Server 3-12
Problems with the Nexus 1000V Installation Management Center 3-13
Licenses 4-1
Information About Licenses 4-1
Contents of the License File 4-2
Prerequisites to License Troubleshooting 4-2
Problems with Licenses 4-3
License Troubleshooting Commands 4-4
Upgrade 5-1
Information about Upgrades 5-1
Problems with the In Service Software Upgrade 5-1

Send document comments to nexus1k-docfeedbac[email protected].
Contents
v
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
Problems with the VEM Upgrade 5-5
Problems with the GUI Upgrade 5-6
Recovering a Secondary VSM with Active Primary 5-8
Stopping a VSM Upgrade 5-9
Changing Boot Variables 5-10
Powering On the VSM 5-11
Changing the HA Role 5-12
Recovering a Primary VSM with Active Secondary 5-13
Disconnecting the Port Groups 5-13
Powering Off the VSM 5-15
Connecting the Port Groups 5-15
Problems with VSM-VEM Layer 2 to 3 Conversion Tool 5-17
Upgrade Troubleshooting Commands 5-17
High Availability 6-1
Information About High Availability 6-1
System-Level High Availability 6-2
Network-Level High Availability 6-2
Problems with High Availability 6-2
High Availability Troubleshooting Commands 6-5
VSM and VEM Modules 7-1
Information About Modules 7-1
Troubleshooting a Module Not Coming Up on the VSM 7-1
Guidelines for Troubleshooting Modules 7-2
Flow Chart for Troubleshooting Modules 7-3
Problems with the VSM 7-4
Verifying the VSM Is Connected to the vCenter Server 7-6
Verifying the VSM Is Configured Correctly 7-7
Verifying the Domain Configuration 7-7
Verifying the System Port Profile Configuration 7-8
Verifying the Control and Packet VLAN Configuration 7-8
Checking the vCenter Server Configuration 7-10
Checking Network Connectivity Between the VSM and the VEM 7-10
Recovering Management and Control Connectivity of a Host when a VSM is Running on a VEM 7-12
Using the VEM Connect Script 7-12
Checking the VEM Configuration 7-14
Collecting Logs 7-17
VSM and VEM Troubleshooting Commands 7-18

Send document comments to nexus1k-docfeedbac[email protected].
Contents
vi
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
Ports 8-1
Information About Ports 8-1
Information About Interface Characteristics 8-1
Information About Interface Counters 8-2
Information About Link Flapping 8-2
Information About Port Security 8-2
Port Diagnostic Checklist 8-3
Problems with Ports 8-3
Cannot Enable an Interface 8-4
Port Link Failure or Port Not Connected 8-4
Link Flapping 8-5
Port ErrDisabled 8-5
VM Cannot Ping a Secured Port 8-6
Port Security Violations 8-7
Port Troubleshooting Commands 8-8
Port Profiles 9-1
Information About Port Profiles 9-1
Problems with Port Profiles 9-2
Recovering a Quarantined Offline Interface 9-5
Port Profile Logs 9-6
Port Profile Troubleshooting Commands 9-6
Port Channels and Trunking 10-1
Overview 10-1
Port Channel Overview 10-1
Port Channel Restriction 10-2
Trunking Overview 10-2
Initial Troubleshooting Checklist 10-2
Troubleshooting Asymmetric Port Channels 10-3
Cannot Create Port Channel 10-4
Newly Added Interface Does Not Come Online In a Port Channel 10-4
Forcing Port Channel Characteristics onto an Interface 10-4
Verifying a Port Channel Configuration 10-5
VLAN Traffic Does Not Traverse Trunk 10-5
Layer 2 Switching 11-1
Information About Layer 2 Ethernet Switching 11-1
Port Model 11-1

Send document comments to nexus1k-docfeedbac[email protected].
Contents
vii
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
Viewing Ports from the VEM 11-2
Viewing Ports from the VSM 11-3
Port Types 11-4
Layer 2 Switching Problems 11-4
Verifying a Connection Between VEM Ports 11-4
Verifying a Connection Between VEMs 11-5
Isolating Traffic Interruptions 11-6
Verifying Layer 2 Switching 11-7
Troubleshooting Microsoft NLB Unicast Mode 11-12
Limitations and Restrictions 11-12
Disabling Automatic Static MAC Learning on vEthernet 11-12
Checking Status on a VSM 11-12
Checking Status on a VEM 11-13
Configuring MS NLB for Multiple VM NICs in the Same Subnet 11-14
Enabling UUFB 11-14
Disabling UUFB for VMs that use Dynamic MAC Addresses 11-14
VLANs 12-1
Information About VLANs 12-1
Initial Troubleshooting Checklist 12-2
Cannot Create a VLAN 12-3
Private VLANs 13-1
Information About Private VLANs 13-1
Private VLAN Domain 13-1
Spanning Multiple Switches 13-1
Private VLAN Ports 13-2
Troubleshooting Guidelines 13-2
Private VLAN Troubleshooting Commands 13-2
NetFlow 14-1
Information About NetFlow 14-1
NetFlow Troubleshooting Commands 14-2
Common NetFlow Problems 14-3
Debugging a Policy Verification Error 14-3
Debugging Statistics Export 14-3
ACLs 15-1
About Access Control Lists (ACLs) 15-1
ACL Configuration Limits 15-1

Send document comments to nexus1k-docfeedbac[email protected].
Contents
viii
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
ACL Restrictions 15-2
Troubleshooting ACLs 15-2
Displaying ACL Policies on the VEM 15-2
Debugging Policy Verification Issues 15-3
Troubleshooting ACL Logging 15-3
Using the CLI to Troubleshoot ACL Logging on a VEM 15-4
Viewing Current Flows 15-4
Viewing Active Flows 15-4
Flushing All ACL Flows 15-4
Showing Flow Debug Statistics 15-5
ACL Logging Troubleshooting Scenarios 15-5
Troubleshooting a Syslog Server Configuration 15-5
Troubleshooting an ACL Rule That Does Not Have a Log Keyword 15-6
Troubleshooting a Maximum Flow Limit Value That is Too Low 15-6
Troubleshooting a Mismatched Configuration between a VSM and a VEM 15-7
Quality of Service 16-1
Information About Quality of Service 16-1
QoS Configuration Limits 16-1
QoS Troubleshooting Commands 16-2
Troubleshooting the VEM 16-2
Debugging Policing Verification Errors 16-3
SPAN 17-1
Information About SPAN 17-1
SPAN Session Guidelines 17-1
Problems with SPAN 17-2
SPAN Troubleshooting Commands 17-3
Multicast IGMP 18-1
Information About Multicast 18-1
Multicast IGMP Snooping 18-1
Problems with Multicast IGMP Snooping 18-2
Troubleshooting Guidelines 18-2
Troubleshooting Commands 18-2
Symptoms, Causes, and Solutions 18-4
DHCP, DAI, and IPSG 19-1
Information About DHCP Snooping 19-1
Information About Dynamic ARP Inspection 19-2

Send document comments to nexus1k-docfeedbac[email protected].
Contents
ix
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
Information About IP Source Guard 19-2
Guidelines and Limitations for Troubleshooting 19-2
Problems with DHCP Snooping 19-3
Troubleshooting Dropped ARP Responses 19-4
Problems with IP Source Guard 19-5
Collecting and Evaluating Logs 19-5
VSM Logging 19-5
Host Logging 19-6
DHCP, DAI, and IPSG Troubleshooting Commands 19-6
Virtual Service Domain 20-1
Information about Virtual Service Domain 20-1
Problems with Virtual Service Domain 20-1
Collecting and Evaluating Logs 20-2
Virtual Service Domain Troubleshooting Commands 20-3
System 21-1
Information About the System 21-1
General Restrictions for vCenter Server 21-2
Extension Key 21-2
Recovering a DVS 21-2
Recovering a DVS With a Saved Copy of the VSM 21-3
Recovering a DVS Without a Saved Copy of the VSM 21-4
Problems Related to VSM and vCenter Server Connectivity 21-5
Connection Failure After ESX Reboot 21-6
Setting the System MTU 21-7
Recovering Lost Connectivity Due to MTU Mismatch 21-8
VSM Creation 21-9
Port Profiles 21-9
Problems with Port Profiles 21-10
Problems with Hosts 21-10
Problems with VM Traffic 21-10
VEM Troubleshooting Commands 21-11
VEM Log Commands 21-12
Error Messages 21-12
Network Segmentation Manager 22-1
Information About Network Segmentation Manager 22-1

Send document comments to nexus1k-docfeedbac[email protected].
Contents
x
Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.1)
OL-28795-01
Problems with Network Segmentation Manager 22-2
Network Segmentation Manager Troubleshooting Commands 22-7
VXLANs 23-1
Information About VXLANs 23-1
Overview 23-1
VXLAN Tunnel EndPoint 23-2
VXLAN Gateway 23-2
VXLAN Trunks 23-2
Multi-MAC Capability 23-3
Fragmentation 23-3
Scalability 23-3
Maximum Number of VXLANs 23-3
Supported Features 23-3
Jumbo Frames 23-3
Disabling the VXLAN Feature Globally 23-4
VXLAN Troubleshooting Commands 23-4
VSM Commands 23-4
VXLAN Gateway Commands 23-5
VEM Commands 23-7
VEM Packet Path Debugging 23-9
VEM Multicast Debugging 23-10
VXLAN Datapath Debugging 23-11
Vemlog Debugging 23-11
Vempkt 23-12
Statistics 23-12
Show Commands 23-13
Cisco TrustSec 24-1
Information About Cisco TrustSec 24-1
Guidelines and Limitations for Troubleshooting Cisco TrustSec 24-1
Cisco TrustSec Troubleshooting Commands 24-2
Debugging Commands 24-2
Host Logging Commands 24-3
Example 24-3
Show Commands 24-4
Problems with Cisco TrustSec 24-4
vCenter Plug-in 25-1
vCenter Plug-in Overview 25-1