Cisco 9134 - MDS Multilayer Fabric Switch User manual

Send documentation comments to mdsfeedbac[email protected]
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco MDS 9000 Family
Troubleshooting Guide, Release 3.x
Cisco MDS SAN-OS for Release 3.3(1)
May 2008
Text Part Number: OL-9285-05

Send documentation comments to mdsfeedbac[email protected]
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
CCDE, CCENT, Cisco Eos, Cisco HealthPresence, the Cisco logo, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, DCE, and Welcome
to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS,
Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS,
Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step,
Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone,
MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase,
SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of
Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0812R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the
document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
© 2009 Cisco Systems, Inc. All rights reserved.

Send documentation comments to mdsfeedbac[email protected]
iii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
CONTENTS
New and Changed Information xxv
Preface xxvii
Document Organization i-xxvii
Document Conventions i-xxix
Related Documentation i-xxx
Release Notes i-xxx
Compatibility Information i-xxx
Regulatory Compliance and Safety Information i-xxx
Hardware Installation i-xxx
Cisco Fabric Manager i-xxxi
Command-Line Interface i-xxxi
Troubleshooting and Reference i-xxxi
Installation and Configuration Note i-xxxi
Obtaining Documentation, Obtaining Support, and Security Guidelines i-xxxii
CHAPTER 1Troubleshooting Overview 1-1
Overview of the Troubleshooting Process 1-1
Best Practices 1-2
Troubleshooting Basics 1-2
General Steps 1-2
Gathering Information Using Common Fabric Manager Tools and CLI Commands 1-3
Common Fabric Manager Tools 1-3
Common CLI Commands 1-4
Verifying Basic Connectivity 1-4
Verifying SAN Element Registration 1-5
Fibre Channel End-to-End Connectivity 1-5
Fabric Issues 1-5
Port Issues 1-6
Primary Troubleshooting Flowchart 1-9
System Messages 1-10
System Message Text 1-10
Syslog Server Implementation 1-10
Implementing Syslog with Fabric Manager 1-11

Send documentation comments to mdsfeedbac[email protected]
Contents
iv
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Implementing Syslog with the CLI 1-12
Troubleshooting with Logs 1-13
Viewing Logs with Fabric Manager 1-13
Viewing Logs with the CLI 1-13
Viewing the Log from the Supervisor 1-14
Viewing NVRAM logs 1-14
Contacting Customer Support 1-15
CHAPTER 2Troubleshooting Installs, Upgrades, and Reboots 2-1
Overview 2-1
Guidelines 2-2
Guidelines for Installations 2-2
Guidelines for Upgrading 2-2
Guidelines for Reboots 2-4
Disruptive Module Upgrades 2-4
Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4
Troubleshooting Fabric Manager Installations 2-5
Verifying Cisco SAN-OS Software Installations 2-6
Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7
Software Installation Reports an Incompatibility 2-7
Diagnosing Compatibility Issues 2-7
Software Installation Ends with Error 2-9
Installing SAN-OS Software Using Fabric Manager 2-10
Installing Cisco SAN-OS Software from the CLI 2-11
Troubleshooting Cisco SAN-OS Software System Reboots 2-13
Power On or Switch Reboot Hangs 2-13
Corrupted Bootflash Recovery 2-14
Recovery Using BIOS Setup for Supervisor-1 2-16
Recovery from the loader> Prompt on Supervisor-2 Modules 2-19
Recovery from the loader> Prompt on Supervisor-1 Modules 2-20
Recovery from the switch(boot)# Prompt 2-21
Recovery for Switches with Dual Supervisor Modules 2-22
Recovering One Supervisor Module With Corrupted Bootflash 2-23
Recovering Both Supervisor Modules With Corrupted Bootflash 2-24
Recognizing Error States 2-25
Switch or Process Resets 2-26
Recoverable System Restarts 2-27
Unrecoverable System Restarts 2-31

Send documentation comments to mdsfeedbac[email protected]
Contents
v
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Recovering the Administrator Password 2-32
Miscellaneous Software Image Issues 2-32
All Ports Down Because of System Health Failure 2-33
Switch Reboots after FCIP Reload 2-33
FCIP Link Fails to Come Up 2-33
Cannot Create, Modify, or Delete Admin Role 2-34
FC IDs Change after Link Reset 2-34
Switch Displays Wrong User 2-34
CHAPTER 3Managing Storage Services Modules 3-1
SSM Overview 3-1
Initial Troubleshooting Checklist 3-3
Common Troubleshooting Tools in Fabric Manager 3-3
Common Troubleshooting Commands in the CLI 3-3
SSM Issues 3-3
SSM Fails to Boot 3-4
Upgrading the SSI Image 3-4
Verifying the SSI Boot Image 3-5
Using the install ssi Command 3-6
Recovering a Replacement SSM 3-8
SSM Upgrade Is Disruptive 3-9
Installing EPLD Images on Modules 3-9
CHAPTER 4Troubleshooting Hardware 4-1
Overview 4-1
SNMP Traps 4-2
Troubleshooting Startup Issues 4-2
Troubleshooting Power Supply Issues 4-3
All Power Supply LEDS Are Off 4-4
Power Supply Input Ok LED is Red 4-5
Power Supply Output Failed LED is On 4-6
Power Supply Fan Ok LED is Red 4-6
Troubleshooting the Power Supplies 4-7
Troubleshooting Fan Issues 4-8
Fan Is Not Spinning 4-8
Fan Is Spinning; Fan LED is Red 4-8
Troubleshooting a Fan Failure Using Device Manager 4-9
Troubleshooting a Fan Failure Using the CLI 4-10

Send documentation comments to mdsfeedbac[email protected]
Contents
vi
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Temperature Threshold Violations 4-11
Troubleshooting Clock Module Issues 4-12
Troubleshooting Other Hardware Issues 4-13
Troubleshooting Supervisor Issues 4-14
Active Supervisor Reboots 4-15
Standby Supervisor Not Recognized by Active Supervisor 4-17
Verifying That a Standby Supervisor Failed to Synchronize Using the CLI 4-17
Standby Supervisor Stays in Powered-Up State 4-19
Verifying That a Standby Supervisor Is in the Powered-Up State Using Device Manager 4-20
Verifying That a Standby Supervisor Is in Powered-Up State Using the CLI 4-20
Troubleshooting Supervisor Modules 4-20
Troubleshooting Switching and Services Modules 4-21
Overview of Module Status 4-21
Module Initialization Overview 4-22
Module Bootup 4-23
Image Download 4-23
Runtime Diagnostics 4-24
Runtime Configuration 4-24
Online and Operational 4-24
Analyzing The Logs 4-25
Troubleshooting Module Issues 4-25
Troubleshooting Powered-Down Modules 4-26
Diagnosing a Powered-Down Module 4-28
Troubleshooting Reloaded Modules 4-31
Diagnosing a Reloaded Module 4-33
Troubleshooting Modules in an Unknown State 4-34
Diagnosing a Module in the Unknown State 4-34
Troubleshooting Modules Not Detected by the Supervisor 4-35
Diagnosing a Module Not Detected by the Supervisor 4-35
Reinitializing a Failed Module Using Fabric Manager 4-36
Reinitializing a Failed Module Using the CLI 4-37
Module Resets 4-38
CHAPTER 5Troubleshooting Mixed Generation Hardware 5-1
Overview 5-1
Port Groups 5-2
Port Speed Mode 5-3
Dynamic Bandwidth Management 5-3
Out-of-Service Interfaces 5-4

Send documentation comments to mdsfeedbac[email protected]
Contents
vii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Port Index Availability 5-4
Combining Modules and Supervisors 5-7
Initial Troubleshooting Checklist 5-7
Generation 1 and Generation 2 Issues 5-7
Module Does Not Come Online 5-8
Verifying Port Index Allocation Using Device Manager 5-8
Verifying Port Index Allocation Using the CLI 5-9
Cannot Configure Port in Dedicated Mode 5-10
Verifying Bandwidth Utilization in a Port Group Using Device Manager 5-11
Verifying Bandwidth Utilization in a Port Group Using the CLI 5-12
Cannot Enable a Port 5-13
Cannot Upgrade Supervisor System Image 5-13
Selecting the Correct Software Images 5-13
CHAPTER 6Troubleshooting Licensing 6-1
License Overview 6-1
Chassis Serial Numbers 6-2
Grace Period 6-2
Initial Troubleshooting Checklist 6-3
Displaying License Information Using Fabric Manager 6-3
Displaying License Information Using Device Manager 6-4
Displaying License Information Using Fabric Manager Web Client 6-4
Displaying License Information Using the CLI 6-4
Licensing Installation Issues 6-6
One-Click License Install Fails or Cannot Connect to Licensing Website 6-6
Serial Number Issues 6-6
RMA Chassis Errors or License Transfers Between Switches 6-7
Receiving Grace Period Warnings After License Installation 6-7
Incorrect Number of Licenses in Use for Multiple Modules 6-8
Grace Period Alerts 6-9
Checking in the Fabric Manager Server License From Device Manager 6-10
License Listed as Missing 6-11
CHAPTER 7Troubleshooting Cisco Fabric Services 7-1
Overview 7-1
Initial Troubleshooting Checklist 7-2
Verifying CFS Using Fabric Manager 7-3
Verifying CFS Using the CLI 7-3

Send documentation comments to mdsfeedbac[email protected]
Contents
viii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Merge Failure Troubleshooting 7-5
Recovering from a Merge Failure with Fabric Manager 7-6
Recovering from a Merge Failure with the CLI 7-6
Lock Failure Troubleshooting 7-7
Resolving Lock Failure Issues Using Fabric Manager 7-7
Resolving Lock Failure Issues Using the CLI 7-8
System State Inconsistent and Locks Being Held 7-9
Clearing Locks Using Fabric Manager 7-9
Clearing Locks Using the CLI 7-9
Distribution Status Verification 7-9
Verifying Distribution Using Fabric Manager 7-9
Verifying Distribution Using the CLI 7-9
CFS Regions Troubleshooting 7-10
Distribution Failure 7-10
Regions for Conditional Service 7-11
Changing Regions 7-11
CHAPTER 8Troubleshooting Ports 8-1
Overview 8-1
Initial Troubleshooting Checklist 8-1
Limitations and Restrictions 8-4
Overview of the FC-MAC Driver and the Port Manager 8-4
Port Manager Overview 8-5
Troubleshooting Port States with the Device Manager 8-6
Device View 8-6
Device Manager: Summary View 8-6
Device Manager: Port Selection 8-7
Device Manager: Port Monitoring 8-8
Isolating Port Issues Using Device Manager 8-9
Troubleshooting Port States from the CLI 8-10
Using Port Debug Commands 8-10
Useful Commands at the FC-MAC Level 8-11
Isolating Port Issues Using the CLI 8-12
Common Problems with Port Interfaces 8-13
Port Remains in a Link Failure or Not Connected State 8-13
Troubleshooting Port Problems 8-15
Port Remains in Initializing State 8-16
Troubleshooting Port Registration Issues Using the CLI 8-17
Unexpected Link Flapping Occurs 8-21

Send documentation comments to mdsfeedbac[email protected]
Contents
ix
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Link Initialization Flow 8-23
Viewing Port Counters 8-25
Port Bounces Between Initializing and Offline States 8-26
Troubleshooting ELP Issues Using the CLI 8-27
E Port Bounces Remains Isolated After a Zone Merge 8-28
Troubleshooting E port Isolation using Fabric Manager 8-29
Troubleshooting E port Isolation Using the CLI 8-30
Port Cycles Through Up and Down States 8-31
Port Is in ErrDisabled State 8-31
Verifying the ErrDisable State Using the CLI 8-31
Troubleshooting Fx Port Failure 8-32
Overview of Symptoms 8-32
CHAPTER 9Troubleshooting N-Port Virtualization 9-1
Overview 9-1
Initial Troubleshooting Checklist 9-2
Limitations and Restrictions 9-2
Common CLI Commands for NPV 9-2
Common Problems with NPV 9-4
Moving the Login of an End Device 9-4
NPIV Is Not Enabled 9-5
VSAN Mismatches 9-5
Core NPV Device Is Not a Switch 9-6
NPV Core Switch Port Is Down 9-6
Server Interface is Down 9-6
Waiting on FLOGI from the Server or Target 9-7
Waiting on External Link to Come Up 9-7
CHAPTER 10 Troubleshooting PortChannels and Trunking 10-1
PortChannel Overview 10-1
Trunking Overview 10-2
Initial Troubleshooting Checklist 10-2
Common Troubleshooting Tools in Fabric Manager 10-2
Common Troubleshooting Commands in the CLI 10-3
PortChannel Issues 10-3
Cannot Configure a PortChannel 10-3
Newly Added Interface Does Not Come Online In a PortChannel 10-4
Configuring Port Channel Modes Using Fabric Manager 10-4
Trunking Issues 10-4

Send documentation comments to mdsfeedbac[email protected]
Contents
x
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Cannot Configure Trunking 10-5
VSAN Traffic Does Not Traverse Trunk 10-5
CHAPTER 11 Troubleshooting VSANs, Domains, and FSPF 11-1
Overview 11-1
Initial Troubleshooting Checklist 11-1
Common Troubleshooting Tools in Fabric Manager 11-2
Common Troubleshooting Commands in the CLI 11-2
VSAN Issues 11-3
Host Cannot Communicate with Storage 11-3
Verifying VSAN Membership Using Fabric Manager 11-4
Verifying VSAN Membership Using the CLI 11-4
E Port Is Isolated in a VSAN 11-5
Resolving an Isolated E Port Using Fabric Manager 11-6
Resolving an Isolated E Port Using the CLI 11-6
Resolving an Isolated ISL Using Fabric Manager 11-7
Resolving an Isolated ISL Using the CLI 11-7
Resolving Fabric Timer Issues Using Fabric Manager 11-9
Resolving Fabric Timer Issues Using the CLI 11-9
Troubleshooting Interop Mode Issues 11-9
Dynamic Port VSAN Membership Issues 11-9
Troubleshooting DPVM Using Fabric Manager 11-10
Troubleshooting DPVM Using the CLI 11-11
DPVM Configuration Not Available 11-11
DPVM Database Not Distributed 11-12
DPVM Autolearn Not Working 11-12
No Autolearn Entries in Active Database 11-13
VSAN Membership not Added to Database 11-13
DPVM Config Database Not Activating 11-14
Cannot Copy Active to Config DPVM Database 11-14
Port Suspended or Disabled After DPVM Activation 11-15
DPVM Merge Failed 11-15
DPVM Process Terminates 11-16
Disabling DPVM 11-16
Domain Issues 11-17
Domain ID Conflict Troubleshooting 11-17
Switch Cannot See Other Switches in a VSAN 11-18
FC Domain ID Overlap 11-18
Assigning a New Domain ID Using Fabric Manager 11-19

Send documentation comments to mdsfeedbac[email protected]
Contents
xi
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Assigning a New Domain ID Using the CLI 11-19
Using Fabric Reconfiguration for Domain ID Assignments 11-20
CFS Distribution of Domain ID List Fails 11-22
Allowed Domain ID List Incorrect After a VSAN Merge 11-23
Changes to fcdomain Do Not Take Effect 11-23
FSPF Issues 11-24
Troubleshooting FSPF 11-24
Troubleshooting FSPF Using Device Manager 11-25
Troubleshooting FSPF Using the CLI 11-25
Loss of Two-Way Communication 11-28
Resolving a Wrong Hello Interval on an ISL Using Device Manager 11-28
Resolving a Wrong Hello Interval on an ISL Using the CLI 11-29
Resolving a Mismatched Retransmit Interval on an ISL Using Device Manager 11-30
Resolving a Mismatched Retransmit Interval on an ISL Using the CLI 11-30
Resolving a Mismatch in Dead Intervals on an ISL Using Fabric Manager 11-31
Resolving a Mismatch in Dead Intervals on an ISL Using the CLI 11-31
Resolving a Region Mismatch Using Fabric Manager 11-32
Resolving a Region Mismatch Using the CLI 11-32
CHAPTER 12 Troubleshooting SAN Device Virtualization 12-1
Overview 12-1
Initial Troubleshooting Checklist 12-1
Debugging and Verifying SDV Configuration Using the CLI 12-2
SDV Limitations and Restrictions 12-2
SDV Issues 12-3
SDV Commit Fails 12-3
SDV Commit Partially Fails 12-4
Host Cannot Locate Disk 12-5
SDV Merge Fails When ISL Comes Up 12-6
Zone Activation Fails in a SDV Zone 12-6
CHAPTER 13 Troubleshooting IVR 13-1
Overview 13-1
Configuration Guidelines 13-1
Transit VSANs 13-2
Border Switches 13-2
Limitations and Restrictions 13-2
Initial Troubleshooting Checklist 13-3
Verifying IVR Configuration Using Fabric Manager 13-3

Send documentation comments to mdsfeedbac[email protected]
Contents
xii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Verifying IVR Configuration Using the CLI 13-4
IVR Enhancements by Cisco SAN-OS Release 13-5
IVR Issues 13-6
IVR Licensing Issues 13-6
Cannot Enable IVR 13-7
IVR Network Address Translation Fails 13-8
IVR Zone Set Activation Fails 13-8
Border Switch Fails 13-10
Traffic Does Not Traverse IVR Path 13-11
Link Isolated 13-12
Persistent FC ID for IVR Failed 13-12
LUN Configuration Failure in IVR Zoning 13-13
Host Does Not Have Write Access to Storage 13-13
Locked IVR CFS Session 13-13
CFS Merge Failed 13-14
Troubleshooting the IVR Wizard 13-15
Warning: Not All Switches Are IVR NAT Capable or Are Unmanageable 13-16
Error: The Following Switches Do Not Have Unique Domain IDs 13-16
Error: Pending Action/ Pending Commits 13-17
Error: Fabric Is Changing. Please Retry the Request Later 13-17
CHAPTER 14 Troubleshooting Zones and Zone Sets 14-1
Overview 14-1
Troubleshooting Checklist 14-1
Troubleshooting Zone Configuration Issues with Fabric Manager 14-2
Troubleshooting Zone Configuration Issues with the CLI 14-2
Zone and Zone Set Issues 14-4
Host Cannot Communicate with Storage 14-5
Resolving Host Not Communicating with Storage Issue Using Fabric Manager 14-5
Resolving Host Not Communicating with Storage Using the CLI 14-6
Troubleshooting Zone Set Activation 14-8
Troubleshooting Zone Activation Using Fabric Manager 14-9
Troubleshooting Zone Activation Using the CLI 14-10
Troubleshooting Full Zone Database Synchronization Across Switches 14-11
Resolving Out of Sync Full Zone Database Using Fabric Manager 14-11
Resolving an Out of Sync Full Zone Database Using the CLI 14-11
Mismatched Default Zone Policy 14-12
Resolving Mismatched Default Zone Policies Using Fabric Manager 14-12
Resolving Mismatched Default Zone Policies Using the CLI 14-13

Send documentation comments to mdsfeedbac[email protected]
Contents
xiii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Zone Merge Failure 14-13
Recovering from Link Isolation 14-15
Resolving a Link Isolation Because of a Failed Zone Merge Using Fabric Manager 14-15
Resolving a Link Isolation Because of a Failed Zone Merge Using the CLI 14-16
Mismatched Active Zone Sets Within the Same VSAN 14-17
Resolving Mismatched Active Zone Sets Within the Same VSAN Using Fabric Manager 14-17
Resolving Mismatched Active Zone Sets Within the Same VSAN Using the CLI 14-18
Deactivating a Zone Set and Restarting the Zone Merge Process Using Fabric Manager 14-20
Deactivating a Zone Set and Restarting the Zone Merge Process Using the CLI 14-20
Enhanced Zoning Issues 14-21
Resolving Enhanced Zoning Lock Issues with Fabric Manager 14-23
Resolving Enhanced Zoning Lock Issues with the CLI 14-23
CHAPTER 15 Troubleshooting Distributed Device Alias Services 15-1
Overview 15-1
Initial Troubleshooting Checklist 15-1
Merge Failure Messages 15-2
Merge Validation Failure Messages 15-2
Commit Failure Messages 15-3
Verifying Device Alias Database Status Using the CLI 15-3
Limitations and Restrictions 15-3
Merge Failure Issues 15-4
Resolving Duplicate Device Alias Names 15-5
Resolving Mapping a pWWN to Different Device Alias Names 15-6
Resolving Mode Mismatch 15-6
Resolving Merge Failures in Mixed Fabric 15-7
Resolving a Validation Failure 15-8
Resolving Merge in Progress Issues 15-9
Validation and Commit Failure Issues 15-11
Resolving Database Conflicts 15-12
Resolving Application Busy Situations 15-13
Resolving Database Size Issues 15-14
Resolving Mode Issues 15-15
CHAPTER 16 Troubleshooting FICON 16-1
FICON Overview 16-1
FICON Port Numbering 16-2
Default FICON Port Numbering Scheme 16-2
Reserved FICON Port Numbering Scheme 16-4

Send documentation comments to mdsfeedbac[email protected]
Contents
xiv
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
FICON Configuration Files 16-5
CUP In-Band Management 16-6
Fabric Binding 16-6
FICON Configuration Requirements 16-6
Initial Troubleshooting Checklist 16-7
Common Troubleshooting Tools in Fabric Manager or Device Manager 16-8
Common Troubleshooting Commands in the CLI 16-8
Port Swapping 16-8
Swapping Ports 16-9
FICON Issues 16-9
Cannot Enable FICON 16-10
Switch ISL Isolated 16-10
Fabric Manager or Device Manager Cannot Configure FICON 16-10
Mainframe Cannot Configure FICON 16-11
Cannot Enable FICON Port 16-11
Cannot Configure FCIP or PortChannel for FICON 16-12
FCIP fails for FICON 16-12
FICON Tape Acceleration Not Working 16-12
CHAPTER 17 Troubleshooting RADIUS and TACACS+ 17-1
AAA Overview 17-1
Initial Troubleshooting Checklist 17-1
Common Troubleshooting Tools in Fabric Manager 17-2
Common Troubleshooting Commands in the CLI 17-2
AAA Issues 17-2
Switch Does Not Communicate with AAA Server 17-2
Verifying RADIUS Configuration Using Fabric Manager 17-4
Verifying RADIUS Configuration Using the CLI 17-4
Verifying TACACS+ Configuration Using Fabric Manager 17-5
Verifying TACACS+ Configuration Using the CLI 17-5
Verifying RADIUS Server Monitor Configuration Using Fabric Manager 17-6
Verifying RADIUS Server Monitor Configuration Using the CLI 17-6
Verifying TACACS+ Server Monitor Configuration Using Fabric Manager 17-7
Verifying TACACS+ Server Monitor Configuration Using the CLI 17-7
User Authentication Fails 17-8
Verifying RADIUS Server Groups Using Fabric Manager 17-9
Verifying RADIUS Server Groups Using the CLI 17-9
Verifying TACACS+ Server Groups Using Fabric Manager 17-9
Verifying TACACS+ Server Groups Using the CLI 17-10

Send documentation comments to mdsfeedbac[email protected]
Contents
xv
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
User Is Not in Any Configured Role 17-10
User Cannot Access Certain Features 17-11
Troubleshooting RADIUS and TACACS+ With Cisco ACS 17-11
CHAPTER 18 Troubleshooting Users and Roles 18-1
Overview 18-1
User Accounts 18-1
Role-Based Authorization 18-2
Rules and Features for Each Role 18-3
Initial Troubleshooting Checklist 18-4
Common Troubleshooting Tools in Fabric Manager 18-4
Common Troubleshooting Commands in the CLI 18-4
User and Role Issues 18-4
User Cannot Log into Switch 18-5
Verifying User Login with System Messages Using Device Manager 18-5
Verifying User Login with System Messages Using the CLI 18-6
User Cannot Create Roles 18-7
User Cannot Create Other Users With Fabric Manager or Device Manager 18-7
User Cannot Access Certain Features 18-8
Verifying Roles Using Device Manager 18-8
Verifying Roles Using the CLI 18-9
User Has Too Much Access 18-10
User Cannot Configure Some VSANs 18-10
Verifying VSAN-Restricted Roles Using Fabric Manager 18-10
Verifying VSAN-Restricted Roles Using the CLI 18-11
User Cannot Configure E Ports 18-11
Unexpected User Displayed in Logs 18-12
Troubleshooting Users and Roles with Cisco ACS 18-12
CHAPTER 19 Troubleshooting FC-SP, Port Security, and Fabric Binding 19-1
FC-SP Overview 19-1
Port Security Overview 19-2
Fabric Binding Overview 19-2
Initial Troubleshooting Checklist 19-2
Common Troubleshooting Tools in Fabric Manager 19-3
Common Troubleshooting Commands in the CLI 19-3
FC-SP Issues 19-4
Switch or Host Blocked from Fabric 19-4

Send documentation comments to mdsfeedbac[email protected]
Contents
xvi
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Verifying FC-SP Configuration Using Fabric Manager 19-5
Verifying FC-SP Configuration Using the CLI 19-5
Verifying Local FC-SP Database Using Fabric Manager 19-5
Verifying Local FC-SP Database Using the CLI 19-6
Authentication Fails When Using Cisco ACS 19-7
Port Security Issues 19-7
Device Does Not Log into a Switch When AutoLearn Is Disabled 19-8
Device Does Not Log into a Switch When Autolearn Is Enabled 19-8
Verifying the Active Port Security Database Using Fabric Manager 19-9
Verifying the Active Port Security Database Using the CLI 19-9
Verifying Port Security Violations Using Fabric Manager 19-10
Verifying Port Security Violations Using the CLI 19-11
Cannot Activate Port Security 19-12
Unauthorized Device Gains Access to Fabric 19-12
Disabling Autolearn Using Fabric Manager 19-13
Disabling Autolearn Using the CLI 19-13
Port Security Settings Lost After Reboot 19-13
Merge Fails 19-14
Configuring Port Security with Autolearn Using Fabric Manager 19-14
Configuring Port Security with Autolearn Using the CLI 19-15
Fabric Binding Issues 19-15
Switch Cannot Attach to the Fabric 19-16
Verifying Fabric Binding Violations Using Fabric Manager 19-16
Verifying Fabric Binding Violations Using the CLI 19-17
Cannot Activate Fabric Binding 19-18
Verifying the Config Fabric Binding Database Using Fabric Manager 19-18
Verifying the Config Fabric Binding Database Using the CLI 19-18
Unauthorized Switch Gains Access to Fabric 19-19
Fabric Binding Settings Lost After Reboot 19-19
Configuring Fabric Binding Using Fabric Manager 19-19
Configuring Fabric Binding Using the CLI 19-20
CHAPTER 20 Troubleshooting IP Storage Services 20-1
Overview 20-1
iSCSI Restrictions 20-2
iSLB Restrictions 20-2
Initial Troubleshooting Checklist 20-3
Common Troubleshooting Tools in Fabric Manager 20-3
Common Troubleshooting Commands in the CLI 20-3

Send documentation comments to mdsfeedbac[email protected]
Contents
xvii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
IP Issues 20-5
Verifying Basic Connectivity 20-5
Verifying Basic Connectivity Using Device Manager 20-5
Verifying Basic Connectivity Using the CLI 20-5
Verification of Switch Connectivity 20-7
Verifying Switch Connectivity Using Fabric Manager 20-7
Verifying Switch Connectivity Using the CLI 20-7
Verification of Static IP Routing 20-8
Verifying Static IP Routing Using Device Manager 20-8
Verifying Static IP Routing Using the CLI 20-8
Cannot Assign IP Address to an Interface 20-9
FCIP Issues 20-9
One-to-One FCIP Tunnel Creation and Monitoring 20-10
Configuring the First Switch with the CLI 20-10
Displaying the Default Values Using the CLI 20-11
Setting the Static Route for FCIP Tunnels Using the CLI 20-11
Debugging the Configuration of the Second Switch Using the CLI 20-11
Displaying the Debug Output from FCIP Tunnel Supervisor Using the CLI 20-13
Displaying the Debug Output from the FCIP Tunnel IPS Module Using the CLI 20-14
Verifying the Configuration of the Profiles Using the CLI 20-15
Verifying the Establishment of the FCIP Tunnel Using the CLI 20-15
Verifying the Establishment of Default TCP Connections for Each Configured FCIP Tunnel Using
the CLI 20-17
Verifying the Statistics of the ASIC Chip on Each Gigabit Ethernet Port Using the CLI 20-17
Ethereal Screen Captures of the TCP Connection and FCIP Tunnels 20-18
One-to-Three FCIP Tunnel Creation and Monitoring 20-20
Displaying the Configuration of the First Switch Using the CLI 20-20
Creating the FCIP Interface for the Second Tunnel Using the CLI 20-21
FCIP Profile Misconfiguration Examples 20-21
Displaying Incorrect or Nonexistent IP Address for an FCIP Profile Using the CLI 20-21
Displaying Configuration Errors When Bringing Up a Tunnel on a Selected Port Using the
CLI 20-22
FCIP Interface Misconfiguration Examples 20-24
Displaying FCIP Misconfiguration Examples Using the CLI 20-24
Displaying the FCIP Interface as Administratively Shut Down Using the CLI 20-25
Displaying the Debug Output from the Second Switch Using the CLI 20-26
Displaying Passive Mode Set on Both Sides of the FCIP Tunnel with the CLI 20-27
Displaying a Time Stamp Acceptable Difference Failure Using the CLI 20-28
FCIP Special Frame Tunnel Creation and Monitoring 20-30
Configuring and Displaying an FCIP Tunnel with Special Frame Using the CLI 20-31

Send documentation comments to mdsfeedbac[email protected]
Contents
xviii
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Special Frame Misconfiguration Example 20-33
Troubleshooting FCIP Link Flaps 20-34
Troubleshooting FCIP ISL Link Failures 20-34
Troubleshooting FCIP and Compression 20-35
iSCSI Issues 20-35
Troubleshooting iSCSI Authentication 20-35
Displaying iSCSI Authentication Using Fabric Manager 20-36
Displaying iSCSI Authentication Using the CLI 20-36
Troubleshooting User Name and Password Configuration 20-37
Verifying iSCSI User Account Configuration Using Fabric Manager 20-37
Verifying iSCSI User Account Configuration Using the CLI 20-37
RADIUS Configuration Troubleshooting 20-38
Verifying RADIUS Key and Port for Authentication and Accounting 20-38
Troubleshooting RADIUS Routing Configuration 20-40
Displaying the Debug Output for RADIUS Authentication Request Routing Using the CLI 20-40
Troubleshooting Dynamic iSCSI Configuration 20-41
Checking the Configuration 20-41
Performing Basic Dynamic iSCSI Troubleshooting 20-41
Useful Show Commands to Debug Dynamic iSCSI Configuration 20-42
Virtual Target Access Control 20-43
Useful Show Commands to Debug Static iSCSI Configuration 20-43
iSCSI TCP Performance Issues 20-48
CLI Commands Used to Access Performance Data 20-49
Understanding TCP Parameters for iSCSI 20-49
Lab Setup 20-50
Configuring from the Bottom Switch Using the CLI 20-50
Verifying Connectivity Between Client and IPS iSCSI Service 20-51
TCP Parameter Changes 20-54
Displaying the Gigabit Ethernet Interface 20-54
Verifying that the Host Is Configured for High MTU or MSS with the CLI 20-57
iSLB Issues 20-58
iSLB Configuration Not Distributed to All Switches in the Fabric 20-58
iSCSI Initiator and Virtual Target Configuration Not Distributed 20-59
iSLB Configuration, Commit, or Merge Failed—”VSAN ID is Not Yet Configured” 20-59
iSLB Configuration, Commit, or Merge Failed—”Failed to Allocate WWN” 20-60
iSLB Configuration, Commit, or Merge Failed—”Duplicate WWN Found as...” 20-60
iSLB Configuration, Commit, or Merge Failed—”Duplicate Node Name” 20-60
iSLB Configuration Failed—”Pending iSLB CFS Config Has Reached Its Limit...” 20-61
iSCSI Disable Failed—”Cannot Disable Iscsi - Large Iscsi Config Present...” 20-61

Send documentation comments to mdsfeedbac[email protected]
Contents
xix
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
iSLB Commit Timeout 20-61
Session Down—”pWWN in Use At Remote Switch” 20-62
Redirected Session Does Not Come Up 20-62
iSLB Zones Not Present in Active Zone Set 20-63
Traffic Description After iSLB Commit or Activation of Zone Set 20-63
VRRP Master Overutilized 20-64
iSLB Zone Set Activation Failed 20-64
iSLB CFS Commit Fails 20-65
Resolving an iSLB Merge Failure 20-65
CHAPTER 21 Troubleshooting IP Access Lists 21-1
Overview 21-1
Protocol Information 21-1
Address Information 21-2
Port Information 21-2
ICMP Information 21-3
ToS Information 21-3
Initial Troubleshooting Checklist 21-4
Common Troubleshooting Tools in Fabric Manager 21-4
Common Troubleshooting Commands in the CLI 21-4
IP-ACL Issues 21-4
All Packets Are Blocked 21-5
Re-creating IP-ACLs Using Fabric Manager 21-5
Re-creating IP-ACLs Using the CLI 21-6
No Packets Are Blocked 21-7
PortChannel Not Working with ACL 21-8
Cannot Remotely Connect to Switch 21-8
CHAPTER 22 Troubleshooting IPsec 22-1
Overview 22-1
IPsec Compatibility 22-1
Supported IPsec and IKE Algorithms for Microsoft Windows and Linux Platforms 22-2
IKE Allowed Transforms 22-3
IPsec Allowed Transforms 22-4
Initial Troubleshooting Checklist 22-4
Common Troubleshooting Tools in Fabric Manager 22-4
Common Troubleshooting Commands in the CLI 22-5
IPsec Issues 22-5
Verifying IKE Configuration Compatibility 22-6

Send documentation comments to mdsfeedbac[email protected]
Contents
xx
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x
OL-9285-05
Verifying IPsec Configuration Compatibility Using Fabric Manager 22-6
Verifying IPsec Configuration Compatibility Using the CLI 22-7
Verifying Security Policy Databases Compatibility 22-8
Verifying Interface Status Using Fabric Manager 22-9
Verifying Interface Status Using the CLI 22-9
Verifying Security Associations 22-12
Security Associations Do Not Re-Key 22-15
Clearing Security Associations 22-15
Debugging the IPsec Process 22-15
Debugging the IKE Process 22-15
Obtaining Statistics from the IPsec Process 22-15
CHAPTER 23 Troubleshooting SANTap 23-1
Overview 23-1
Definitions 23-2
Limitations 23-2
Interface Restrictions 23-3
Initial Troubleshooting Checklist 23-3
Common Troubleshooting Tools in Fabric Manager 23-4
Common Troubleshooting Commands in the CLI 23-4
Messages, Logs and Databases 23-6
SANTap Issues 23-6
Host Login Problems 23-7
ITL Problems 23-7
Common Mismatch Problems 23-7
CHAPTER 24 Troubleshooting Digital Certificates 24-1
Overview 24-1
Digital Certificates 24-1
Certificate Authorities 24-1
RSA Key Pairs and Identity Certificates 24-2
Peer Certificate Verification 24-2
CRLs and OCSP Support 24-2
Import and Export Support for Certificates and Associated Key Pairs 24-2
PKI Enrollment Support 24-2
Maximum Limits 24-3
Initial Troubleshooting Checklist 24-3
Common Troubleshooting Tools in Fabric Manager 24-3
Common Troubleshooting Commands in the CLI 24-4