Cisco MDS 9000 Series User manual

Send comments to [e-mail].
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco MDS 9000 Family Troubleshooting
Guide
Cisco MDS SAN-OS Release 1.3
April 2004
Customer Order Number:
Text Part Number: OL-5183-02

Send comments to [e-mail].
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISINGOUT OF THEUSE OR INABILITYTO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco MDS 9000 Family Troubleshooting Guide
Copyright © 2003, Cisco Systems, Inc.
All rights reserved.
Copyright © 2003 Cisco Systems, Inc. All rights reserved. CCIP, CCSP, the Cisco Arrow logo, the Cisco Powered Network mark, the Cisco Systems Verified logo, Cisco Unity, Follow Me Browsing, FormShare, iQ
Net Readiness Scorecard, Networking Academy, and ScriptShare are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, The Fastest Way to Increase Your Internet Quotient, and
iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo,
Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, Fast Step, GigaStack, Internet Quotient, IOS, IP/TV, iQ
Expertise, the iQ logo, LightStream, MGX, MICA, the Networkers logo, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, SlideCast, SMARTnet, StrataView Plus, Stratm,
SwitchProbe, TeleRouter, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.
All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.
(0303R)

Send comments to [e-mail].
iii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
CONTENTS
Preface ix
Document Organization ix
Document Conventions x
Related Documentation x
Obtaining Documentation xii
Cisco.com xii
Documentation CD-ROM xii
Ordering Documentation xiii
Documentation Feedback xiii
Obtaining Technical Assistance xiii
Cisco.com xiv
Technical Support Center xiv
Cisco Technical Support Website xiv
Cisco Technical Support Escalation Center xv
Obtaining Additional Publications and Information xv
CHAPTER 1Troubleshooting Overview 1-1
Introduction 1-1
Basics 1-2
Basic Connectivity 1-2
Fibre Channel End-to-End Connectivity 1-2
Fabric Issues 1-3
Contacting Customer Support 1-3
Using Host Diagnostic Tools 1-3
Using Cisco MDS 9000 Family Tools 1-4
Command-Line-Interface (CLI) 1-5
CLI Debug 1-5
FC Ping and FC Traceroute 1-7
Cisco Fabric Manager 1-9
Fabric Manager and Device Manager 1-9
Analyzing Switch Device Health 1-10
Analyzing End-to-End Connectivity 1-11
Analyzing Switch Fabric Configuration 1-11
Analyzing the Results of Merging Zones 1-12

Send comments to [e-mail].
Contents
iv
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Alerts and Alarms 1-13
SCSI Target Discovery 1-13
SNMP and RMON Support 1-14
Using RADIUS 1-15
Using Syslog 1-16
Using Fibre Channel SPAN 1-16
Using Cisco Network Management Products 1-18
Cisco MDS 9000 Port Analyzer Adapter 1-18
Cisco Fabric Analyzer 1-19
CiscoWorks RME 1-21
Using Other Troubleshooting Products 1-22
Fibre Channel Testers 1-22
Fibre Channel Protocol Analyzers 1-22
CHAPTER 2Troubleshooting Switch System Issues 2-1
Recovering the Administrator Password 2-1
Troubleshooting System Restarts 2-1
Overview 2-1
Working with a Firmware Upgrade Failure 2-2
Working with Recoverable Restarts 2-2
Working with Unrecoverable System Restarts 2-6
Troubleshooting a Failed Supervisor 2-7
CHAPTER 3Troubleshooting Switch Level Issues and Interswitch Connectivity 3-1
Troubleshooting E Port Connectivity - ISL Isolation 3-1
Overview 3-2
Troubleshooting steps 3-3
Troubleshoot Switch and Port Parameters 3-3
Troubleshooting a ZONE Merge Failures 3-5
Troubleshooting a VSAN Configuration Conflict 3-6
Troubleshooting a Domain ID Conflict 3-7
Troubleshooting TE Port Connectivity - VSAN Isolation 3-10
Troubleshooting Fx Port Connectivity 3-13
Fx Port Fails to Achieve Up State 3-13
FCOT Is Not Present 3-13
Link_Failure or Not Connected 3-14
Interface Bouncing between Offline and Initializing 3-15
Point-to-point link comes up as FL_Port 3-17
Interface UP and Connectivity Problems - Troubleshooting VSANs and Zones 3-18

Send comments to [e-mail].
Contents
v
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Troubleshooting Zones – Case of end devices belonging to the default zone 3-19
Troubleshooting Zones – Case of end devices belonging to a specific zone 3-19
Verify active zoneset configuration 3-19
Verify active zoneset membership 3-20
Other useful commands 3-21
Using the GUI to Troubleshoot Zoning Configuration Issues 3-22
Troubleshooting VSANs 3-23
Using the GUI to Troubleshoot VSAN Membership Problems 3-24
Troubleshooting Hardware Problems 3-25
Using Port Debug Commands 3-25
SAN-OS version 1.3(4) and above: 3-26
Software versions below 1.3(4) and above 1.2(1A): 3-26
Software vesrion 1.2(1A) and below: 3-27
CHAPTER 4Troubleshooting Switch Fabric Level Issues 4-1
Troubleshooting Name Server Issues 4-1
Overview 4-1
Nx Port Registration Problems 4-2
Troubleshooting FSPF Issues 4-6
Overview 4-6
Loss of Two-way Communication 4-7
Wrong Hello Interval on an ISL Interface 4-7
Resolving the Wrong Hello Interval Problem 4-8
Wrong Dead Interval on an ISL Interface 4-8
Resolving a Wrong Dead Interval Problem 4-9
Region Mismatch on Switch 4-9
Resolving a Region Mismatch Problem 4-10
FSPF Issues in a Single-VSAN Environment 4-11
FSPF Issues in a Multi-VSAN Environment 4-13
Troubleshooting Zoning Issues 4-14
Mismatched Active Zonesets Within the Same VSAN 4-14
Importing or Exporting a Zoneset Between Switches 4-16
Deactivating a Zoneset and Restarting the Zone Merge Process 4-17
Misconfigured Zones Within an Active Zoneset in the Same VSAN 4-19
CHAPTER 5Troubleshooting IP Storage Issues 5-1
Overview 5-1
Troubleshooting IP Connections 5-2
Verifying Basic Connectivity 5-2

Send comments to [e-mail].
Contents
vi
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Verifying Static IP Routing 5-4
Troubleshooting FCIP Connections 5-5
One-to-One FCIP Tunnel Creation and Monitoring 5-5
One to three FCIP tunnel creation and monitoring 5-15
FCIP Profile Misconfiguration Examples 5-16
Interface FCIP Misconfiguration Examples 5-19
FCIP Special Frame Tunnel Creation and Monitoring 5-26
Special Frame Misconfiguration Examples 5-28
Troubleshooting iSCSI Issues 5-31
Troubleshooting iSCSI Authentication 5-31
Configuring Authentication 5-32
Troubleshooting Username/Password Configuration 5-33
Troubleshooting Radius Configuration 5-33
Troubleshooting Radius Routing Configuration 5-36
Troubleshooting Dynamic iSCSI Configuration 5-36
Checking the Configuration 5-36
Performing Basic Dynamic iSCSI Troubleshooting 5-37
Useful show Commands for Debugging Dynamic iSCSI Configuration 5-37
Virtual Target Access Control 5-39
Useful show Commands for Debugging Static iSCSI Configuration 5-39
Fine Tuning/Troubleshooting IPS iSCSI TCP Performance 5-44
Lab Setup 5-45
Configuration from the Bottom MDS 5-45
Changing TCP Parameters 5-49
CHAPTER 6Troubleshooting the Fabric 55
Analyzing Switch Device Health 55
Analyzing End-to-End Connectivity 56
Analyzing Switch Fabric Configuration 56
Analyzing the Results of Merging Zones 57
Issuing the Show Tech Support Command 58
Using Traceroute and Other Troubleshooting Tools 59
Locating Other Switches 59
Configuring an OUI 60
CHAPTER 7Troubleshooting Fabric Manager Issues 61
Can I Set the Map Layout So It Stays After I Restart Fabric Manager? 61
Two Switches Show on my Map, But I Only Have One Switch 62

Send comments to [e-mail].
Contents
vii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
There is a Red Line Through the Switch. What’s Wrong? 62
There is a Dotted Orange Line Through the Switch. What’s Wrong? 62
Can I Upgrade Without Losing My Map Settings? 62
Are There Any Restrictions When Using Fabric Manager Across FCIP? 62
Running Cisco Fabric Manager with Multiple Interfaces 63
Specifying an Interface for Fabric Manager Server 63
Specifying an Interface for Performance Manager 63
Specifying an Interface for Fabric Manager Client or Device Manager 64
Configuring a Proxy Server 64
Clearing Topology Maps 64
Can I Use Fabric Manager in a Mixed Software Environment? 65
I NDEX

Send comments to [e-mail].
Contents
viii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3

Send comments to [e-mail].
ix
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
This document is intended to provide guidance for troubleshooting issues that may appear when
deploying a storage area network (SAN) using the Cisco MDS 9000 Family of switches. This document
will help you investigate the configuration of the different systems included in the SAN environment,
such as hubs, hosts, and storage arrays and Cisco MDS 9000 switches. It also covers basic storage
commands, switch configurations, and common storage parameters. It introduces tools and
methodologies to recognize a problem, determine its cause, and find possible solutions.
Document Organization
This document is organized into the following chapters:
Chapter Title Description
Chapter 1 Troubleshooting Overview Describes basic concepts, methodology, and tools
to use for troubleshooting.
Chapter 2 Troubleshooting Switch
Hardware and Booting Problems Describes how to identify and resolve problems for a single Cisco MDS 9000 Family switch.
Chapter 3 Troubleshooting Switch Level
Issues and Interswitch
Connectivity
Describes how to identify and resolve problems
that affect basic connectivity between switches,
hosts, and storage in the network fabric.
Chapter 4 Troubleshooting Switch Fabric
Level Issues Describes switch fabric-level troubleshooting procedures.
Chapter 5 Troubleshooting IP Storage
Issues Describes IP storage troubleshooting procedures for the FCIP and iSCSI features.
Chapter 6 Troubleshooting the Fabric Provides information on using Fabric Manager to
troubleshoot your fabric.
Chapter 7 TroubleshootingFabricManager
Issues Describes some common issues you may experiencewhile usingCiscoFabric Manager, and
provides solutions.

Send comments to [e-mail].
x
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Document Conventions
Document Conventions
Command descriptions use these conventions:
Screen examples use these conventions:
This document uses the following conventions:
Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the
manual.
Caution Means reader be careful. In this situation, you might do something that could result in equipment
damage or loss of data.
Related Documentation
For Fabric Manager and Device Manager field descriptions, refer to the Cisco MDS 9000 Family Fabric
Manager Online Help. For additional information, refer to the following documents:
•Regulatory Compliance and Safety Information for the Cisco MDS 9000 Family
•Cisco MDS 9100 Series Hardware Installation Guide
•Cisco MDS 9200 Series Hardware Installation Guide
•Cisco MDS 9500 Series Hardware Installation Guide
•Cisco MDS 9000 Family Configuration Guide
•Cisco MDS 9000 Family Command Reference
•Cisco MDS 9000 Family System Messages Guide
boldface font Commands and keywords are in boldface.
italic font Arguments for which you supply values are in italics.
[ ] Elements in square brackets are optional.
[ x | y | z ] Optional alternative keywords are grouped in brackets and separated by vertical
bars.
string A nonquoted set of characters. Do not use quotation marks around the string or
the string will include the quotation marks.
screen font Terminal sessions and information the switch displays are in screen font.
boldface screen font Information you must enter is in boldface screen font.
italic screen font Arguments for which you supply values are in italic screen font.
< > Nonprinting characters, such as passwords are in angle brackets.
[ ] Default responses to system prompts are in square brackets.
!, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code
indicates a comment line.