Cisco MDS 9000 Series User manual

Send comments to [e-mail].
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco MDS 9000 Family Troubleshooting
Guide
Cisco MDS SAN-OS Release 1.3
April 2004
Customer Order Number:
Text Part Number: OL-5183-02

Send comments to [e-mail].
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISINGOUT OF THEUSE OR INABILITYTO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco MDS 9000 Family Troubleshooting Guide
Copyright © 2003, Cisco Systems, Inc.
All rights reserved.
Copyright © 2003 Cisco Systems, Inc. All rights reserved. CCIP, CCSP, the Cisco Arrow logo, the Cisco Powered Network mark, the Cisco Systems Verified logo, Cisco Unity, Follow Me Browsing, FormShare, iQ
Net Readiness Scorecard, Networking Academy, and ScriptShare are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, The Fastest Way to Increase Your Internet Quotient, and
iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo,
Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, Fast Step, GigaStack, Internet Quotient, IOS, IP/TV, iQ
Expertise, the iQ logo, LightStream, MGX, MICA, the Networkers logo, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, SlideCast, SMARTnet, StrataView Plus, Stratm,
SwitchProbe, TeleRouter, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.
All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.
(0303R)

Send comments to [e-mail].
iii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
CONTENTS
Preface ix
Document Organization ix
Document Conventions x
Related Documentation x
Obtaining Documentation xii
Cisco.com xii
Documentation CD-ROM xii
Ordering Documentation xiii
Documentation Feedback xiii
Obtaining Technical Assistance xiii
Cisco.com xiv
Technical Support Center xiv
Cisco Technical Support Website xiv
Cisco Technical Support Escalation Center xv
Obtaining Additional Publications and Information xv
CHAPTER 1Troubleshooting Overview 1-1
Introduction 1-1
Basics 1-2
Basic Connectivity 1-2
Fibre Channel End-to-End Connectivity 1-2
Fabric Issues 1-3
Contacting Customer Support 1-3
Using Host Diagnostic Tools 1-3
Using Cisco MDS 9000 Family Tools 1-4
Command-Line-Interface (CLI) 1-5
CLI Debug 1-5
FC Ping and FC Traceroute 1-7
Cisco Fabric Manager 1-9
Fabric Manager and Device Manager 1-9
Analyzing Switch Device Health 1-10
Analyzing End-to-End Connectivity 1-11
Analyzing Switch Fabric Configuration 1-11
Analyzing the Results of Merging Zones 1-12

Send comments to [e-mail].
Contents
iv
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Alerts and Alarms 1-13
SCSI Target Discovery 1-13
SNMP and RMON Support 1-14
Using RADIUS 1-15
Using Syslog 1-16
Using Fibre Channel SPAN 1-16
Using Cisco Network Management Products 1-18
Cisco MDS 9000 Port Analyzer Adapter 1-18
Cisco Fabric Analyzer 1-19
CiscoWorks RME 1-21
Using Other Troubleshooting Products 1-22
Fibre Channel Testers 1-22
Fibre Channel Protocol Analyzers 1-22
CHAPTER 2Troubleshooting Switch System Issues 2-1
Recovering the Administrator Password 2-1
Troubleshooting System Restarts 2-1
Overview 2-1
Working with a Firmware Upgrade Failure 2-2
Working with Recoverable Restarts 2-2
Working with Unrecoverable System Restarts 2-6
Troubleshooting a Failed Supervisor 2-7
CHAPTER 3Troubleshooting Switch Level Issues and Interswitch Connectivity 3-1
Troubleshooting E Port Connectivity - ISL Isolation 3-1
Overview 3-2
Troubleshooting steps 3-3
Troubleshoot Switch and Port Parameters 3-3
Troubleshooting a ZONE Merge Failures 3-5
Troubleshooting a VSAN Configuration Conflict 3-6
Troubleshooting a Domain ID Conflict 3-7
Troubleshooting TE Port Connectivity - VSAN Isolation 3-10
Troubleshooting Fx Port Connectivity 3-13
Fx Port Fails to Achieve Up State 3-13
FCOT Is Not Present 3-13
Link_Failure or Not Connected 3-14
Interface Bouncing between Offline and Initializing 3-15
Point-to-point link comes up as FL_Port 3-17
Interface UP and Connectivity Problems - Troubleshooting VSANs and Zones 3-18

Send comments to [e-mail].
Contents
v
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Troubleshooting Zones – Case of end devices belonging to the default zone 3-19
Troubleshooting Zones – Case of end devices belonging to a specific zone 3-19
Verify active zoneset configuration 3-19
Verify active zoneset membership 3-20
Other useful commands 3-21
Using the GUI to Troubleshoot Zoning Configuration Issues 3-22
Troubleshooting VSANs 3-23
Using the GUI to Troubleshoot VSAN Membership Problems 3-24
Troubleshooting Hardware Problems 3-25
Using Port Debug Commands 3-25
SAN-OS version 1.3(4) and above: 3-26
Software versions below 1.3(4) and above 1.2(1A): 3-26
Software vesrion 1.2(1A) and below: 3-27
CHAPTER 4Troubleshooting Switch Fabric Level Issues 4-1
Troubleshooting Name Server Issues 4-1
Overview 4-1
Nx Port Registration Problems 4-2
Troubleshooting FSPF Issues 4-6
Overview 4-6
Loss of Two-way Communication 4-7
Wrong Hello Interval on an ISL Interface 4-7
Resolving the Wrong Hello Interval Problem 4-8
Wrong Dead Interval on an ISL Interface 4-8
Resolving a Wrong Dead Interval Problem 4-9
Region Mismatch on Switch 4-9
Resolving a Region Mismatch Problem 4-10
FSPF Issues in a Single-VSAN Environment 4-11
FSPF Issues in a Multi-VSAN Environment 4-13
Troubleshooting Zoning Issues 4-14
Mismatched Active Zonesets Within the Same VSAN 4-14
Importing or Exporting a Zoneset Between Switches 4-16
Deactivating a Zoneset and Restarting the Zone Merge Process 4-17
Misconfigured Zones Within an Active Zoneset in the Same VSAN 4-19
CHAPTER 5Troubleshooting IP Storage Issues 5-1
Overview 5-1
Troubleshooting IP Connections 5-2
Verifying Basic Connectivity 5-2

Send comments to [e-mail].
Contents
vi
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Verifying Static IP Routing 5-4
Troubleshooting FCIP Connections 5-5
One-to-One FCIP Tunnel Creation and Monitoring 5-5
One to three FCIP tunnel creation and monitoring 5-15
FCIP Profile Misconfiguration Examples 5-16
Interface FCIP Misconfiguration Examples 5-19
FCIP Special Frame Tunnel Creation and Monitoring 5-26
Special Frame Misconfiguration Examples 5-28
Troubleshooting iSCSI Issues 5-31
Troubleshooting iSCSI Authentication 5-31
Configuring Authentication 5-32
Troubleshooting Username/Password Configuration 5-33
Troubleshooting Radius Configuration 5-33
Troubleshooting Radius Routing Configuration 5-36
Troubleshooting Dynamic iSCSI Configuration 5-36
Checking the Configuration 5-36
Performing Basic Dynamic iSCSI Troubleshooting 5-37
Useful show Commands for Debugging Dynamic iSCSI Configuration 5-37
Virtual Target Access Control 5-39
Useful show Commands for Debugging Static iSCSI Configuration 5-39
Fine Tuning/Troubleshooting IPS iSCSI TCP Performance 5-44
Lab Setup 5-45
Configuration from the Bottom MDS 5-45
Changing TCP Parameters 5-49
CHAPTER 6Troubleshooting the Fabric 55
Analyzing Switch Device Health 55
Analyzing End-to-End Connectivity 56
Analyzing Switch Fabric Configuration 56
Analyzing the Results of Merging Zones 57
Issuing the Show Tech Support Command 58
Using Traceroute and Other Troubleshooting Tools 59
Locating Other Switches 59
Configuring an OUI 60
CHAPTER 7Troubleshooting Fabric Manager Issues 61
Can I Set the Map Layout So It Stays After I Restart Fabric Manager? 61
Two Switches Show on my Map, But I Only Have One Switch 62

Send comments to [e-mail].
Contents
vii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
There is a Red Line Through the Switch. What’s Wrong? 62
There is a Dotted Orange Line Through the Switch. What’s Wrong? 62
Can I Upgrade Without Losing My Map Settings? 62
Are There Any Restrictions When Using Fabric Manager Across FCIP? 62
Running Cisco Fabric Manager with Multiple Interfaces 63
Specifying an Interface for Fabric Manager Server 63
Specifying an Interface for Performance Manager 63
Specifying an Interface for Fabric Manager Client or Device Manager 64
Configuring a Proxy Server 64
Clearing Topology Maps 64
Can I Use Fabric Manager in a Mixed Software Environment? 65
I NDEX

Send comments to [e-mail].
Contents
viii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3

Send comments to [e-mail].
ix
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
This document is intended to provide guidance for troubleshooting issues that may appear when
deploying a storage area network (SAN) using the Cisco MDS 9000 Family of switches. This document
will help you investigate the configuration of the different systems included in the SAN environment,
such as hubs, hosts, and storage arrays and Cisco MDS 9000 switches. It also covers basic storage
commands, switch configurations, and common storage parameters. It introduces tools and
methodologies to recognize a problem, determine its cause, and find possible solutions.
Document Organization
This document is organized into the following chapters:
Chapter Title Description
Chapter 1 Troubleshooting Overview Describes basic concepts, methodology, and tools
to use for troubleshooting.
Chapter 2 Troubleshooting Switch
Hardware and Booting Problems Describes how to identify and resolve problems for a single Cisco MDS 9000 Family switch.
Chapter 3 Troubleshooting Switch Level
Issues and Interswitch
Connectivity
Describes how to identify and resolve problems
that affect basic connectivity between switches,
hosts, and storage in the network fabric.
Chapter 4 Troubleshooting Switch Fabric
Level Issues Describes switch fabric-level troubleshooting procedures.
Chapter 5 Troubleshooting IP Storage
Issues Describes IP storage troubleshooting procedures for the FCIP and iSCSI features.
Chapter 6 Troubleshooting the Fabric Provides information on using Fabric Manager to
troubleshoot your fabric.
Chapter 7 TroubleshootingFabricManager
Issues Describes some common issues you may experiencewhile usingCiscoFabric Manager, and
provides solutions.

Send comments to [e-mail].
x
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Document Conventions
Document Conventions
Command descriptions use these conventions:
Screen examples use these conventions:
This document uses the following conventions:
Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the
manual.
Caution Means reader be careful. In this situation, you might do something that could result in equipment
damage or loss of data.
Related Documentation
For Fabric Manager and Device Manager field descriptions, refer to the Cisco MDS 9000 Family Fabric
Manager Online Help. For additional information, refer to the following documents:
•Regulatory Compliance and Safety Information for the Cisco MDS 9000 Family
•Cisco MDS 9100 Series Hardware Installation Guide
•Cisco MDS 9200 Series Hardware Installation Guide
•Cisco MDS 9500 Series Hardware Installation Guide
•Cisco MDS 9000 Family Configuration Guide
•Cisco MDS 9000 Family Command Reference
•Cisco MDS 9000 Family System Messages Guide
boldface font Commands and keywords are in boldface.
italic font Arguments for which you supply values are in italics.
[ ] Elements in square brackets are optional.
[ x | y | z ] Optional alternative keywords are grouped in brackets and separated by vertical
bars.
string A nonquoted set of characters. Do not use quotation marks around the string or
the string will include the quotation marks.
screen font Terminal sessions and information the switch displays are in screen font.
boldface screen font Information you must enter is in boldface screen font.
italic screen font Arguments for which you supply values are in italic screen font.
< > Nonprinting characters, such as passwords are in angle brackets.
[ ] Default responses to system prompts are in square brackets.
!, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code
indicates a comment line.

Send comments to [e-mail].
xi
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Related Documentation
•Cisco MDS 9000 Family MIB Reference Guide
•Cisco MDS 9000 Family CIM Programming Reference Guide
•Cisco MDS 9000 Family SAN Volume Controller Configuration Guide

Send comments to [e-mail].
xii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Obtaining Documentation
For information on VERITAS Storage Foundation™ for Networks 1.0, Cisco software, refer to the
following Veritas documents available at http://support.veritas.com/:
•VERITAS Storage Foundation for Networks Overview
•VERITAS Storage Foundation for Networks Installation and Configuration Guide
•VERITAS Storage Foundation for Networks Obtaining and Installing Licenses
•VERITAS Storage Foundation for Networks GUI Administrator's Guide
•VERITAS Storage Foundation for Networks CLI Administrator's Guide
•VERITAS Storage Foundation for Networks README
For information on IBM TotalStorage SAN Volume Controller Storage Software for Cisco MDS 9000,
refer to the following IBM documents available at http://www.ibm.com/storage/support/2062-2300
•IBM TotalStorage SAN Volume Controller Storage Software for Cisco MDS 9000 Getting Started
•IBM TotalStorage SAN Volume Controller Storage Software for Cisco MDS 9000 Configuration
Guide
•IBM TotalStorage SAN Volume Controller Storage Software for Cisco MDS 9000 Command-Line
Interface User's Guide
•IBM TotalStorage Enterprise Storage Server
•IBM TotalStorage SAN Volume Controller
•IBM TotalStorage SAN Volume Controller for Cisco MDS 9000
•Subsystem Device Driver User's Guide
Obtaining Documentation
Cisco provides several ways to obtain documentation, technical assistance, and other technical
resources. These sections explain how to obtain technical information from Cisco Systems.
Cisco.com
You can access the most current Cisco documentation on the World Wide Web at this URL:
http://www.cisco.com/univercd/home/home.htm
You can access the Cisco website at this URL:
http://www.cisco.com
International Cisco websites can be accessed from this URL:
http://www.cisco.com/public/countries_languages.shtml
Documentation CD-ROM
Cisco documentation and additional literature are available in a Cisco Documentation CD-ROM
package, which may have shipped with your product. The Documentation CD-ROM is updated regularly
and may be more current than printed documentation. The CD-ROM package is available as a single unit
or through an annual or quarterly subscription.

Send comments to [e-mail].
xiii
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Obtaining Technical Assistance
Registered Cisco.com users can order a single Documentation CD-ROM (product number
DOC-CONDOCCD=) through the Cisco Ordering tool:
http://www.cisco.com/en/US/partner/ordering/ordering_place_order_ordering_tool_launch.html
All users can order monthly or quarterly subscriptions through the online Subscription Store:
http://www.cisco.com/go/subscription
Ordering Documentation
You can find instructions for ordering documentation at this URL:
http://www.cisco.com/univercd/cc/td/doc/es_inpck/pdi.htm
You can order Cisco documentation in these ways:
•Registered Cisco.com users (Cisco direct customers) can order Cisco product documentation from
the Networking Products MarketPlace:
http://www.cisco.com/en/US/partner/ordering/index.shtml
•Nonregistered Cisco.com users can order documentation through a local account representative by
calling Cisco Systems Corporate Headquarters (California, U.S.A.) at 408 526-7208 or, elsewhere
in North America, by calling 800 553-NETS (6387).
Documentation Feedback
You can submit comments electronically on Cisco.com. On the Cisco Documentation home page, click
Feedback at the top of the page.
You can e-mail your comments to [e-mail].
You can submit comments by using the response card (if present) behind the front cover of your
document or by writing to the following address:
Cisco Systems
Attn: Customer Document Ordering
170 West Tasman Drive
San Jose, CA 95134-9883
We appreciate your comments.
Obtaining Technical Assistance
Note If you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you purchased
support directly from Cisco, contact Cisco Technical Support at this URL:
http://www.cisco.com/warp/public/687/Directory/DirTAC.shtm
Cisco provides Cisco.com, which includes the Cisco Technical Support website, as a starting point for
all technical assistance. Customers and partners can obtain online documentation, troubleshooting tips,
and sample configurations from the Cisco Technical Support Website. Cisco.com registered users have
complete access to the technical support resources on the Cisco Technical Support Website, including
technical support tools and utilities.

Send comments to [e-mail].
xiv
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Obtaining Technical Assistance
Cisco.com
Cisco.com offers a suite of interactive, networked services that let you access Cisco information,
networking solutions, services, programs, and resources at any time, from anywhere in the world.
Cisco.com provides a broad range of features and services to help you with these tasks:
•Streamline business processes and improve productivity
•Resolve technical issues with online support
•Download and test software packages
•Order Cisco learning materials and merchandise
•Register for online skill assessment, training, and certification programs
To obtain customized information and service, you can self-register on Cisco.com at this URL:
http://tools.cisco.com/RPF/register/register.do
Technical Support Center
Cisco technical support is available to all customers who need technical assistance with a Cisco product,
technology, or solution. Two types of support are available: the Cisco Technical Support Website and the
Cisco Escalation Center. The type of support that you choose depends on the priority of the problem and
the conditions stated in service contracts, when applicable.
We categorize Cisco technical support inquiries according to urgency:
•Priority level 4 (P4)—You need information or assistance concerning Cisco product capabilities,
product installation, or basic product configuration. There is little or no impact to your business
operations.
•Priority level 3 (P3)—Operational performance of the network is impaired, but most business
operations remainfunctional.You and Cisco are willing to commit resources during normal business
hours to restore service to satisfactory levels.
•Priority level 2 (P2)—Operation of an existing network is severely degraded, or significant aspects
of your business operations are negatively impacted by inadequate performance of Cisco products.
You and Cisco will commit full-time resources during normal businesshours toresolve the situation.
•Priority level 1 (P1)—An existing network is “down,” or there is a critical impact to your business
operations. You and Cisco will commit all necessary resources around the clock to resolve the
situation.
Cisco Technical Support Website
The Cisco Technical Support Website provides online documents and tools to help troubleshoot and
resolve technical issues with Cisco products and technologies. To access the Cisco Technical Support
Website, go to this URL:
http://www.cisco.com/techsupport
All customers, partners, and resellers who have a valid Cisco service contract have complete access to
the technical support resources on the Cisco Technical Support Website. Some services on the Cisco
Technical Support Website require a Cisco.com login ID and password. If you have a valid service
contract but do not have a login ID or password, go to this URL to register:
http://tools.cisco.com/RPF/register/register.do

Send comments to [e-mail].
xv
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Obtaining Additional Publications and Information
If you are a Cisco.com registered user, and you cannot resolve your technical issues by using the Cisco
Technical Support Website, you can open a case online at this URL:
http://www.cisco.com/techsupport/caseopen
If you have Internet access, we recommend that you open P3 and P4 cases online so that you can fully
describe the situation and attach any necessary files.
Cisco Technical Support Escalation Center
The Cisco Technical Support Escalation Center addresses priority level 1 or priority level 2 issues. These
classifications are assigned when severe network degradation significantly impacts business operations.
When you contact the Technical Support Escalation Center with a P1 or P2 problem, a Cisco technical
support engineer automatically opens a case.
To obtain a directory of toll-free Cisco Technical Supporttelephone numbers for your country, go to this
URL: http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml
Before calling, please check with yournetwork operations center todeterminethe Cisco support services
to which your company is entitled: for example, SMARTnet, SMARTnet Onsite, or Network Supported
Accounts (NSA). When you call the center, please have available your service agreement number and
your product serial number.
Obtaining Additional Publications and Information
Information about Cisco products, technologies, and network solutions is available from various online
and printed sources.
•The Cisco Product Catalog describes the networking products offered by Cisco Systems, as well as
ordering and customer support services. Access the Cisco Product Catalog at this URL:
http://www.cisco.com/en/US/products/products_catalog_links_launch.html
•Cisco Press publishes a wide range of networking publications. Cisco suggests these titles for new
and experienced users: Internetworking Terms and Acronyms Dictionary, Internetworking
Technology Handbook, Internetworking Troubleshooting Guide, and the Internetworking Design
Guide. For current Cisco Press titles and other information, go to Cisco Press online at this URL:
http://www.ciscopress.com
•Packet magazine is the Cisco quarterly publication that provides the latest networking trends,
technology breakthroughs, and Cisco products and solutions to help industry professionals get the
most from their networking investment. Included are networking deployment and troubleshooting
tips, configuration examples, customer case studies, tutorials and training, certification information,
and links to numerous in-depth online resources. You can access Packet magazine at this URL:
http://www.cisco.com/go/packet
•iQ Magazine is the Cisco bimonthly publication that delivers the latest information about Internet
business strategies for executives. You can access iQ Magazine at this URL:
http://www.cisco.com/go/iqmagazine
•Internet Protocol Journal is a quarterly journal published by Cisco Systems for engineering
professionals involved in designing, developing, and operating public and private internets and
intranets. You can access the Internet Protocol Journal at this URL:
http://www.cisco.com/en/US/about/ac123/ac147/about_cisco_the_internet_protocol_journal.html

Send comments to [e-mail].
xvi
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Preface
Obtaining Additional Publications and Information
•Training—Cisco offers world-class networking training. Current offerings in network training are
listed at this URL:
http://www.cisco.com/en/US/learning/le31/learning_recommended_training_list.html

CHAPTER
Send comments to [e-mail].
1-1
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
1
Troubleshooting Overview
This chapter introduces the basic concepts, methodology, and tools to use for troubleshooting problems
that may occur when configuring and using a Cisco MDS 9000 Family switch. The two most common
symptoms of problems occurring in a storage network are:
•A host not accessing its allocated storage
•An application not responding after attempting to access the allocated storage
To identify the possible problems, you need to use a variety of tools and understand the overall storage
environment. For this reason, this chapter describes a number of general troubleshooting tools and
procedures in addition to those that are specific to the Cisco MDS 9000 family. This chapter also
provides a plan of attack for investigating storage issues. Refer to the other chapters in this book for
detailed explanations of specific issues.
This chapter includes the following sections:
•Introduction, page 1-1
•Contacting Customer Support, page 1-3
•Using Host Diagnostic Tools, page 1-3
•Using Cisco MDS 9000 Family Tools, page 1-4
•Using Cisco Network Management Products, page 1-18
•Using Other Troubleshooting Products, page 1-22
Introduction
Some basic questions should be answered before you go into too much detail about specific problems
and solutions. A process of elimination can determine which network components have been subject to
change and therefore may be the cause of your problems. The main steps you should follow to identify
a problem in a SAN environment include:
1. Verify physical connectivity and registration to the fabric
2. Verify storage subsystem and server configuration
3. Verify end-to-end connectivity and fabric configuration
This section provides a series of questions that may be useful when troubleshooting a problem with a
Cisco MDS 9000 family switch or connected devices. Use the answers to thesequestionsto plan a course
of action and to determine the scope of the problem. For example, if a host can only see some of the

Send comments to [e-mail].
1-2
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Chapter 1 Troubleshooting Overview
Introduction
LUNs, (but not all of them) on an existing subsystem, then fabric-specific issues (FSPF, ISLs, FCNS) do
not need to be investigated, as they are currently working correctly. The fabric components can therefore
be eliminated from the problem.
By answering the questions in the following subsections, you can determine the paths you need to follow
and the components that you should investigate further. These questions are independent of host, switch
or subsystem vendor.
Basics
•Is this a newly installed system or an existing installation? (It could be a new SAN, host or
subsystem, or new LUNs exported to an existing host.)
•Has the host ever been able to see its storage?
•Does the host recognize any LUNs in the subsystem?
•Are you trying to solve an existing application problem (too slow, too high latency, excessively long
response time) or did the problem show up recently?
•What changed in the configuration or in the overall infrastructure immediately before the
applications started to have problems?
Basic Connectivity
•Are you using the correct fiber (SM or MM)?
•Did you check for a broken fiber?
•Is the LED on the connected module Fibre Channel port green, and do the LEDs on any
HBA/Storage Subsystem ports indicate normal functionality?
•Is there a LUN masking policy applied on the storage subsystem? If yes, is the server allowed to see
the LUNs exported by the storage array?
•Is there any LUN masking policy configured on the host? Did you enable the server to see all the
LUNs it can access?
•If LUN masking software is used, is the host’s PWWN listed in the LUN masking database?
•Is the subsystem configured for NPort?
Examine the FLOGI database on the two switches that are directly connected to the host HBA and
subsystem ports. Also, verify that both ports (attached port on MDS-A and MDS-B) are members of the
same VSAN. If both devices are listed in the FCNS database then ISLs are not an issue.
Fibre Channel End-to-End Connectivity
Answering the following questions will help todetermine ifend-to-end fibre channel connectivity exists
from a host or subsystem perspective:
•Does the host list the subsystem’s WWPN or FCID in its logs?
•Does the subsystem list the host’s WWPN or FCID in its logs or LUN masking database?
•Can the host complete a port login (PLOGI) to the storage subsystem?
•Is there any SCSI exchange that takes place between the server and the disk array?
•Is the HBA configured for NPort?

Send comments to [e-mail].
1-3
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Chapter 1 Troubleshooting Overview
Contacting Customer Support
You can use the HBA configuration utilities or the host system logs to determine if the subsystem
PWWN or FCID is listed as a device. This can validate that FSPF is working correctly.
Fabric Issues
•Are both the host bus adapter (HBA) and the subsystem port successfully registered with the fabric
name server?
•Does the correct PWWN for the Server HBA and the storage subsystem port show up on the correct
port in the FLOGI database? In other words, is the device plugged into the correct port?
•Does any single zone contain both devices? The zone members can be WWNs, FCIDs.
•Is the zone correctly configured and part of the active configuration or zoneset within the same
VSAN?
•Do the ISLs show any VSAN isolation?
•Do the host and storage belong to the same VSAN?
•Are anyparameters, such as FSPF, Static Domain Assignment, VSAN or Zoning, mismatched in the
configuration of the different switches in the fabric?
Contacting Customer Support
If you are unable to solve a problem after using the troubleshooting suggestions in this guide, contact a
customer service representative for assistance and further instructions. Before you call, have the
following information ready to help your service provider assist you as quickly as possible:
•Date you received the switch
•Chassis serial number (located on a label on the right of the rear panel of the chassis)
•Type of software and release number
•Maintenance agreement or warranty information
•Brief description of the problem
•Brief explanation of the steps you have already taken to isolate and resolve the problem
After you have collected this information, refer to Obtaining Technical Assistance in the Preface.
Using Host Diagnostic Tools
Most host systems provide utilities or other tools that you can use for troubleshooting the connection to
the allocated storage. For example, on a Windows system, you can use the Diskmon or Disk Management
tool to verify accessibility of the storage and to perform some basic monitoring and administrative tasks
on the visible volumes.
Alternatively, you can use Iometer,an I/O subsystem measurement and characterization tool, to generate
a simulated load and measure performance. Iometer is a public domain software utility for Windows,
originally written by Intel, that provides correlation functionality to assist with performance analysis.
Iometer measures the end-to-end performance of a SAN without cache hits. This can be an important
measurement because if write or read requests go to the cache on the controller (a cache hit) rather than
to the disk subsystems, performance metrics will be artificially high. You can obtain Iometer from
SourceForge.net at the following URL:

Send comments to [e-mail].
1-4
Cisco MDS 9000 Family Troubleshooting Guide
OL-5183-02, Cisco MDS SAN-OS Release 1.3
Chapter 1 Troubleshooting Overview
Using Cisco MDS 9000 Family Tools
http://sourceforge.net/projects/iometer/
Iometer is not the only I/O generator you can use to simulate traffic through the SAN fabric. Other
popular I/O generators and benchmark tools used for SAN testing include Iozone and Postmark. Iozone
is a file system benchmark tool that generates and measures a variety of file operations. It has been
ported to many systems and is useful for performing a broad range of file system tests and analysis.
Postmark was designed to create a large pool of continually changing files, which simulates the
transaction rates of a large Internet mail server.
PostMark generates an initial pool of random text files in a configurable range of sizes. Creation of the
pool produces statistics on continuous small file creation performance. Once the pool is created,
PostMark generates a specified number of transactions, each of which consists of a pair of smaller
transactions:
•Create file or Delete file
•Read file or Append file
Benchmark is available from Network Appliance, Inc. at the following URL:
http://www.netapp.com/tech_library/3022.html
Benchmarking tools offer a variety of capabilities and you should select the one that provides the best
I/O characteristics of your application environment.
Utilities provided by the Sun Solaris operating system let you determine if the remote storage has been
recognized and exported to you in form of a raw device or mounted file system, and to issue some basic
queries and tests to the storage. You can measure performance and generate loads using the iostatutility,
the perfmeter GUI utility, the dd utility, or a third-party utility like Extreme SCSI.
Every UNIX version provides similar utilities, but this guide only provides examples for Solaris. Refer
to the documentation for your specific operating system for details.
Using Cisco MDS 9000 Family Tools
If the server does not see its storage and you cannot use the information available on the host side to
determine the root cause of the problem, you can obtain additional information from a different
viewpoint using the troubleshooting tools provided with the Cisco MDS 9000 family of switches. This
section introduces these tools and describes the kinds of problems for which you can use each tool. It
includes the following topics:
•Command-Line-Interface (CLI), page 1-5
•CLI Debug, page 1-5
•FC Ping and FC Traceroute, page 1-7
•Cisco Fabric Manager, page 1-9
•SCSI Target Discovery, page 1-13
•SNMP and RMON Support, page 1-14
•Using RADIUS, page 1-15
•Using Syslog, page 1-16
•Using Fibre Channel SPAN, page 1-16