Brocade Communications Systems A7533A - Brocade 4Gb SAN Switch Base Operating instructions

53-1000853-01
12 March 2008
Fabric OS
Troubleshooting and Diagnostics Guide
Supporting Fabric OS v6.1.0

Copyright © 2008 Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing symbol,
DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or
services of their respective owners.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning
any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to
this document at any time, without notice, and assumes no responsibility for its use. This informational document describes
features that may not be currently available. Contact a Brocade sales office for information on feature and product availability.
Export of technical data contained in this document may require an export license from the United States government.
The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any person or entity with
respect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs that
accompany it.
The product described by this document may contain “open source” software covered by the GNU General Public License or other
open source license agreements. To find-out which open source software is included in Brocade products, view the licensing
terms applicable to the open source software, and obtain a copy of the programming source code, please visit
http://www.brocade.com/support/oscd.
Brocade Communications Systems, Incorporated
Document History
Corporate Headquarters
Brocade Communications Systems, Inc.
1745 Technology Drive
San Jose, CA 95110
Tel: [phone]
Fax: [phone]
Email: [e-mail]
Asia-Pacific Headquarters
Brocade Communications Singapore Pte. Ltd.
9 Raffles Place
#59-02 Republic Plaza 1
Singapore 048619
Tel: [phone]
Fax: [phone]
Email: [e-mail]
European and Latin American Headquarters
Brocade Communications Switzerland Sàrl
Centre Swissair
Tour A - 2ème étage
29, Route de l'Aéroport
Case Postale 105
CH-1215 Genève 15
Switzerland
Tel: [phone]
Fax: [phone]
Email: emea[email protected]
Title Publication number Summary of changes Date
Fabric OS Troubleshooting and
Diagnostics Guide
53-0000853-01 First released edition. 12 March 2008

Fabric OS Troubleshooting and Diagnostics Guide iii
53-1000853-01
Contents
About This Document
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Supported hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . . x
What’s new in this document. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
Document conventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Text formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Notes, cautions, and warnings . . . . . . . . . . . . . . . . . . . . . . . . . . .xi
Key terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Additional information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Brocade resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Other industry resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Getting technical help. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Document feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
Chapter 1 Introduction to Troubleshooting
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
About troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Network time protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Most common problem areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Questions for common symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Gathering information for your switch support provider . . . . . . . . . . 5
Setting up your switch for FTP. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Capturing a supportSave. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Capturing a supportShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Capturing output from a console . . . . . . . . . . . . . . . . . . . . . . . . . 6
Capturing command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Building a case for your switch support provider . . . . . . . . . . . . . . . . 7
Basic switch information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Detailed problem information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 2 General Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Licensing issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

iv Fabric OS Troubleshooting and Diagnostics Guide
53-1000853-01
Switch Message Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Checking fan components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Checking the switch temperature. . . . . . . . . . . . . . . . . . . . . . . . 13
Checking the power supply . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Checking the temperature, fan, and power supply . . . . . . . . . .13
Fibre Channel Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Checking for Fibre Channel connectivity problems . . . . . . . . . .14
Third party applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 3 Connections Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Port initialization and FCP auto discovery process . . . . . . . . . . . . . . 17
Link issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Connection problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Checking the logical connection . . . . . . . . . . . . . . . . . . . . . . . . . 19
Checking the name server (NS) . . . . . . . . . . . . . . . . . . . . . . . . .20
Link failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Determining a successful negotiation . . . . . . . . . . . . . . . . . . . . 22
Checking for a loop initialization failure . . . . . . . . . . . . . . . . . . .22
Checking for a point-to-point initialization failure . . . . . . . . . . .23
Correcting a port that has come up in the wrong mode . . . . . .23
Marginal links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Troubleshooting a marginal link . . . . . . . . . . . . . . . . . . . . . . . . . 24
Device login issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Pinpointing problems with device logins . . . . . . . . . . . . . . . . . .27
Media-related issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Testing a port’s external transmit and receive path . . . . . . . . . 30
Testing a switch’s internal components. . . . . . . . . . . . . . . . . . . 30
Testing components to and from the HBA . . . . . . . . . . . . . . . . .30
Segmented fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
Reconciling fabric parameters individually . . . . . . . . . . . . . . . .31
Downloading a correct configuration . . . . . . . . . . . . . . . . . . . . . 32
Reconciling a domain ID conflict . . . . . . . . . . . . . . . . . . . . . . . . 32
Chapter 4 Configuration Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Configupload and download issues. . . . . . . . . . . . . . . . . . . . . . . . . .33
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 35
Brocade configuration form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Chapter 5 FirmwareDownload Errors
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Blade troubleshooting tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37
Firmware download issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Fabric OS Troubleshooting and Diagnostics Guide v
53-1000853-01
Troubleshooting firmwareDownload . . . . . . . . . . . . . . . . . . . . . . . . . 41
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 41
Brocade DCX error handling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
USB error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Considerations for downgrading firmware . . . . . . . . . . . . . . . . . . . .44
Preinstallation messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Blade types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Firmware versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
IP settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Port settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
Chapter 6 Security Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Password issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Password recovery options . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58
Protocol and certificate management issues . . . . . . . . . . . . . . . . . .58
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 59
SNMP issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 60
FIPS issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Chapter 7 ISL Trunking Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Link issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Buffer credit issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Chapter 8 Zone Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Overview of corrective action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
Verifying a fabric merge problem . . . . . . . . . . . . . . . . . . . . . . . . 63
Segmented fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
Zone conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64
Correcting a fabric merge problem quickly . . . . . . . . . . . . . . . . 65
Editing zone configuration members . . . . . . . . . . . . . . . . . . . . . 66
Reordering the zone member list . . . . . . . . . . . . . . . . . . . . . . . .66
Checking for Fibre Channel connectivity problems . . . . . . . . . .66
Checking for zoning problems. . . . . . . . . . . . . . . . . . . . . . . . . . .68
Gathering additional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

vi Fabric OS Troubleshooting and Diagnostics Guide
53-1000853-01
Chapter 9 FCIP Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
FCIP tunnel issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
FCIP links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 72
Port mirroring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Supported hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73
Port mirroring considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Port mirroring management . . . . . . . . . . . . . . . . . . . . . . . . . . . .75
FTRACE concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76
Tracing Fibre Channel information . . . . . . . . . . . . . . . . . . . . . . . 76
Chapter 10 FICON Fabric Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
FICON issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Troubleshooting FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80
General information to gather for all cases . . . . . . . . . . . . . . . . 80
Identifying ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81
Single-switch topology checklist . . . . . . . . . . . . . . . . . . . . . . . . . 82
Cascade mode topology checklist . . . . . . . . . . . . . . . . . . . . . . . 82
Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 82
Troubleshooting FICON CUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83
Troubleshooting FICON NPIV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85
Chapter 11 iSCSI Issues
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90
Chapter 12 Working With Diagnostic Features
In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
About Fabric OS diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91
Diagnostic information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92
Power-on self test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92
Switch status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Viewing the overall status of the switch. . . . . . . . . . . . . . . . . . .94
Displaying switch information . . . . . . . . . . . . . . . . . . . . . . . . . . .95
Displaying the uptime for a switch . . . . . . . . . . . . . . . . . . . . . . .95

Fabric OS Troubleshooting and Diagnostics Guide vii
53-1000853-01
Port information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Viewing the status of a port . . . . . . . . . . . . . . . . . . . . . . . . . . . .96
Displaying the port statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . .97
Displaying a summary of port errors for a switch . . . . . . . . . . .98
Equipment status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Displaying the status of the fans . . . . . . . . . . . . . . . . . . . . . . . .99
Displaying the status of a power supply. . . . . . . . . . . . . . . . . . .99
Displaying temperature status . . . . . . . . . . . . . . . . . . . . . . . . .100
System message log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100
Displaying the system message log, with no page breaks . . .100
Displaying the system message log one message at a time .100
Clearing the system message log . . . . . . . . . . . . . . . . . . . . . . .101
Port log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
Viewing the port log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
Syslogd configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102
Configuring the host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
Configuring the switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103
Automatic trace dump transfers . . . . . . . . . . . . . . . . . . . . . . . . . . .104
Specifying a remote server . . . . . . . . . . . . . . . . . . . . . . . . . . . .105
Enabling the automatic transfer of trace dumps. . . . . . . . . . .105
Setting up periodic checking of the remote server . . . . . . . . .105
Saving comprehensive diagnostic files to the server . . . . . . .105
Diagnostic tests not supported by M-EOS 9.6.2 and FOS 6.0. . . .106
Appendix A Switch Type
Appendix B Hexidecimal
Index

viii Fabric OS Troubleshooting and Diagnostics Guide
53-1000853-01

Fabric OS Troubleshooting and Diagnostics Guide ix
53-1000853-01
About This Document
In this chapter
•How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
•Supported hardware and software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
•What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
•Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
•Additional information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
•Getting technical help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
•Document feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
How this document is organized
The document contains the following chapters:
•Chapter 1, “Introduction to Troubleshooting,” gives a brief overview of Fabric OS, explains the
Fabric OS CLI Help feature, and provides typical connection and configuration procedures.
•Chapter 2, “General Issues,” provides information on licensing, hardware, and syslog issues.
•Chapter 3, “Connections Issues,” provides information and procedures on managing
authentication and user accounts for the switch management channel.
•Chapter 4, “Configuration Issues,” provides information and procedures for configuring ACL
policies for FC port and switch binding and managing the fabric-wide consistency policy.
•Chapter 5, “FirmwareDownload Errors,” provides procedures for maintaining and backing up
your switch configurations.
•Chapter 6, “Security Issues,” provides procedures for basic password and user account
management.
•Chapter 7, “ISL Trunking Issues,” describes the concepts and provides procedures for using
administrative domains.
•Chapter 8, “Zone Issues,” provides preparations and procedures for performing firmware
downloads, as well troubleshooting information.
•Chapter 9, “FCIP Issues,” provides information and procedures specific to Brocade 48000 and
Brocade DCX models. Because these models have CP blades and port blades, they require
procedures that are not relevant to the Brocade fixed-port models.
•Chapter 10, “FICON Fabric Issues,” provides information and procedures specific to Brocade
48000 and Brocade DCX models. Because these models have CP blades and port blades, they
require procedures that are not relevant to the Brocade fixed-port models.

xFabric OS Troubleshooting and Diagnostics Guide
53-1000853-01
•Chapter 11, “iSCSI Issues,” provides information and procedures specific to Brocade 48000
and Brocade DCX models. Because these models have CP blades that support the iSCSI
feature.
•Chapter 12, “Working With Diagnostic Features,” provides procedures for use of the Brocade
Adaptive Networking suite of tools, including Traffic Isolation, QoS Ingress Rate Limiting, and
QoS SID/DID Traffic Prioritization.
•The appendices provide special information to guide you in understanding switch output.
Supported hardware and software
In those instances in which procedures or parts of procedures documented here apply to some
switches but not to others, this guide identifies exactly which switches are supported and which are
not.
Although many different software and hardware configurations are tested and supported by
Brocade Communications Systems, Inc. for 6.1.0, documenting all possible configurations and
scenarios is beyond the scope of this document.
The following hardware platforms are supported by this release of Fabric OS:
•Brocade 200E switch
•Brocade 300 switch
•Brocade 4016 switch
•Brocade 4018 switch
•Brocade 4020 switch
•Brocade 4024 switch
•Brocade 4100 switch
•Brocade 4900 switch
•Brocade 5000 switch
•Brocade 5100 switch
•Brocade 5300 switch
•Brocade 7500 switch
•Brocade 7600 switch
•Brocade 48000 director
•Brocade DCX Backbone
What’s new in this document
This manual is a compilation of all the information originally distributed through out the Fabric OS
Administrator’s Guide chapters.