Dell Networking S5000 User manual

Humair Ahmed
Dell Technical Marketing –Data Center Networking
August 2013
Dell Networking S5000:
Deployment of a Converged Infrastructure
with FCoE
Deployment/Configuration Guide

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
2 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
This document is for informational purposes only and may contain typographical errors. The
content is provided as is, without express or implied warranties of any kind.
© 2013 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions
in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc. Intel and
Xeon are registered trademarks of Intel Corporation in the U.S. and other countries. Microsoft,
Windows, and Windows Server are either trademarks or registered trademarks of Microsoft
Corporation in the United States and/or other countries. Other trademarks and trade names may be
used in this document to refer to either the entities claiming the marks and names or their products.
Dell disclaims proprietary interest in the marks and names of others.
August 2013 | Rev 3.5

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
3 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Contents
Overview................................................................................................................................................................. 4
A: Converged Network Solution - Dell PowerEdge Server, Dell Compellent storage array, and Dell
S5000 as NPIV Proxy Gateway...................................................................................................................... 4
B: Converged Network Solution - Dell PowerEdge Server, Dell PowerVault storage array, and Dell
S5000 as NPIV Proxy Gateway.................................................................................................................... 33
C: Using Dell S4810 or Dell MXL/IOA Blade switch as a FIP-snooping Bridge........................................ 57
D: FCoE CNA adapter configuration specifics ............................................................................................... 65
Broadcom BCM57810S................................................................................................................................. 65
Creating a NIC Team .................................................................................................................................68
Dell QLogic QLE8262.................................................................................................................................... 74
Creating a NIC Team. ................................................................................................................................ 76

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
4 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Overview
In the “Dell Networking S5000: The Building Blocks of Unified Fabric and LAN/SAN Convergence”
whitepaper we demonstrated and explained the movement from a traditional non-converged
LAN/SAN network to a converged LAN/SAN infrastructure and how the Dell S5000 switch is an ideal
solution for this transition. In addition, we covered the many benefits of moving to a converged
infrastructure such as less maintenance and considerable cost savings. The Dell S5000 converged
switch with its unique modular design allows end users to migrate to a converged solution and
increase port count at their own pace without replacing the entire switch. This benefit is unmatched in
the industry. In this whitepaper we cover detailed Dell S5000 topology and configuration examples.
A: Converged Network Solution - Dell PowerEdge Server, Dell
Compellent storage array, and Dell S5000 as NPIV Proxy
Gateway
We will first demonstrate a non-converged setup and then add the Dell S5000 to the picture. This will
allow us to see how the connections and configuration change from a traditional non-converged
environment to a converged environment with the introduction of the Dell S5000 switch. You’ll be
surprised how easy the setup is and how the backend LAN and SAN can remain untouched.
The traditional LAN/SAN non-converged setup example is shown below in Figure 1. As you can see, a
Dell PowerEdge R720 server with a 2-port FC HBA is used to connect to two FC switches which are
then connected to a Dell Compellent storage array composed of two SC8000 controllers and one
SC220 enclosure. Each FC port is connecting to a different fabric. Windows Server 2008 R2 Enterprise
is installed on the server.
In the below setup, the LAN side will be the usual setup with an active/standby configuration up to
separate ToR Dell S4810 switches which have VLT employed up to the core Z9000 switches.
For the below diagram I’ll focus in on the SAN configuration.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
5 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 1: Traditional LAN/SAN non-converged network
The Dell Compellent Storage Center controllers are used to support various I/O adapters including FC,
iSCSI, FCoE, and SAS. A Dell Compellent Storage Center consists of one or two controllers, FC
switches, and one or more enclosures. In the above example, two Compellent SC8000 controllers,
one Compellent SC220 enclosure, two FC switches, and one 4-port FC HBA card on each Compellent
controller is used for the SAN network. The FC switches provide robust connectivity to servers,
allowing for the use of multiple controllers and redundant transport paths.
SAS enclosures hold disks for data storage and connect to the controllers through back-end ports via
SAS cables; you can see how the SC220 enclosure and controllers are cabled together in Figure 1
above.
To keep the diagram uncluttered yet detailed, the only connections not shown are the ‘eth0’ ports on
each controller connecting to the management network and the ‘eth1’ port on each controller
connecting to the ‘eth1’ port on the other controller. The ‘eth0’ connection supports system login and
access for the software. It’s used to send emails, alerts, SNMP traps, and Phone Home data. The ‘eth1’
connection is used for dedicated Inter-Process Communication (IPC) between controllers in a dual-
controller Storage Center. There is no default gateway for ‘eth1’and it does not need to be set. See the
“CT-SC040 and SC8000 Connectivity Guide” and “Compellent Storage Center System Setup Guide” to
get started on cabling and configuring your Compellent storage array.
In this example setup, two SC8000 controllers and one SC220 disk enclosure have been cabled
together. There are two paths available from the server to the FC switches and four paths available
from each FC switch to the Compellent storage array.
Compellent SC8000 Load Balancing Policy Options:
The Compellent SC8000 controller uses Microsoft Multipath I/O (MPIO) for load balancing over ports.
Microsoft MPIO is a framework that allows administrators to configure load balancing and failover

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
6 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
processes for FC and iSCSI connected storage devices. You can configure load balancing to use up to
32 independent paths from the connected storage devices. The MPIO framework uses Device Specific
Modules (DSM) to allow path configuration. For Windows Server 2008 and above, Microsoft provides a
built-in generic Microsoft DSM (MSDSM) and it should be used. For Windows Server 2003 only, Dell
Compellent provides a DSM.
A load balance policy is used to determine which path is used to process I/O. Once the Compellent
volume has been created and mapped accordingly as will be demonstrated shortly, to see the selected
MPIO policy in Windows Server 2008 R2 Enterprise navigate to ‘Start->Administrative Tools-
>Computer Management’. On the left-hand pane navigate to ‘Computer Management->Storage->Disk
Management’ and right click the disk created on the Compellent storage array and select ‘Properties’.
Next, select the ‘Hardware’ tab, click the ‘Properties’ button at the bottom right, and select the ‘MPIO’
tab. Figure 2 below displays what you should see. Note that the default will be “Round Robin”.
The Dell Compellent SC8000 will only support ‘Round Robin’ or ‘Fail Over Only’ MPIO policies. The
other policies require Asymmetric Logical Unit Access (ALUA) support on the array and additional
support through a DSM.
Figure 2: Checking MPIO settings in Windows Server 2008 R2 Enterprise
Additionally, there are two IO connection options available with the Dell Compellent Storage Center
that allow multiple paths to be presented to the servers: ‘Legacy Ports’and ‘Virtual Ports’. You will be
asked which one you would like to use when initially setting-up the Compellent Storage Center and

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
7 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
configuring the FC IO cards. See the “Storage Center 6.2 System Setup Guide” for more information
on initial setup of the Dell Compellent Storage Center.
In legacy mode, front-end IO ports (in this case FC ports) are broken into primary and reserve ports
based on a fault domain. The reserve port is in a standby mode until a primary port fails over to the
reserve port. In terms of MPIO, this requires twice the IO ports to enable multiple paths. For
redundancy, a primary port connects to one controller, and the reserved port in that fault domain
connects to the other controller. While this is a highly robust failover solution, it requires a large
number of ports.
Dell Compellent introduced virtual ports in Storage Center 5.0. Virtual ports allow all front-end IO
ports to be virtualized. All FC ports can be used at the same time for load balancing as well as failover
to another port. Although a virtual disk can still only be written to from the controller that owns the
disk, virtual ports allow for better performance in terms of failover as the virtual connection can simply
be moved to another physical port in the same fault domain. To use virtual ports all FC switches and
HBAs must support N_Port ID Virtualization (NPIV). See the “Dell Compellent Storage Center Microsoft
Multipath IO (MPIO) Best Practices Guide” for more information on multipathing with Microsoft
Windows 2008 R2 Enterprise.
The two FC switches I am using are Brocade 6505s and the zoning configurations are below. The
WWPNs starting with ‘10’ are the FC HBA WWPNs and the other WWPNs are for the Compellent
storage array.
Figure 3: Zoning for fabric A FC switch
Figure 4: Zoning for fabric B FC switch
> zonecreate
financeServer1_p1_test,"10:00:8c:7c:ff:30:7d:28;50:00:d3:10:00:ed:b2:3d;50:00:d3:10:00:ed:b2:43;
50:00:d3:10:00:ed:b2:3b;50:00:d3:10:00:ed:b2:41"
> cfgcreate zoneCfg_test,"financeServer1_p1_test"
> cfgenable zoneCfg_test
> cfgsave
> zonecreate
financeServer1_p2_test,"10:00:8c:7c:ff:30:7d:29;50:00:d3:10:00:ed:b2:3c;50:00:d3:10:00:ed:b2:42;
50:00:d3:10:00:ed:b2:3a;50:00:d3:10:00:ed:b2:40"
> cfgcreate zoneCfg_test,"financeServer1_p2_test"
> cfgenable zoneCfg_test
> cfgsave

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
8 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
During initial configuration of the Compellent Storage Center, we created a disk pool labeled “Pool_1”
consisting of seven 300 GB drives. The total disk space is 1.64 TB; this can be seen in the screen shot
of the Storage Center System Manager GUI as shown below in Figure 5.
Figure 5: Storage Center System Manager GUI displays disk pool “Pool_1” with 1.64 TB free space
Since we have two fabrics, fabric A and fabric B, we create two fault domains. Domain 1 is already
created by default and all the FC ports are currently in domain 1. To create another domain click
‘Storage Management’ on the top left of the webpage and then select ‘System->Setup->Configure
Local Ports’. Next, click the ‘Edit Fault Domains’ button at the bottom right of the dialog box. On the
next dialog box click the ‘Create Fault Domain’ button on the lower right of the dialog box. In the
‘Name’ field type a name for the new domain. In this case, we used “Domain 2”. Make sure ‘FC’ is
selected in the ‘Type’ field and click ‘Continue’. Figure 6 below shows that we have already created the
second domain.
Figure 6: Creating an additional Fault Domain on Compellent Storage Array
Now we can navigate back to the ‘Configure Local Ports’ dialog and select the appropriate Domain to
put each port in. Each fabric should be in its own Domain; we put all ports going to fabric A in Domain
1 and all ports going to fabric B in Domain 2 as shown below.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
9 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 7: Assigning ports on Compellent Storage to respective Fault Domains
If you get a warning that paths are not balanced, navigate to the left-hand pane, right click
‘Controllers’ and select ‘Rebalance Local Ports’.
Next, a server object needs to be created and the respective FC ports have to be selected to be used
by the server object. This can be accomplished by right clicking ‘Servers’ on the left pane and selecting
‘Create Server’. In Figure 8 below, you can see a server object named “Finance_Server” was created
that includes both of the FC ports on the FC HBA card.
Figure 8: Added Dell PowerEdge Server HBAs to ‘Server Object’ on Dell Compellent Storage Array
The next step is to enable mulipathing on Windows Server 2008 R2 Enterprise. Navigate to
‘Start->Administrative Tools->Server Manager->Features->Add Features’and select ‘Multipath I/O’.
You can see in Figure 9 below that we have already installed the ‘Multipath I/O’ feature.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
10 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 9: Installing Windows Server 2008 R2 Enterprise Multipath I/O feature
Now navigate to ‘Start->Control Panel->MPIO’and click the ‘Add’ button. When prompted for a
‘Device Hardware ID’, input “COMPELNTCompellent Vol” and click the ‘OK’ button. The system will
need to be restarted for the changes to take effect. Figure 10 displays the “COMPELNTCompellent Vol”
text that you should see on the MPIO Devices tab in ‘MPIO Properties’once the system is brought
back up.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
11 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 10: Installing Windows Server 2008 R2 Enterprise Multipath I/O for Compellent array
Next, create a volume and map it to a server object so the respective server can write to the FC
storage array. Simply right click ‘Volumes’ on the left-hand pane and select ‘Çreate Volume’ to get
started. During the process, you will be asked to select a ‘Replay Profile’; this is simply asking you how
often ‘snapshots/recovery points’ of the storage volume should be taken. A ‘snapshot/recovery point’
allows you to revert a volume back to a certain point in time (for example if files are accidentally
deleted). In Figure 11 below, you can see that a 20 GB volume named “Finance_Data_Compellent” has
already been created. Figure 12 displays the dialog box where you can select a ‘Replay Profile’.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
12 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 11: Created 20 GB “Finance_Data_Compellent” volume on Compellent array

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
13 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 12: Confirming to keep the default value for ‘Replay Profiles’
The last step in configuring the Dell Compellent Storage Center array is mapping the newly created
volume to the server. Once you create the volume, you will be asked if you want to map it to a server
object. You can do it at this time or later. If mapping the volume to a server object later, on the left-
hand pane under ‘Storage->Volumes’, simply right click on the volume you just created and select
‘Map Volume to Server’. You can then select the respective server object that you created prior.
As soon as the HBA on the Windows server detects storage available for it, it will be detected in the
Windows disk management administration tool after performing a disk scan. To perform a disk scan,
right click ‘Disk Management’ on the left-hand pane and select ‘Rescan Disks’. You must right click the
detected virtual disk and initialize it. Below in Figure 13, you can see we have already initialized the disk
(Disk 1) and formatted it as NTFS.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
14 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 13: Initialized and formatted virtual disk within Windows Server 2008 R2 Enterprise
Now the volume on the Compellent storage array displays in Windows just like a typical hard drive.
Note, no special configuration was needed on the HBA.
Figure 14: Remote storage on Compellent as seen in Windows as drive ‘T:’

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
15 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
To observe that the storage ports and HBA ports are logged into the fabric, you can use the ‘nsshow’
command on the Brocade FC switch as shown below in Figure 15. Note that since the command is
run on the fabric A FC switch, only eight storage ports and one host FC HBA port is logged into the
fabric as expected. The reason we see eight storage ports instead of four is because we are using
virtual port mode on the Dell Compellent array so we are seeing the physical WWPN and the virtual
WWPN. We would see similar (with different WWPNs) on the fabric B FC switch.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
16 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 15: Node logins on the fabric A FC switch
You can also see the node WWPN by looking at what is logged in on the physical port as shown in
Figure 16 below.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
17 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Figure 16: Check WWPNs logged in on port 2 of fabric A FC switch
We can use the respective port WWPNs to create a specific zoning configuration such as that
displayed below in Figure 17.
Figure 17: Zoning configuration created on fabric A FC switch
On the fabric A FC switch you can see the WWPN of the server HBA port is ‘10:00:8c:7c:ff:30:7d:28’
and the WWPNs of the storage ports are ‘50:00:d3:10:00:ed:b2:3d’, ‘50:00:d3:10:00:ed:b2:43,

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
18 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
‘50:00:d3:10:00:ed:b2:3b’, and ‘50:00:d3:10:00:ed:b2:41’. This zoning configuration is allowing all four
storage ports to communicate only to each other and the server FC HBA node.
On the fabric B FC switch you can see the WWPN of the server HBA port is ‘10:00:8c:7c:ff:30:7d:28’
and the WWPNs of the storage ports are ‘50:00:d3:10:00:ed:b2:3c’, ‘50:00:d3:10:00:ed:b2:42’,
‘50:00:d3:10:00:ed:b2:3a’, and ‘50:00:d3:10:00:ed:b2:40’.
Another useful FC switch command to check what ports are connected is ‘switchshow’.
Figure 18: ‘switchshow’ command on fabric A FC switch displaying connections on FC ports
As you can see in Figure 18 above, since we are using virtual port mode on the Dell Compellent
storage array, instead of the normal F_Port text as shown on port 2 which is connected to the FC HBA
on the server, we see “1 N Port + 1 NPIV public”. In this case the F_Port is actually a VF_Port and the
N_Port is actually a NV_Port.
Note, both controllers on the Compellent storage array are active and each fabric has two paths to
controller A and two paths to controller B. They are all logged into the fabric. Unlike in legacy mode,
with virtual port mode, a virtual connection from a VN_Port can fail over to another physical port in
the same domain as long as the port being failed over to is on the controller that is the primary
controller for the volume. In legacy mode, in this case, four ports would be reserved for failover. See
Compellent documentation for more information on Compellent configuration.

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
19 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
Adding the Dell S5000 Converged Switch to the Topology
In Figure 19, you can see how the traditional non-converged topology has changed with the
introduction of the Dell S5000 switch in a possible use case. Note how the Dell S4810 Ethernet
switches have been replaced by Dell S5000 converged switches. Also, note how the separate Ethernet
NIC and FC adapters on the server have been replaced by one converged network adapter (CNA). FC
frames are now encapsulated in Ethernet frames and both LAN and SAN traffic are carried over the
same Ethernet links up to the Dell S5000 which separates the two different types of traffic. For
different possible use cases of the Dell S5000, see the “Dell Networking S5000: Data Center ToR
Architecture and Design”document.
Figure 19: Dell S5000 acting as a NPIV Proxy Gateway and allowing for a converged infrastructure
It’s important to note that as long as the appropriate drivers for both FC and Ethernet are installed, the
operating system can see two CNA ports as multiple Ethernet ports and FC HBA ports if NIC
partitioning (NPAR) is employed. Figure 20 displays how Windows logically sees a CNA card with two
ports with NPAR and FCoE Enabled as a 2-port NIC and a 2-port FC HBA.
Figure 20: Windows view in Device Manager of one Dell QLogic QLE8262 CNA with NPAR and FCoE enabled

Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
20 Deployment of a Converged Infrastructure with FCoE
Deployment/Configuration Guide
As in the traditional non-converged setup, the LAN side will be the usual setup with either an
active/standby configuration up to separate ToR Dell S5000 switches which have VLT employed up to
the core Z9000 switches. The difference here is that the Ethernet ports connecting up to the ToR are
virtual ports.
The Dell PowerEdge R720 server has its virtual Ethernet NICs configured via NIC teaming and
connecting to two separate Dell S5000 switches. The virtual HBA ports are connecting to the same
Dell S5000 switches but are logically separated from the Ethernet NICs and the NIC teaming
configuration is not taken into account.
Figure 21: Logical view of how operating system sees CNA with NPAR and FCoE enabled
Since we are using a Dell QLogic QLE8262 CNA, the first thing we need to do is configure it for FCoE.
Note, since we NIC team with ‘Failsafe Team’, no configuration is required on the S5000 switches and
the switches are not aware of the NIC team. One member of the team is active while the other is
standby. See the “Dell QLogic QLE8262” section in section D: “FCoE CNA adapter configuration
specifics” for details of the configuration.
As no change is required on the backend LAN/SAN networks except for some zoning/access controls,
the main task in the new topology is the configuration of the Dell S5000 switches for both fabric A and
fabric B. This configuration is shown below in Figure 22 and Figure 23.