Dell EMC PowerEdge MX Series User manual

Dell EMC PowerEdge MX
Networking Architecture Guide
January 2019
Architecture Guide
Abstract
This architecture guide provides an overview of the architecture,
features, and functionality of the PowerEdge MX networking
infrastructure.
Dell EMC Solutions

Copyright
2
Dell EMC PowerEdge MX
Networking Architecture Guide
The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect
to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular
purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other
trademarks are trademarks of Dell Inc. or its subsidiaries. Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks
of Intel Corporation in the U.S. and/or other countries. Other trademarks may be trademarks of their respective owners.
Published in the USA January 2019 Architecture Guide [Part Number].
Dell Inc. believes the information in this document is accurate as of its publication date. The information is subject to change
without notice.

Contents
3
Dell EMC PowerEdge MX
Networking Architecture Guide
Contents
Chapter 1 Dell EMC PowerEdge MX Platform Overview 4
PowerEdge MX models and components..............................................................5
PowerEdge MX7000 (front)...................................................................................5
PowerEdge MX7000 (rear)....................................................................9
PowerEdge MX7000 chassis fabric slots.............................................................16
Chapter 2 OpenManage Enterprise - Modular Console 18
Introduction.........................................................................................................19
PowerEdge MX7000 initial deployment...............................................................19
PowerEdge MX7000 Multi-Chassis Management groups....................................19
PowerEdge MX7000 OME-M component management......................................22
Chapter 3 PowerEdge MX Scalable Fabric Architecture 25
Scalable Fabric architecture................................................................................26
QSFP28 double density connectors....................................................................28
Interfaces and port groups ..................................................................................29
Embedded Top of Rack switching.......................................................................33
Scalable Fabric OS10EE CLI Commands...........................................................35
Chapter 4 PowerEdge MX Ethernet Switch Operating Modes 36
Operating modes description ..............................................................................37
PowerEdge MX7000 SmartFabric topologies......................................................39
SmartFabric uplink topologies.............................................................................41
Deploying a SmartFabric.....................................................................................46
Appendix A Additional Guidelines and Equipment Details 49
Scalable Fabric restrictions and guidelines .........................................................50
Supported cables and optical connectors............................................................57
Rate limited 32 Gb Fibre Channel.......................................................................64
Spanning Tree Protocol recommendations .........................................................65
NIC teaming restrictions......................................................................................66
PowerEdge MX IOM slot support matrix..............................................................67
PTM port mapping...............................................................................................69
Examples of unsupported Scalable Fabric topologies.........................................71
Appendix B Additional Information 74
Documentation and Support................................................................................75

Chapter 1: Dell EMC PowerEdge MX Platform Overview
4
Dell EMC PowerEdge MX
Networking Architecture Guide
Chapter 1 Dell EMC PowerEdge MX
Platform Overview
This chapter presents the following topics:
PowerEdge MX models and components..........................................................5
PowerEdge MX7000 (rear)..................................................................................9
PowerEdge MX7000 chassis fabric slots ........................................................16

Chapter 1: Dell EMC PowerEdge MX Platform Overview
5
Dell EMC PowerEdge MX
Networking Architecture Guide
PowerEdge MX models and components
The Dell EMC PowerEdge MX, a unified, high-performance data center infrastructure,
provides the agility, resiliency, and efficiency to optimize a wide variety of traditional and
new, emerging data center workloads and applications. With its kinetic architecture and
agile management, PowerEdge MX dynamically configures compute, storage, and fabric,
increases team effectiveness and accelerates operations. Its responsive design and
delivers the innovation and longevity customers of all sizes need for their IT and digital
business transformations.
This section contains information about the hardware and options available in the
PowerEdge MX7000. The section is divided into two parts:
•The front of the MX7000 chassis, containing compute and storage sleds
•The back of the MX7000 chassis, containing networking, storage, and management
components
Figure 1. Dell EMC PowerEdge MX7000 chassis
PowerEdge MX7000 (front)
Figure 2 shows the front view of the Dell EMC PowerEdge MX7000 chassis. The left side
of the chassis can have one of three control panel options:
•LED status light panel
•Touch screen LCD panel
Introduction
Hardware
options
Overview

Chapter 1: Dell EMC PowerEdge MX Platform Overview
6
Dell EMC PowerEdge MX
Networking Architecture Guide
•Touch screen LCD panel equipped with Dell EMC PowerEdge iDRAC Quick Sync 2
The bottom of Figure 2 shows six hot-pluggable, redundant, 3,000-watt power supplies.
Above the power supplies eight slots are available. In the example below, the slots
contain:
•Four Dell EMC PowerEdge MX740c sleds in slots one through four
•One Dell EMC PowerEdge MX840C sled in slots five and six
•Two Dell EMC PowerEdge MX5016s sleds in slots seven and eight
Figure 2. Dell EMC PowerEdge MX7000 –front
The Dell EMC PowerEdge MX740c is a two-socket, full-height, single-width compute sled
that offers impressive performance and scalability. It is ideal for dense virtualization
environments and can serve as a foundation for collaborative workloads. The MX7000
chassis supports up to eight MX740c sleds.
Key features of the MX740c include:
•Single-width slot design
•Two CPU sockets
•24 DIMM slots of DDR4 memory
•Boot options include BOSS-S1 or Internal Dual SD Modules (IDSDM)
•Up to six SAS/SATA SSD/HDD and NVMe PCIe SSDs
•Two PCIe mezzanine card slots for connecting to network Fabric A and B
•One PCIe mini-mezzanine card slots for connecting to storage Fabric C
Dell EMC
PowerEdge
MX740c compute
sled

Chapter 1: Dell EMC PowerEdge MX Platform Overview
7
Dell EMC PowerEdge MX
Networking Architecture Guide
•iDRAC9 with Lifecycle Controller
Figure 3. Dell EMC PowerEdge MX740c sled with six 2.5-inch SAS drives
The Dell EMC PowerEdge MX840c is a powerful four-socket, full-height, double-width
server that features dense compute, exceptionally large memory capacity, and a highly
expandable storage subsystem. It is the ultimate scale-up server that excels at running a
wide range of database applications, substantial virtualization, and software-defined
storage environments. The MX7000 chassis supports up to four MX840c compute sleds.
Key features of the MX840c include:
•Dual-width slot design
•Four CPU sockets
•48 DIMM slots of DDR4 memory
•Boot options include BOSS-S1 or IDSDM
•Up to eight SAS/SATA SSD/HDD and NVMe PCIe SSDs
Dell EMC
PowerEdge
MX840c compute
sled

Chapter 1: Dell EMC PowerEdge MX Platform Overview
8
Dell EMC PowerEdge MX
Networking Architecture Guide
•Four PCIe mezzanine card slots for connecting to network Fabric A and B
•Two PCIe mini-mezzanine card slots for connecting to storage Fabric C
•iDRAC9 with Lifecycle Controller
Figure 4. Dell EMC PowerEdge MX840c sled with eight 2.5-inch SAS drives
The PowerEdge MX5016s storage sled delivers scale-out, shared storage within the
PowerEdge MX architecture. The MX5016s provides customizable 12 GB/s direct-
attached SAS storage with up to 16 SAS HDDs/SSDs. Both the MX740c and the MX840c
compute sleds can share drives with the MX5016s using the PowerEdge MX5000s SAS
module. Internal server drives may be combined with up to seven MX5016s sleds in one
chassis for extensive scalability. The MX7000 chassis supports up to seven MX5016s
storage sleds.
Dell EMC
PowerEdge
MX5016s storage
sled

Chapter 1: Dell EMC PowerEdge MX Platform Overview
9
Dell EMC PowerEdge MX
Networking Architecture Guide
Figure 5. Dell EMC PowerEdge MX5016s sled with the drive bay extended
PowerEdge MX7000 (rear)
The MX7000 includes three I/O fabrics. Fabrics A and B are for Ethernet and future I/O
module connectivity, and Fabric C is for SAS and Fibre Channel (FC) connectivity. Each
fabric provides two slots to provide redundancy. Figure 6 shows the rear of the
PowerEdge MX7000 chassis. From top to bottom the chassis is configured with:
•One Dell EMC Networking MX9116n Fabric Switching Engine (FSE) installed in
fabric slot A1
•One Dell EMC Networking MX7116n Fabric Expander Module (FEM) installed in
fabric slot A2
•Two Dell EMC Networking MX5108n Ethernet switches installed in fabric slots B1
and B2
•Two Dell EMC Networking MXG610s Fibre Channel switches installed in fabric slots
C1 and C2
•Two Dell EMC PowerEdge MX9002m modules that are installed in management
slots MM1 and MM2
Oveview

Chapter 1: PowerEdge MX Platform Overview
10
Dell EMC PowerEdge MX
Networking Architecture Guide
Figure 6. Dell EMC PowerEdge MX7000 –rear
The Dell EMC PowerEdge MX9002m management module controls the overall chassis
power, cooling, and hosts the OpenManage Enterprise - Modular (OME-M) console. Two
external 1G-BaseT Ethernet ports are provided to enable management connectivity and to
connect more MX7000 chassis into a single logical chassis. The MX7000 chassis
supports two MX9002m modules for redundancy. Figure 7 shows a single MX9002m
module and its components.
Figure 7. Dell EMC PowerEdge MX9002m module
The following MX9002m module components are labeled Figure 7:
1. Handle release
2. Gigabit Ethernet port 1
Dell EMC
PowerEdge
MX9002m
management
module

Chapter 1: Dell EMC PowerEdge MX Platform Overview
11
Dell EMC PowerEdge MX
Networking Architecture Guide
3. Gigabit Ethernet port 2
4. ID button and health status LED
5. Power status LED
6. Micro-B USB port
The Dell EMC Networking MX9116n Fabric Switching Engine (FSE) is a scalable, high-
performance, low latency 25 GbE switch purpose-built for the PowerEdge MX platform.
The MX9116n FSE provides enhanced capabilities and cost-effectiveness for enterprise,
mid-market, Tier 2 cloud, and NFV service providers with demanding compute and
storage traffic environments.
In addition to 16 internal 25GbE ports, the MX9116n FSE provides:
•Two 100GbE QSFP28 ports
•Two 100GbE QSFP28 unified ports
•Twelve 2x100GbE QSFP28-Double Density (DD) ports
The QSFP28 ports can be used for Ethernet uplink connectivity (see Section 3.1 for more
information). The unified ports can be used for Ethernet uplink connectivity and supporting
eight 32Gb Fibre Channel ports for SAN connectivity supporting both NPIV Proxy
Gateway (NPG) and direct attach FC capabilities.
The QSFP28-DD ports provide fabric expansion connections for up to nine more MX7000
chassis using the MX7116n Fabric Expander Module. The QSFP28-DD ports also provide
capacity for extra uplinks, VLTi links, and connections to rack servers at 10GbE or 25GbE
using breakout cables. The MX7000 chassis supports up to four MX9116n FSEs in Fabric
A and/or B. See Appendix A for more information.
Figure 8. Dell EMC Networking MX9116n FSE
The following MX9116n FSE components are labeled in Figure 8:
1. Express service tag
2. Storage USB port
3. Micro-B USB console port
4. Power and indicator LEDs
5. Handle release
Dell EMC
Networking
MX9116n Fabric
Switching
Engine

Chapter 1: Dell EMC PowerEdge MX Platform Overview
12
Dell EMC PowerEdge MX
Networking Architecture Guide
6. Two QSFP28 ports
7. Two QSFP28 unified ports
8. 12 QSFP28-DD ports
The Dell EMC Networking MX7116n Fabric Expander Module (FEM) acts as an Ethernet
repeater, taking signals from an attached compute sled and repeating them to the
associated lane on the external QSFP28-DD connector. The MX7116n FEM provides two
QSFP28-DD interfaces, each providing up to eight 25 GbE connections to the chassis.
There is no operating system or switching ASIC on the MX7116n FEM, so it never
requires an upgrade. There is also no management or user interface, making the
MX7116n FEM maintenance-free. The MX7000 chassis supports up to four MX7116n
FEMs in Fabric A and/or B. See Appendix A for more information.
Figure 9. Dell EMC Networking MX7116n FEM
The following MX7116n FEM components are labeled in Figure 9:
1. Express service tag
2. Supported optic LED
3. Power and indicator LEDs
4. Module insertion/removal latch
5. Two QSFP28-DD fabric expander ports
Note: The MX7116n FEM cannot act as a stand-alone switch and must be connected to the
MX9116n FSE to function.
The Dell EMC Networking MX5108n Ethernet switch is targeted at small PowerEdge
MX7000 deployments of one or two chassis. While not a scalable switch, it still provides
high-performance and low latency with a non-blocking switching architecture. The
MX5108n provides line-rate 25 Gbps Layer 2 and Layer 3 forwarding capacity to all
connected servers with no oversubscription.
Besides eight internal 25GbE ports, the MX5108n provides:
•One 40GbE QSFP+ port
•Two 100GbE QSFP28 ports
•Four 10GbE RJ45 BASE-T ports
Dell EMC
Networking
MX7116n Fabric
Expander
Module
Dell EMC
Networking
MX5108n
Ethernet switch

Chapter 1: Dell EMC PowerEdge MX Platform Overview
13
Dell EMC PowerEdge MX
Networking Architecture Guide
The ports can be used to provide a combination of network uplinks, VLT interconnects
(VLTi), or for FCoE connectivity. The MX5108n supports FCoE Initilization Protocol (FIP)
Snooping Bridge (FSB) mode but does not support NPG or direct attach FC capabilities.
The MX7000 chassis supports up to four MX5106n Ethernet switches in Fabric A and/or
B. See Appendix A for more information.
Figure 10. Dell EMC Networking MX5108n Ethernet switch
The following MX5108n components are labeled in Figure 10:
1. Express service tag
2. Storage USB port
3. Micro-B USB console port
4. Power and indicator LEDs
5. Module insertion/removal latch
6. One QSFP+ port
7. Two QSFP28 ports
8. Four 10GBASE-T ports
There are two Ethernet Pass-Through Modules (PTM) providing non-switched Ethernet
connections to Top of Rack switches. Each PTM provides 16 internal 10GbE and/or
25GbE ports. Figure 11 shows the 25GbE Ethernet PTM. The 25GbE PTM provides 16
external SFP+ ports that can operate at 10/25GbE.
The 10GBASE-T Ethernet PTM, shown in Figure 12, provides 16 external RJ45 BASE-T
ports that can operate at 10GbE. An MX7000 chassis supports a mix of four PTMs in
Fabric A and/or B. See Appendix A for more information.
Figure 11. Dell EMC Networking 25GbE PTM
The following 25GbE PTM components are labeled in Figure 11:
1. Express service tag
Dell EMC
PowerEdge MX
pass-through
modules

Chapter 1: Dell EMC PowerEdge MX Platform Overview
14
Dell EMC PowerEdge MX
Networking Architecture Guide
2. Power and indicator LEDs
3. Module insertion/removal latch
4. 16 SFP28 ports
Figure 12. Dell EMC Networking 10GBASE-T PTM
The following 10GBASE-T PTM components are labeled in Figure 12:
1. Express service tag
2. Power and indicator LEDs
3. Module insertion/removal latch
4. 16 10GBASE-T ports
Note: For information about PTM port to compute sled-mapping, see Appendix A.
The Dell EMC Networking MXG610s is a high-performance, 32 Gb Fibre Channel switch
based on Brocade Gen 6 technology. The MXG610s features 8/16/32 GbE capable
interfaces. It is ideal for connectivity to all-flash SAN storage solutions and is designed for
maximum flexibility and value with “pay-as-you-grow” scalability using a Port on Demand,
or PoD, license model. The MXG610s is compatible with Brocade and Cisco
fabric/director class switches.
Besides sixteen internal 32Gb FC ports, the MXG610s provides:
•Eight 32Gb FC SFP ports
•Two 4x 32Gb FC QSFP ports
The MX7000 chassis supports redundant MXG610s modules in Fabric C and must be
purchased in pairs.
Dell EMC
Networking
MXG610s Fibre
Channel switch

Chapter 1: Dell EMC PowerEdge MX Platform Overview
15
Dell EMC PowerEdge MX
Networking Architecture Guide
Figure 13. Dell EMC Networking MXG610s Fibre Channel switch module
The following MXG610s components are labeled in Figure 13:
1. Express service tag
2. Module insertion/removal latch
3. Micro-B USB console port
4. Power and indicator LEDs
5. Eight 32Gb FC SFP ports
6. Two 4x 32Gb FC QSFP ports
The Dell EMC PowerEdge MX5000s SAS module supports x4 SAS internal connections
to all eight front-facing slots in the PowerEdge MX7000 chassis. The MX5000s uses T10
SAS zoning to provide multiple SAS zones/domains for the compute sleds. Storage
management is conducted through the OEM-M console.
The MX5000s provides Fabric C SAS connectivity to each compute and one or more
MX5016s storage sleds. Compute sleds connect to the MX5000s using either SAS Host
Bus Adapters (HBA) or a PowerEdge Raid Controller (PERC) in the mini-mezzanine PCI-
e slot.
The MX5000s switches are deployed as redundant pairs to offer multiple SAS paths to the
individual SAS disk drives. An MX7000 chassis supports redundant MX5000s in Fabric C.
Note: A PowerEdge MX5000s SAS module and a MXG610s is not supported in the same
MX7000 chassis.
Dell EMC
PowerEdge
MX5000s SAS
module

Chapter 1: Dell EMC PowerEdge MX Platform Overview
16
Dell EMC PowerEdge MX
Networking Architecture Guide
Figure 14. Dell EMC PowerEdge MX5000s SAS module
The following MX5000s components are labeled in Figure 14:
1. Express service tag
2. Module insertion/removal latch
3. Power and indicator LEDs
4. Six SAS ports
PowerEdge MX7000 chassis fabric slots
The PowerEdge MX7000 chassis includes two general-purpose I/O fabrics, Fabric A and
B. The vertically aligned compute sleds, in slots one through eight, connect to the
horizontally aligned I/O modules (IOMs), in fabric slots A1, A2, B1, and B2. This
orthogonal connection method results in a mid-plane free design and enables the
adoption of new I/O technologies without the burden of having to upgrade the mid-plane.
The MX740c supports two mezzanine cards, which are installed in slots A1 and B1 and
the MX840c supports four mezzanine cards, which are installed in slots A1, A2, B1, and
B2. Each mezzanine card provides redundant connections to each fabric, A or B, shown
in Figure 15. A mezzanine card connects orthogonally to the pair of IOMs installed in the
corresponding fabric slot. For example, port one of mezzanine card A1 connects to fabric
slot A1, a MX9116n FSE (not shown). The second port of mezzanine card A1 connects to
fabric slot A2, a MX7116n FEM (not shown).
Overview
Mezzanine cards

Chapter 1: Dell EMC PowerEdge MX Platform Overview
17
Dell EMC PowerEdge MX
Networking Architecture Guide
Figure 15. Dell EMC PowerEdge MX740c mezzanine cards
The MX7000 chassis also provides Fabric C, shown in Figure 16, supporting redundant
MXG610s FC switches, or MX5000s SAS modules. This fabric uses a mid-plane
connecting the C1 and C2 modules to each compute or storage sled. The MX740c
supports one mini-mezzanine card, which is installed in slot C1, and the MX840c supports
two mini-mezzanine cards, which are installed in slots C1 and C2.
Figure 16. Dell EMC PowerEdge MX740c mini-mezzanine card
Mini-mezzanine
card

Chapter 2: OpenManage Enterprise - Modular Console
18
Dell EMC PowerEdge MX
Networking Architecture Guide
Chapter 2 OpenManage Enterprise -
Modular Console
This chapter presents the following topics:
Introduction.......................................................................................................19
PowerEdge MX7000 initial deployment ...........................................................19
PowerEdge MX7000 Multi-Chassis Management groups...............................19
PowerEdge MX7000 OME-M component management ..................................22

Chapter 2: OpenManage Enterprise - Modular Console
19
Dell EMC PowerEdge MX
Networking Architecture Guide
Introduction
The PowerEdge MX9002m module hosts the OpenManage Enterprise (OME-M) console.
OME-M is the latest addition to the Dell OpenManage Enterprise suite of tools and
provides a centralized management interface for the PowerEdge MX platform. The OME-
M console features include:
•End-to-end life cycle management for servers, storage, and networking
•Touch LCD for initial setup, and fielding error conditions
•Use of iDRAC9 intelligent automation and security features
•Manage one or multiple chassis from a single web or REST API using multi-chassis
management groups
•OpenManage Mobile for configuration and troubleshooting including wireless server
vKVM
PowerEdge MX7000 initial deployment
Initial PowerMX7000 chassis configuration consists of initial deployment, assigning
network settings for OME-M, and completing the Chassis Deployment Wizard.
There are three methods for initial configuration:
•Using the LCD touchscreen on the front-left of the MX7000 chassis
•Setting the initial OME-M console IP address through the KVM ports on the front-
right side of the MX7000 chassis
•Setting the initial OME-M console IP address through the serial port on the
MX9002m module
On first logging into the OME-M console, a getting started wizard is displayed. The
Chassis Deployment Wizard enables you to configure the following:
•System time
•Alert settings
•iDRAC9 Quick deployment settings
•Network IOM access settings
•Firmware updates
•Proxy settings
•MCM group definition
PowerEdge MX7000 Multi-Chassis Management groups

Chapter 2: OpenManage Enterprise - Modular Console
20
Dell EMC PowerEdge MX
Networking Architecture Guide
Multiple PowerEdge MX9002m modules are grouped to form domains called Multi-
Chassis Management (MCM) groups. A single MCM group can have up to 10 chassis,
where a single chassis serves as the lead and the remaining chassis are members. The
OME-M console supports a daisy chain topology using the redundant 1GbE ports on the
MX9002m module.
The following tasks can be done with an MCM group:
•View the rollup health status of the OME-M chassis group
•Automatically propagate lead chassis settings to member chassis
•Perform any chassis operating on the entire group
•Create a Scalable Fabric Architecture with the MX9116n FSE and MX7116n FEM
Note: See Chapter 3 for more information about Scalable Fabric Architectures.
Creating an MCM group consists of physically cabling all MX9002m modules in the
supported daisy chain topology, and then using the OME-M console to create the group.
Figure 17 shows the supported MX9002m module cabling method. Chassis one, two, and
three are shown while chassis n represents the last chassis in the group.
This manual suits for next models
9
Table of contents
Other Dell EMC Chassis manuals
Popular Chassis manuals by other brands

National Instruments
National Instruments PXIe-1088 user guide

Sonnet
Sonnet Echo Express SEL user guide

IBM
IBM 3453-C2E Customer Install Guide

National Instruments
National Instruments SLSC-12001 GETTING STARTED GUIDE AND SPECIFICATIONS

Advantech
Advantech IPC-623 Series user manual

SilverStone
SilverStone RMS08 Series manual