Intel S1400SP User manual

Intel®I/O Expansion Modules for
Intel®Platforms based on Intel®
Xeon®Processor E5-
4600/2600/2400 Product Families
Hardware Specification
Intel order number: G30021-004
Revision 1.1
November 2012
Enterprise Platforms and Services Division

Revision History Intel®I/O Expansion Modules for Intel®Platforms HWS
Revision 1.1
Intel order number: G30021-004
ii
Revision History
Date
Revision
Number
Modifications
March, 2011
0.5
Initial Release.
August, 2011
0.9
Added the following sections:
Section 2.4 –Intel®I350 Gb Ethernet Controller
Section 3.4 –Intel®X540 10Gb Ethernet Controller
Section 4.4 –Intel®82599 10Gb Ethernet Controller
Section 5.4 - ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI
Section 6 –FDR InfiniBand* ConnectX*-3 I/O Module (AXX1FDRIOIOM)
August, 2011
0.91
Added figures for port identification.
January, 2012
1.0
Deleted Section 5 - QDR InfiniBand* ConnectX*-3 I/O Module, and added new
section for Dual Port FDR InfiniBand* ConnectX*-3 I/O Module.
November, 2012
1.1
Added AXX10GBTWLHW.
Updated Table 1.
Disclaimers
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL®PRODUCTS. NO LICENSE,
EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS
GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL®'S TERMS AND CONDITIONS OF SALE FOR
SUCH PRODUCTS, INTEL®ASSUMES NO LIABILITY WHATSOEVER AND INTEL®DISCLAIMS ANY EXPRESS
OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL®PRODUCTS INCLUDING LIABILITY OR
WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR
INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel®Product could result, directly or
indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL®'S PRODUCTS FOR ANY SUCH
MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL®AND ITS SUBSIDIARIES,
SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH,
HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS'
FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL
INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR
NOT INTEL®OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING
OF THE INTEL®PRODUCT OR ANY OF ITS PARTS.
Intel®may make changes to specifications and product descriptions at any time, without notice. Designers must not
rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel®
reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities
arising from future changes to them. The information here is subject to change without notice. Do not finalize a
design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the
product to deviate from published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your
product order.
Copies of documents which have an order number and are referenced in this document, or other Intel®literature,
may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.

Intel® I/O Expansion Modules for Intel® Platforms HWS Table of Contents
Revision 1.1
Intel order number: G30021-004
iii
Table of Contents
1. Introduction ........................................................................................................................1
2. Quad Port Intel®I350 GbE I/O Module (AXX4P1GBPWLIOM)..........................................3
2.1 Feature Set............................................................................................................3
2.2 Functional Block Diagram ......................................................................................4
2.3 Mechanical Dimensions .........................................................................................5
2.4 Intel®I350 Gb Ethernet Controller..........................................................................6
3. Dual Port Intel®X540 10GbE I/O Module (AXX10GBTWLIOM and AXX10GBTWLHW)...8
3.1 Feature Set............................................................................................................8
3.2 Functional Block Diagram ......................................................................................9
3.3 Mechanical Dimensions .......................................................................................10
3.4 Intel®X540 10Gb Ethernet Controller...................................................................12
4. Dual Port Intel®82599 10GbE I/O Module (AXX10GBNIAIOM).......................................14
4.1 Feature Set..........................................................................................................14
4.2 Functional Block Diagram ....................................................................................15
4.3 Mechanical Dimensions .......................................................................................16
4.4 Intel®82599 10Gb Ethernet Controller.................................................................18
4.5 LAN LED Functionality.........................................................................................19
5. Single Port FDR InfiniBand* ConnectX*-3 I/O Module (AXX1FDRIBIOM) .....................20
5.1 Feature Set..........................................................................................................20
5.2 Functional Block Diagram ....................................................................................21
5.3 Mechanical Dimensions .......................................................................................22
5.4 ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI.......................................23
5.4.1 Overview..............................................................................................................23
5.4.2 Key Features........................................................................................................24
5.5 Network LED Functionality...................................................................................24
6. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module (AXX2FDRIBIOM) ........................25
6.1 Feature Set..........................................................................................................25
6.2 Functional Block Diagram ....................................................................................26
6.3 Mechanical Dimensions .......................................................................................27
6.4 ConnectX*-3 Single/Dual-Port Adapter Silicon with VPI.......................................28
6.4.1 Overview..............................................................................................................28
6.4.2 Key Features........................................................................................................29

Table of Contents Intel®I/O Expansion Modules for Intel®Platforms HWS
Revision 1.1
Intel order number: G30021-004
iv
6.5 Network LED Functionality...................................................................................29

Intel® I/O Expansion Modules for Intel® Platforms HWS List of Figures
Revision 1.1
Intel order number: G30021-004
v
List of Figures
Figure 1. Quad Port Intel®I350 GbE I/O Module.........................................................................3
Figure 2. Quad Port Intel®I350 GbE I/O Module Block Diagram.................................................4
Figure 3. Quad Port Intel®I350 GbE I/O Port Identification.........................................................4
Figure 4. Quad Port Intel®I350 GbE I/O Module Dimensions; Top and Side Views....................5
Figure 5. Quad Port Intel®I350 GbE I/O Module Dimensions; Bottom View ...............................6
Figure 6. Dual Port Intel®X540 10GbE I/O Module.....................................................................8
Figure 7. Dual Port Intel®X540 10GbE I/O Module Block Diagram.............................................9
Figure 8. Dual Port Intel®X540 10GbE I/O Module Port Identification ........................................9
Figure 9. Dual Port Intel®X540 10GbE I/O Module Dimensions; Top and side View ................10
Figure 10. Dual Port Intel®X540 10GbE I/O Module Dimensions; Bottom View........................11
Figure 11. Dual Port Intel®82599 10GbE I/O Module ...............................................................14
Figure 12. Dual Port Intel®82599 10GbE I/O Module Block Diagram .......................................15
Figure 13. Dual Port Intel®82599 10GbE I/O Module Port Identification...................................15
Figure 14. Dual Port Intel®82599 10GbE I/O Module Dimensions; Top and side View.............16
Figure 15. Dual Port Intel®82599 10GbE I/O Module Dimensions; Bottom View......................17
Figure 16. Dual Port Intel®82599 10GbE I/O Module; Rear View.............................................19
Figure 17. Single Port FDR InfiniBand* ConnectX*-3 I/O Module .............................................20
Figure 18. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Block Diagram .....................21
Figure 19. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Port Identification.................21
Figure 20. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Top and side
View ...................................................................................................................................22
Figure 21. Single Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Bottom View....23
Figure 22. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module................................................25
Figure 23. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Block Diagram........................26
Figure 24. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Port Identification....................26
Figure 25. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Top and side View27
Figure 26. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module Dimensions; Bottom View.......28

List of Tables Intel®I/O Expansion Modules for Intel®Platforms HWS
Revision 1.1
Intel order number: G30021-004
vi
List of Tables
Table 1. I/O Module Support Matrix ............................................................................................1
Table 2. 80-pin I/O Module Connector Pin-Out ...........................................................................2
Table 3. Dual Port Intel®82599 10GbE I/O Module LED Functionality......................................19
Table 4. Single Port FDR InfiniBand* ConnectX*-3 I/O Module LED Functionality....................24
Table 5. Dual Port FDR InfiniBand* ConnectX*-3 I/O Module LED Functionality.......................29

Intel® I/O Expansion Modules for Intel® Platforms HWS List of Tables
Revision 1.1
Intel order number: G30021-004
vii
<This page is intentionally left blank.>

Intel® I/O Expansion Modules for Intel® Platforms HWS Introduction
Revision 1.1
Intel order number: G30021-004
1
1. Introduction
The Intel®Server Boards support a variety of Intel®I/O Expansion Module options using x4 or
x8 PCI Express* Mezzanine connectors on the server board. Each mezzanine connector is a 2
x 40-pin, surface mount, and 0.8 mm pitch header.
Intel®I/O Expansion Modules for Intel®Platforms based on Intel®Xeon®processor E5-
4600/2600/2400 product families includes:
Quad Port Intel®I350 GbE I/O Module
Dual Port Intel®X540 10GbE I/O Module
Dual Port Intel®82599 10GbE I/O Module
Single Port FDR InfiniBand* ConnectX*-3 I/O Module
Dual Ports FDR InfiniBand* ConnectX*-3 I/O Module
Intel®I/O Expansion Modules for Intel®Platforms based on Intel®Xeon®processor E5-
4600/2600/2400 product families are designed to fit Intel®Server Boards based on Intel®Xeon®
processor E5-4600/2600/2400 product families. The table below shows the support matrix for
the Intel®I/O Expansion Modules.
Table 1. I/O Module Support Matrix
Intel®I/O Expansion Modules
S1400SP
S1600JP
S2400BB
S2600GZ/GL
S2600IP
S2600JF
S2600WP
S4600LH2/LT2
Quad Port Intel®I350 GbE I/O Module -
AXX4P1GBPWLIOM








Dual Port Intel®X540 10GbE I/O Module -
AXX10GBTWLIOM

×



×
×

Dual Port Intel®X540 10GbE I/O Module -
AXX10GBTWLHW
×

×
×
×


×
Dual Port Intel®82599 10GbE I/O Module -
AXX10GBNIAIOM








Single Port FDR InfiniBand* ConnectX*-3
I/O Module - AXX1FDRIBIOM








Dual Port FDR InfiniBand* ConnectX*-3 I/O
Module –AXX2FDRIBIOM








Note:
1. Intel®I/O Expansion Modules for Intel®Platforms based on Intel®Xeon®processor E5-
4600/2600/2400 product families are not supported on previous generation Intel®Server
Boards.
2. For the latest support information on each Intel®Server Platform, please go to
http://serverconfigurator.intel.com/sct_app.aspx.

Introduction Intel® I/O Expansion Modules for Intel® Platforms HWS
Revision 1.1
Intel order number: G30021-004
2
The following table details the pin-out of the I/O module connector:
Table 2. 80-pin I/O Module Connector Pin-Out
Pin
Signal
Signal
Pin
1
3.3V
12V
2
3
3.3V
12V
4
5
3.3V
12V
6
7
3.3V
12V
8
9
RSVD
FRU/TEMP ADDR [A0]
10
11
GND
5VSB
12
13
RSVD+
FM_IO_MODULE_EN
14
15
RSVD-
3.3VSTBY
16
17
GND
LED_GLOBAL ACT#
18
19
RSVD
FM_IOM_PRESENT_N
20
21
RSVD
WAKE#
22
23
GND
PERST#
24
25
SMB CLK
GND
26
27
SMB DAT
rIOM REFCLK+ [0]
28
29
GND
rIOM REFCLK- [0]
30
31
PCIe Gen3 Tn [7]]
GND
32
33
PCIe Gen3 Tp [7]
PCIe Gen3 Rn [7]
34
35
GND
PCIe Gen3 Rp [7]
36
37
PCIe Gen3 Tn [6]
GND
38
39
PCIe Gen3 Tp [6]
PCIe Gen3 Rn [6]
40
41
GND
PCIe Gen3 Rp [6]
42
43
PCIe Gen3 Tn [5]
GND
44
45
PCIe Gen3 Tp [5]
PCIe Gen3 Rn [5]
46
47
GND
PCIe Gen3 Rp [5]
48
49
PCIe Gen3 Tn [4]
GND
50
51
PCIe Gen3 Tp [4]
PCIe Gen3 Rn [4]
52
53
GND
PCIe Gen3 Rp [4]
54
55
PCIe Gen3 Tn [3]
GND
56
57
PCIe Gen3 Tp [3]
PCIe Gen3 Rn [3]
58
59
GND
PCIe Gen3 Rp [3]
60
61
PCIe Gen3 Tn [2]
GND
62
63
PCIe Gen3 Tp [2]
PCIe Gen3 Rn [2]
64
65
GND
PCIe Gen3 Rp [2]
66
67
PCIe Gen3 Tn [1]
GND
68
69
PCIe Gen3 Tp [1]
PCIe Gen3 Rn [1]
70
71
GND
PCIe Gen3 Rp [1]
72
73
PCIe Gen3 Tn [0]
GND
74
75
PCIe Gen3 Tp [0]
PCIe Gen3 Rn [0]
76
77
GND
PCIe Gen3 Rp [0]
78
79
RSVD
GND
80
Note: There are several pins reserved for future use.

Intel® I/O Expansion Modules for Intel® Platforms HWS Quad Port Intel®I350 GbE I/O Module
Revision 1.1
Intel order number: G30021-004
3
2. Quad Port Intel®I350 GbE I/O Module
(AXX4P1GBPWLIOM)
The Quad Port Intel®I350 GbE I/O Module provides four additional 10/100/1000Mbit external
connections. This section provides a high-level description of the implementation of this I/O
module.
Figure 1. Quad Port Intel®I350 GbE I/O Module
2.1 Feature Set
The Quad Port Intel®I350 GbE I/O Module supports the following feature set:
Intel®I350 Controller 17x17mm chip
Quad 1Gb RJ45 Ethernet external connections
Speed and Link/Act LEDs for each RJ45 connection
Onboard 1.0V VR off of 12V and 5V_SB Rails
One WOL port operating in 1 GB mode is supported (Port 2)
1.8V_SB linear VR off 3.3V_SB
256kb LAN EEPROM and 1Mb SPI Flash (SPI Flash unstuffed as baseboard BIOS will
provide OpROM for PXE and iSCSI support)
Three loose screws to mount IOM directly to sheet metal chassis. It's recommended to
use same screws as used to install the baseboard
256Byte FRU EEPROM and TMP75 temp sensor

Quad Port Intel®I350 GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS
Revision 1.1
Intel order number: G30021-004
4
2.2 Functional Block Diagram
Figure 2. Quad Port Intel®I350 GbE I/O Module Block Diagram
Figure 3. Quad Port Intel®I350 GbE I/O Port Identification
Intel®I350
2 x 40-pin
connector
NIC Port 1
PCIe Gen2 x4
SPI Flash
（empty）
EEPROM
NIC Port 2
NIC Port 3
NIC Port 4

Intel® I/O Expansion Modules for Intel® Platforms HWS Quad Port Intel®I350 GbE I/O Module
Revision 1.1
Intel order number: G30021-004
5
2.3 Mechanical Dimensions
Figure 4. Quad Port Intel®I350 GbE I/O Module Dimensions; Top and Side Views

Quad Port Intel®I350 GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS
Revision 1.1
Intel order number: G30021-004
6
Figure 5. Quad Port Intel®I350 GbE I/O Module Dimensions; Bottom View
2.4 Intel®I350 Gb Ethernet Controller
The Intel®Ethernet Controller I350 is a single, compact, low power component that supports
quad port and dual port gigabit Ethernet designs. The device offers four fully-integrated gigabit
Ethernet media access control (MAC), physical layer (PHY) ports and four SGMII/SerDes ports
that can be connected to an external PHY. The I350 supports PCI Express* (PCIe v2.1
(2.5GT/s and 5GT/s)).
The device enables two-port or four-port 1000BASE-T implementations using integrated PHYs.
It can be used for server system configurations such as rack mounted or pedestal servers, in an
add-on NIC or LAN on Motherboard (LOM) design. Another possible system configuration is for
blade servers. Here, the I350 can support up to four SerDes ports as LOM or mezzanine card.
It can also be used in embedded applications such as switch add-on cards and network
appliances.

Intel® I/O Expansion Modules for Intel® Platforms HWS Quad Port Intel®I350 GbE I/O Module
Revision 1.1
Intel order number: G30021-004
7
1. External Interfaces provided
PCIe v2.1 (2.5GT/s and 5GT/s) x4/x2/x1
MDI (Copper) standard IEEE 802.3 Ethernet interface for 1000BASE-T, 100BASE-TX,
and 10BASE-T applications (802.3, 802.3u, and 802.3ab)
Serializer-Deserializer (SERDES) to support 1000BASE-SX/LX (optical fiber -
IEEE802.3)
Serializer-Deserializer (SERDES) to support 1000BASE-KX(802.3ap) and 1000BASE-
BX (PICMIG 3.1) for Gigabit backplane applications
SGMII (Serial-GMII Specification) interface for SFP (SFP MSA INF-8074i)/external PHY
connections
NC-SI (DMTF NC-SI) or SMBus for Manageability connection to BMC
IEEE 1149.6 JTAG
2. Performance Enhancements
PCIe v2.1 TLP Process Hints (TPH)
UDP, TCP and IP Checksum offload
UDP and TCP Transmit Segmentation Offload (TSO)
SCTP receive and transmit checksum offload
3. Virtualization ready
Next Generation VMDq support (8 VMs)
Support of up to 8 VMs per port (1 queue allocated to each VM)
PCI-SIG I/O SR-IOV support (Direct assignment)
Queues per port: 8 TX and 8 RX queues
4. Power saving features
Advanced Configuration and Power Interface (ACPI) power management states and
wake-up capability
Advanced Power Management (APM) wake-up functionality
Low power link-disconnect state
PCIe v2.1 LTR
DMA Coalescing for improved system power management
EEE (IEEE802.3az) for reduced power consumption during low link utilization periods
5. IEEE802.1AS - Timing and Synchronization
IEEE 1588 Precision Time Protocol support
Per-packet timestamp
6. Total Cost of Ownership (TCO)
IPMI BMC pass-thru; multi-drop NC-SI
Internal BMC to OS and OS to BMC traffic support
7. Additional product details
17x17 (256 Balls) or 25x25 (576 Balls) PBGA package
Estimated power: 2.8W (max) in dual port mode and 4.2W (max) in quad port mode
Memories have Parity or ECC protection
See Intel®Ethernet Controller I350 Datasheet for more details.

Dual Port Intel®X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS
Revision 1.1
Intel order number: G30021-004
8
3. Dual Port Intel®X540 10GbE I/O Module
(AXX10GBTWLIOM and AXX10GBTWLHW)
Figure 6. Dual Port Intel®X540 10GbE I/O Module
3.1 Feature Set
Following is the feature set:
Intel®X540 Controller 25x25mm chip
Two 10Gb RJ45 Ethernet external connections
Support PXE and iSCSI
WOL support of one 1GbE port
Speed and Link/Act LEDs for each 10Gb RJ45 connection
16M bit SPI Flash
256Byte FRU EEPROM and TMP75 temp sensor
Stainless steel EMI shield
AXX10GBTWLHW is the same with AXX10GBTWLIOM except for the heat sink to fit
Intel®Server Board S2600JF, S2600WP, and Intel®Server Chassis H2000 Family.

Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel®X540 10GbE I/O Module
Revision 1.1
Intel order number: G30021-004
9
3.2 Functional Block Diagram
Figure 7. Dual Port Intel®X540 10GbE I/O Module Block Diagram
Figure 8. Dual Port Intel®X540 10GbE I/O Module Port Identification
Intel®X540
2 x 40-pin
connector
NIC Port 1
PCIe Gen2 x8
SPI Flash
NIC Port 2

Dual Port Intel®X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS
Revision 1.1
Intel order number: G30021-004
10
3.3 Mechanical Dimensions
Figure 9. Dual Port Intel®X540 10GbE I/O Module Dimensions; Top and side View

Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel®X540 10GbE I/O Module
Revision 1.1
Intel order number: G30021-004
11
Figure 10. Dual Port Intel®X540 10GbE I/O Module Dimensions; Bottom View

Dual Port Intel®X540 10GbE I/O Module Intel® I/O Expansion Modules for Intel® Platforms HWS
Revision 1.1
Intel order number: G30021-004
12
3.4 Intel®X540 10Gb Ethernet Controller
1. General
Serial flash interface
Device disable capability
Package size - 25 mm x 25 mm
2. Networking
10 GbE/1 GbE/100 Mb/s copper PHYs integrated on-chip
Support for jumbo frames of up to 15.5 KB
Flow control support: send/receive pause frames and receive FIFO thresholds
Statistics for management and RMON
802.1q VLAN support
TCP segmentation offload: up to 256 KB
IPv6 support for IP/TCP and IP/UDP receive checksum offload
Fragmented UDP checksum offload for packet reassembly
Message Signaled Interrupts (MSI)
Message Signaled Interrupts (MSI-X)
Interrupt throttling control to limit maximum interrupt rate and improve CPU usage
Receive packet split header
Multiple Receive Queues (RSS) 8x8 and 16x4
32 transmit queues
Receive header replication
Dynamic interrupt moderation
DCA support
TCP timer interrupts
No snoop
Relaxed ordering
Support for 16 Virtual Machines Device Queues (VMDq) per port
3. Host Interface
PCIe base specification 2.1 (2.5GT/s or 5GT/s)
Bus width —x1, x2, x4, x8
64-bit address support for systems using more than 4 GB of physical memory
4. MAC Functions
Descriptor ring management hardware for transmit and receive
ACPI register set and power down functionality supportingD0 and D3 states
A mechanism for delaying/reducing transmit interrupts
Software-controlled global reset bit (resets everything except the configuration registers)
Four Software-Definable Pins (SDP) per port
Wake up
IPv6 wake-up filters
Configurable flexible filter (through NVM)
LAN function disable capability
Programmable memory transmit buffers (160 KB/port)
Default configuration by NVM for all LEDs for pre-driver functionality
5. Manageability
Eight VLAN L2 filters
16 Flex L3 port filters

Intel® I/O Expansion Modules for Intel® Platforms HWS Dual Port Intel®X540 10GbE I/O Module
Revision 1.1
Intel order number: G30021-004
13
Four Flexible TCO filters
Four L3 address filters (IPv4)
Advanced pass through-compatible management packet transmit/receive support
SMBus interface to an external Manageability Controller (MC)
NC-SI interface to an external MC
Four L3 address filters (IPv6)
Four L2 address filters
See Intel®Ethernet Controller 10G X540 Datasheet for more details.