Mellanox Technologies Innova IPsec 4 Lx User manual

Mellanox Technologies
www.mellanox.com
Mellanox Innova™ IPsec 4 Lx
Ethernet Adapter Card User Manual
Rev 1.3

Doc #: MLNX-15-50911 2Mellanox Technologies
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
© Copyright 2017. Mellanox Technologies Ltd . All Rights Reserved .
Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,
CoolBox® , CORE-Direct® , EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,
GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack® , Mellanox
CloudXMellanox® , Mellanox Federal Systems® , Mellanox HostDirect® , Mellanox Multi-Host® , Mellanox Open
Ethernet®, Mellanox OpenCloud® , Mellanox OpenCloud Logo® , Mellanox PeerDirect® , Mellanox ScalableHPC® ,
Mellanox StorageX® , Mellanox TuneX® , Mellanox Connect Accelerate Outperform logo , Mellanox Virtual Modular
Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo , PhyX®, PlatformX®,
PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of Open Ethernet
logo, UFM®, Unbreakable Link® , Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are registered
trademarks of Mellanox Technologies , Ltd.
All other trademarks are property of their respective owners .
For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT (S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT
USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY
QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES
CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR
ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT , STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE
PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.

Rev: 1.3 3Mellanox Technologies
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1 Product Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Features and Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3 Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4 Operating Systems/Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.5 Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6 Related Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 2 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1 Ethernet QSFP Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 LED Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Chapter 3 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.2 Operating Systems/Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Pre-installation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Bracket Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4.1 Removing the Existing Bracket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.4.2 Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.5 Card Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.6 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.6.1 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.7 Identify the Card in Your System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Chapter 4 Innova IPsec Offload Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1 Security Engines and IPsec Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.1 Offloaded IPsec Protocols and Internet Protocols . . . . . . . . . . . . . . . . . . . 22
4.1.1.1 IPsec Protocols. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1.1.2 Internet Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2 IPsec Offload Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.1 mlx_ipsec Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.2 mlx_accel_core Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Rev: 1.34 Mellanox Technologies
4.2.3 mlx_accel_tools Module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.4 Key Generation and Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 IPsec Offload for DPDK Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Chapter 5 IPsec Offload Software Installation and Operation. . . . . . . . . . . . 25
5.1 Installation of Kernel with IPsec Offload Module . . . . . . . . . . . . . . . . . . . 25
5.1.1 Obtaining the Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1.2 Installing the Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1.3 Installing the Customized iproute2 Utility . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.2 Installation via MLNX_OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.3 Operating the IPsec Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.3.1 Loading/Unloading the Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.3.2 Setting up an Offloaded IPsec Connection . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.3.3 Destroying IPsec Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.3.4 IPsec Offload Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Chapter 6 mlx_fpga Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.1 Tool Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.2 mlx_fpga Synopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.3 Examples of mlx_fpga Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3.1 Adding FPGA mst Device Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3.1.1 Burning the FPGA’s Flash Device Using the mlx_fpga Burning Tool . . . . 32
6.3.1.2 Loading Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3.1.3 Debugging Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.3.1.4 Update FPGA Image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Chapter 7 Updating Innova IPsec Adapter Card Firmware. . . . . . . . . . . . . . . 34
Chapter 8 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
8.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
8.2 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Chapter 9 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
9.1 MCX4732A-BCIT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
9.2 MNV101511A-BCIT Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
9.3 Innova IPsec 4 Lx EN LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
9.3.1 Network LEDs Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
9.3.2 FPGA Debug LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
9.3.3 FPGA Load-Flow Debug LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
9.4 Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . 43
9.5 Bracket Mechanical Drawing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Appendix A Fast Installation and Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
A.1 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
A.2 Content of Innova IPsec Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Rev: 1.3 5Mellanox Technologies
A.3 Software, Firmware and Tools Update . . . . . . . . . . . . . . . . . . . . . . . . . 47
Appendix B Interface Connectors Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
B.1 QSFP Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
B.2 PCI Express x8 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
B.3 I2C-compatible Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Appendix C Finding the MAC and Serial Number on the Adapter Card . . . . 53
Appendix D Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Appendix E Avertissements de sécurité d’installation (Warnings in French) 56
Appendix F Sicherheitshinweise (Warnings in German) . . . . . . . . . . . . . . . . 58
Appendix G Advertencias de seguridad para la instalación (Warnings in Spanish) 60

Rev: 1.36 Mellanox Technologies
List of Figures
Figure 1: Innova IPsec 4 Lx EN Adapter Card Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Figure 2: IPsec Solution Layers and Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Figure 3: MCX4732A-BCIT LEDs Placement (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Figure 4: MNV101511A-BCIT LEDs Placement (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Figure 5: Mechanical Drawing of the MCX4732A-BCIT Innova IPsec Adapter Card . . . . . . . . . . . . 43
Figure 6: Mechanical Drawing of the MNV101511A-BCIT Innova IPsec Adapter Card . . . . . . . . . . 44
Figure 7: Single-Port Tall Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Figure 8: Single-Port Short Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
Figure 9: Connector and Cage Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Figure 10: PCIe x8 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Figure 11: Compatible Connector Plug and Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Figure 12: MCX4732A-BCIT Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Figure 13: MNV101511A-BCIT Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Rev: 1.3 7Mellanox Technologies
List of Tables
Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Table 2: Single-port Innova IPsec Adapter Cards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Table 3: Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Table 4: Documents List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Table 5: ethtool IPsec Offload Counters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Table 6: MCX4732A-BCIT Specifications Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Table 7: MNV101511A-BCIT Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Table 8: Physical and Logical Link Indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Table 9: FPGA Debug LEDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Table 10: FPGA Load-Flow Debug LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Rev: 1.38 Mellanox Technologies
Revision History
This document was printed on January 31, 2017.
Table 1 - Revision History Table
Date Rev Comments/Changes
January 2017 1.3 • Updated Section 5.1.1, “Obtaining the Kernel Modules,” on
page 25
• Updated Section 5.3.2, “Setting up an Offloaded IPsec Con-
nection,” on page 27
• Updated Section 5.3.3, “Destroying IPsec Tunnels,” on
page 28
• Updated Section 6.1, “Tool Requirements,” on page 31
• Updated Section 6.2, “mlx_fpga Synopsis,” on page 31
• Added Section 6.3.1.1, “Burning the FPGA’s Flash Device
Using the mlx_fpga Burning Tool,” on page 32
• Updated Section 6.3.1.2, “Loading Tool,” on page 32
• Updated Chapter 7,“Updating Innova IPsec Adapter Card
Firmware” on page 34
• Added Figure 8, “Single-Port Short Bracket,” on page 46
• Added Appendix A, “Fast Installation and Update,” on
page 47
September 2016 1.2 • Added MNV101511A-BCIT across document:
- Section 1.1, “Product Overview,” on page 11
- Section 9.2, “MNV101511A-BCIT Specifications,” on
page 39
- Figure 4, “MNV101511A-BCIT LEDs Placement (Exam-
ple),” on page 40
- Figure 6, “Mechanical Drawing of the MNV101511A-BCIT
Innova IPsec Adapter Card,” on page 44
• Added Chapter 5,“IPsec Offload Software Installation and
Operation” on page 25
• Updated Section 5.1.2, “Installing the Kernel Modules,” on
page 25
• Updated Section 5.1.3, “Installing the Customized iproute2
Utility,” on page 26
• Updated Section 5.3.2, “Setting up an Offloaded IPsec Con-
nection,” on page 27
• Added Section 5.3.3, “Destroying IPsec Tunnels,” on page 28
• Removed Innova IPsec 4 Lx EN Card Drivers
• Updated Chapter 6,“mlx_fpga Tool” on page 31
• Updated Section 8.1, “General,” on page 36
• Updated Section 9.1, “MCX4732A-BCIT Specifications,” on
page 38
• Updated Appendix C, “Finding the MAC and Serial Number
on the Adapter Card,” on page 53

Rev: 1.3 9Mellanox Technologies
July 2016 1.1 • Changed mlx_accel_ipsec to mlx_ipsec.
• Added Section 4.2.3, “mlx_accel_tools Module,” on page 23
• Updated Section 5.1.1, “Obtaining the Kernel Modules,” on
page 25
• Updated Section 5.1.2, “Installing the Kernel Modules,” on
page 25
• Updated Section 5.1.3, “Installing the Customized iproute2
Utility,” on page 26
• Added Section 5.3.4, “IPsec Offload Statistics,” on page 29
• Added “Update FPGA Image” on page 33
• Updated Chapter 5.2,“Installation via MLNX_OFED” on
page 26
April 2016 1.0 First Release
Table 1 - Revision History Table
Date Rev Comments/Changes

Rev: 1.310 Mellanox Technologies
1 Introduction
This is the User Guide for Mellanox Technologies Innova IPsec adapter card based on the Con-
nectX ®-4 Lx Ethernet (EN) integrated circuit device with an on-board FPGA device.
The Mellanox Innova IPsec 4 Lx EN adapter card provides security acceleration for IPsec-
enabled networks while taking advantage of the ConnectX-4 Lx EN Network Controller’s best-
in-class performance, unmatched scalability, and efficiency.
The constantly growing demand for security and privacy in modern data centers, private and pub-
lic clouds, Web 2.0 infrastructure, and telecommunication systems, requires the use of security
protocols. IPsec is a protocol suite for secure Internet Protocol (IP) communications by authenti-
cating and encrypting each IP packet of a communication session. However, the high computing
power required by the IPsec algorithms consumes expensive CPU cycles and limits network con-
nection performance.
The Mellanox Innova IPsec 4 Lx EN adapter offloads the processing of the IPsec algorithms,
frees up the CPU, and eases network bottlenecks.
The adapter integrates advanced network capabilities and encryption offloading in one card, uti-
lizing only a single PCIe slot for both networking and crypto functions.
The Mellanox Innova IPsec 4 Lx EN adapter also brings Innova IPsec 4 Lx’s industry leading
technologies: hardware support for RDMA over Converged Ethernet, Ethernet stateless offload
engines, overlay networks, GPUDirect® technologies, and more.
This chapter covers the following topics:
• Section 1.1, “Product Overview,” on page 11
• Section 1.2, “Features and Benefits,” on page 12
• Section 1.3, “Block Diagram,” on page 14
•Section 1.4, “Operating Systems/Distributions,” on page 14
• Section 1.5, “Connectivity,” on page 15
• Section 1.6, “Related Documents,” on page 15

Introduction
Rev: 1.3 11Mellanox Technologies
1.1 Product Overview
The following section provides the ordering part number, port speed, number of ports, and PCI
Express speed.
Table 2 - Single-port Innova IPsec Adapter Cards
Ordering Part Number (OPN)
MCX4732A-BCIT - with Xilinx Kintex® UltraScale™
XCKU060
MNV101511A-BCIT - HHHL card with Xilinx Kintex®
UltraScale™ XCKU060
Data Transmission Rate Ethernet: 10/40Gb/s
Network Connector Types Single-port QSFP
PCI Express (PCIe) SerDes Speed PCIe 3.0 x8 8GT/s
RoHS R6
Adapter IC Part Number MT27711A0-FDCF-BE
Device ID (decimal) 610

Rev: 1.312 Mellanox Technologies
1.2 Features and Benefits
Table 3 - Featuresa
PCI Express (PCIe) Uses PCIe Gen 3.0 (1.1 and 2.0 compatible) through an x8 edge connector up to
8GT/s
Up to 40 Gigabit Ethernet
Mellanox adapters comply with the following IEEE 802.3* standards:
– IEEE Std 802.3ba 40 Gigabit Ethernet
– IEEE Std 802.3ae 10 Gigabit Ethernet
– IEEE Std 802.3ad, Link Aggregation
– IEEE Std 802.1Q, 1P VLAN tags and priority
– IEEE Std 802.1Qau Congestion Notification
– IEEE Std 802.1Qbg
_ IEEE P802.1Qaz D0.2 ETS
_ IEEE P802.1Qbb D1.0 Priority-based Flow Control
Memory
PCI Express - stores and accesses Ethernet fabric connection information and
packet data
SPI - includes two SPI Flash devices:
• one 16MB SPI Flash device (W25Q128FVSIG by WINBOND-NUVOTON)
for ConnectX-4 Lx device
• one 256Mb SPI flash (N25Q256A13E1240E by MICRON TECHNOLOGY)
for the FPGA device
EEPROM - accessible through the I2C-compatible interface. The EEPROM
capacity is 128Kb.
2GByte DDR4 – PC-1600 SODIMM
IPsec Offload
The Innova IPsec adapter provides offloading of compute intensive encryption/
decryption and authentication algorithms, which are used by the IPsec protocol.
Support for Linux and Windows IPsec software interfaces ensures native integra-
tion with existing IPsec applications, with no required changes to the user’s soft-
ware. IPsec offloading is handled by the combination of the ConnectX-4 Lx
network controller and an on-board FPGA, providing high performance and flex-
ibility for future enhancements and customizations. The FPGA is connected to the
ConnectX-4 Lx through a ‘bump-in-the-wire’ topology, hence encryption and
decryption are performed inline with the network flow. This results in lower
latency and additional savings of CPU resources compared to other IPsec protocol
solutions, be it through software or alternative accelerators.
Overlay Networks
In order to better scale their networks, data center operators often create overlay
networks that carry traffic from individual virtual machines over logical tunnels
in encapsulated formats such as NVGRE and VXLAN. While this solves network
scalability issues, it hides the TCP packet from the hardware offloading engines,
placing higher loads on the host CPU. Innova IPsec adapter effectively addresses
this by providing advanced NVGRE, VXLAN and GENEVE hardware offloading
engines that encapsulate and de-capsulate the overlay protocol headers, enabling
the traditional offloads to be performed on the encapsulated traffic for these and
other tunneling protocols (GENEVE, MPLS, QinQ, and so on). With Innova
IPsec adapter, data center operators can achieve native performance in the new
network architecture.

Introduction
Rev: 1.3 13Mellanox Technologies
RDMA and RDMA over
Converged Ethernet (RoCE)
Innova IPsec adapter supports RoCE specifications delivering low-latency and
high- performance over Ethernet networks. Leveraging data center bridging
(DCB) capabilities as well as Innova IPsec adapter advanced congestion control
hardware mechanisms, RoCE provides efficient low-latency RDMA services over
Layer 2 and Layer 3 networks.
Mellanox PeerDirect™
PeerDirect™ communication provides high efficiency RDMA access by eliminat-
ing unnecessary internal data copies between components on the PCIe bus (for
example, from GPU to CPU), and therefore significantly reduces application run
time. Innova IPsec adapter advanced acceleration technology enables higher clus-
ter efficiency and scalability to tens of thousands of nodes.
CPU offload Adapter functionality enabling reduced CPU overhead allowing more available
CPU for computation tasks.
Quality of Service (QoS) Support for port-based Quality of Service enabling various application require-
ments for latency and SLA
Hardware-based I/O
Virtualization
Innova IPsec adapter SR-IOV technology provides dedicated adapter resources
and guaranteed isolation and protection for virtual machines (VMs) within the
server. I/O virtualization with Innova IPsec adapter gives data center administra-
tors better server utilization while reducing cost, power, and cable complexity,
allowing more Virtual Machines and more tenants on the same hardware.
Storage Acceleration
A consolidated compute and storage network achieves significant cost-perfor-
mance advantages over multi-fabric networks. Standard block and
file access protocols can leverage RDMA for high-performance storage access.
Distributed RAID
Innova IPsec adapter delivers advanced Erasure Coding offloading capability,
enabling distributed Redundant Array of Inexpensive Disks (RAID), a data stor-
age technology that combines multiple disk drive components into a logical unit
for the purposes of data redundancy and performance improvement. Innova IPsec
adapter’s Reed-Solomon capability introduces redundant block calculations,
which, together with RDMA, achieves high performance and reliable storage
access.
a. This section describes hardware features and capabilities. Please refer to the driver release notes for feature avail-
ability. See Section 1.6, “Related Documents,” on page 15.
Table 3 - Featuresa

Rev: 1.314 Mellanox Technologies
1.3 Block Diagram
Figure 1: Innova IPsec 4 Lx EN Adapter Card Block Diagram
1.4 Operating Systems/Distributions1
• RHEL/CentOS
• Ubuntu
•Fedora
• OpenFabrics Enterprise Distribution (OFED)
1. Windows will be supported in a future revision.
ConnectX
DRAM
x8 PCIe G en3
FPGA
Config. FLASH
x8 PCIe Gen3
JTAG-IF
QSFP
I2C
JTAG/
GPIO
2GB
DDR 4-
1600[MT/Sec]

Introduction
Rev: 1.3 15Mellanox Technologies
1.5 Connectivity
• Interoperable with 10Gb and 40Gb Ethernet switches
• Passive copper cable with ESD protection
• Powered connectors for optical and active cable support
1.6 Related Documents
Table 4 - Documents List
Document’s Name Location
Mellanox Firmware Tools (MFT) User
Manual
Document no. 2204UG
User Manual describing the set of MFT firmware management
tools for a single node.
See http://www.mellanox.com/page/management_tools
Mellanox OFED for Linux
User Manual
Document no. 2877
User Manual describing OFED features, performance, tools con-
tent and configuration. See http://www.mellanox.com => Products
=> Software => InfiniBand/VPI Drivers => Mellanox OpenFab-
rics Enterprise Distribution for Linux (MLNX_OFED)
Performance Tuning Guidelines
for Mellanox Network Adapters
Document no. 3368
User Manual describes important tuning parameters and settings
that can improve performance for Mellanox drivers.
See http://www.mellanox.com/related-docs/prod_software/Perfor-
mance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf
Mellanox EN for Linux Driver Release
Notes
Release notes for Mellanox Technologies' MLNX_EN for Linux
driver kit for Mellanox adapter cards:
http://www.mellanox.com => Products => Software => Infini-
Band/VPI Drivers => Mellanox OpenFabrics Enterprise Distribu-
tion for Linux (MLNX_OFED) => Release Notes
IEEE Std 802.3 Specification This is the IEEE Ethernet specification
http://standards.ieee.org/getieee802
PCI Express 3.0 Specifications Industry Standard PCI Express 3.0 Base and Card Electromechan-
ical Specifications.
https://pcisig.com/specifications
IETF IPsec specifications https://tools.ietf.org/html/rfc4301

Rev: 1.316 Mellanox Technologies
2 Interfaces
Each adapter card includes the following interfaces:
• “Ethernet QSFP Interface”
• “PCI Express Interface”
• “LED Interface”
The adapter cards include special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
2.1 Ethernet QSFP Interface
The network port of the Innova IPsec adapter is compliant with the IEEE 802.3 Ethernet stan-
dards listed in Table 3, “Features,” on page 12. For connecting to an SFP+ interface, you can use
a Mellanox QSA (QSFP to SFP+) adapter module.
2.2 PCI Express Interface
The Innova IPsec adapter card supports PCI Express 3.0 (1.1 and 2.0 compatible) through an x8
edge connector. The device can be either a master initiating the PCI Express bus operations or a
slave responding to PCI bus operations. The following lists PCIe interface features of the Innova
IPsec adapter card:
• PCIe Gen 3.0 compliant, 1.1 and 2.0 compatible
• 2.5, 5.0, or 8.0GT/s link rate x8
• Auto-negotiates to x8, x4, x2, or x1
• Support for MSI/MSI-X mechanisms
2.3 LED Interface
For Innova IPsec adapter card LED specifications, please refer to Section 9.3, “Innova IPsec 4
Lx EN LEDs,” on page 40.

Hardware Installation
Rev: 1.3 17Mellanox Technologies
3 Hardware Installation
3.1 System Requirements
3.1.1 Hardware
A system with a PCI Express x8 slot is required for installing the card.
3.1.2 Operating Systems/Distributions
Please refer to Section 1.4, “Operating Systems/Distributions,” on page 14.
3.2 Safety Precautions
1. Remove any metallic objects from your hands and wrists.
2. Make sure to use only insulated tools.
3. Verify that the system is powered off and is unplugged.
4. It is strongly recommended to use an ESD strap or other antistatic devices.
3.3 Pre-installation Checklist
1. Verify that your system meets the hardware and software requirements stated above.
2. Shut down your system if active.
3. After shutting down the system, turn off power and unplug the cord.
4. Remove the card from its package. Please note that the card must be placed on an antistatic
surface.
5. Check the card for visible signs of damage. Do not attempt to install the card if damaged.
3.4 Bracket Installation Instructions
The card is usually shipped with a tall bracket installed. If this form factor is suitable for your
requirements, you can skip the remainder of this section and move to Section 3.5, “Card Installa-
tion Instructions,” on page 18.
If you need to replace it with the short bracket that is included in the shipping box, please follow
the instructions in this section.
The adapter is being installed in a system that operates with voltages that can be lethal.
Before opening the case of the system, observe the following precautions to avoid
injury and prevent damage to system components.

Rev: 1.318 Mellanox Technologies
To replace the bracket you will need the following parts:
• The new bracket of the proper height
• The 2 screws saved from the removal of the bracket
• The 2 fiber washers saved from the removal of the bracket
3.4.1 Removing the Existing Bracket
1. Remove the two screws holding the bracket in place. The bracket comes loose from the card.
2. Save the two screws and the two fiber washers.
3.4.2 Installing the New Bracket
1. Place the bracket onto the card until the screw holes line up.
2. Screw on the bracket using the screws and washers saved from the bracket removal procedure
above.
3. Make sure that the LEDs are aligned onto the bracket holes.
4. Use a torque driver to apply up to 2.9 lbs-in torque on the screws.
3.5 Card Installation Instructions
1. Open the system case.
2. Place the adapter in a standard PCI Express slot
3. Applying even pressure at both corners of the card, insert the adapter card into the slot until it
is firmly seated. When the adapter is properly seated, the adapter port connectors are aligned
with the slot opening, and the adapter faceplate is visible against the system chassis.
Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket
more than three times.
Be careful not to put stress on the LED.
Do not force the bracket onto the card. You may have to gently push the LEDs using a
small screwdriver to align the LEDs with the holes in the bracket.

Hardware Installation
Rev: 1.3 19Mellanox Technologies
3.6 Cables and Modules
To obtain the list of supported cables for your adapter, please refer to http://www.mellanox.com/
products/interconnect/cables-configurator.php.
3.6.1 Cable Installation
1. All cables can be inserted or removed with the unit powered on.
2. To insert a cable, press the connector into the port receptacle until the connector is firmly
seated.
a. Support the weight of the cable before connecting the cable to the adapter card. Do this by using a cable
holder or tying the cable to the rack.
b. Determine the correct orientation of the connector to the card before inserting the connector. Do not try and
insert the connector upside down. This may damage the adapter card.
c. Insert the connector into the adapter card. Be careful to insert the connector straight into the cage. Do not
apply any torque, up or down, to the connector cage in the adapter card.
d. Make sure that the connector locks in place.
3. After inserting a cable into a port, the Amber LED indicator will light when the physical con-
nection is established (that is, when the unit is powered on and a cable is plugged into the port
with the other end of the connector plugged into a functioning port). See Section 9.3, “Innova
IPsec 4 Lx EN LEDs,” on page 40.
4. After plugging in a cable, lock the connector using the latching mechanism particular to the
cable vendor. When a logical connection is made, the Green LED will light. When data is
being transferred the Green LED will blink. See Section 9.3, “Innova IPsec 4 Lx EN LEDs,”
on page 40.
Do not use excessive force when seating the card, as this may damage the system or
the adapter.
When installing cables make sure that the latches engage.
Always install and remove cables by pushing or pulling the cable and connector in a
straight line with the card.

Rev: 1.320 Mellanox Technologies
5. Care should be taken as not to impede the air exhaust flow through the ventilation holes. Use
cable lengths which allow for routing horizontally around to the side of the chassis before
bending upward or downward in the rack.
6. To remove a cable, disengage the locks and slowly pull the connector away from the port
receptacle. LED indicator will turn off when the cable is unseated.
3.7 Identify the Card in Your System
Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-
nox Technologies”:
> lspci |grep -i Mellanox
Network controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]