Mellanox Technologies Innova IPsec User manual

Mellanox Technologieswww.mellanox.com
Mellanox Innova™ IPsec
Ethernet Adapter Card
User Manual
Rev 1.8

Doc #: MLNX-15-50911 2Mellanox Technologies
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
© Copyright 2019. Mellanox Technologies Ltd. All Rights Reserved.
Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,
CoolBox®, CORE-
Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,
GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, Mellano
x
CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox Open
Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC
®
Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular
Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, NPS®, Open Ethernet logo, PhyX®,
PlatformX®, PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of
Open Ethernet logo,
UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo ar
e
registered trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT PRODUCT(S) ᶰ AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-ISﺴ WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT
USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST
ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY
QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES
CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR
ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE
PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.

Rev 1.8 3Mellanox Technologies
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1 Product Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Features and Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3 Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4 Operating Systems/Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.5 Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6 Related Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 2 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1 Ethernet QSFP Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 LED Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Chapter 3 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.2 Operating Systems/Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Pre-installation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Bracket Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4.1 Removing the Existing Bracket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.4.2 Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.5 Card Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.6 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.6.1 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.7 Identify the Card in Your System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Chapter 4 Mellanox Innova IPsec Offload Overview . . . . . . . . . . . . . . . . . . . 21
4.1 Security Engines and IPsec Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.1 Offloaded IPsec Protocols and Internet Protocols . . . . . . . . . . . . . . . . . . . 22
4.1.1.1 IPsec Protocols. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1.1.2 Internet Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2 IPsec Offload Kernel and Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Rev 1.84 Mellanox Technologies
4.2.1 Mellanox Innova IPsec Ethernet Driver Module. . . . . . . . . . . . . . . . . . . . . 22
4.2.2 mlx5_fpga_tools Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.3 Key Generation and Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3 IPsec Offload for DPDK Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Chapter 5 IPsec Offload Software Installation and Operation. . . . . . . . . . . . 25
5.1 Installation via MLNX_OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1.1 Downloading Mellanox OFED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1.2 Installing Mellanox OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1.2.1 Installation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.1.2.2 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.1.2.3 Installation Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.1.2.4 Driver Load Upon System Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.1.2.5 mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.3 Uninstalling Mellanox OFED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.4 Updating Firmware After Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.5 UEFI Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.1.5.1 Enrolling Mellanox's x.509 Public Key On your Systems . . . . . . . . . . . . . 32
5.1.5.2 Removing Signature from Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 Installation of Kernel Module with IPsec Offload . . . . . . . . . . . . . . . . . . . 34
5.2.1 Obtaining the Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.2.2 Installing the Kernel and Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.2.3 Installing the Customized iproute2 Utility . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.3 Operating the IPsec Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.3.1 Loading/Unloading the Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.3.1.1 Automatic Load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.3.1.2 Manual Load/Unload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.3.2 Setting up an Offloaded IPsec Connection . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.3.3 Destroying IPsec Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3.4 IPsec Offload Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Chapter 6 mlx_fpga Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.1 Tool Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.2 mlx_fpga Synopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.3 Examples of mlx_fpga Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.3.1 Adding FPGA mst Device Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.3.1.1 Burning the FPGA’s Flash Device Using the mlx_fpga Burning Tool . . . . 42
6.3.1.2 Loading Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.3.1.3 Debugging Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.3.1.4 Update FPGA Image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Chapter 7 Updating Mellanox Innova IPsec Adapter Card Firmware . . . . . . 44
Chapter 8 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
8.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Rev 1.8 5Mellanox Technologies
8.2 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Chapter 9 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
9.1 MNV101512A-BCIT Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
9.2 MNV101511A-BCIT Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
9.3 Mellanox Innova IPsec EN LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
9.3.1 Network LEDs Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
9.3.2 FPGA Debug LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
9.3.3 FPGA Load-Flow Debug LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
9.4 Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . 52
9.5 Bracket Mechanical Drawing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Appendix A Fast Installation and Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A.1 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A.2 Content of Mellanox Innova IPsec Bundle. . . . . . . . . . . . . . . . . . . . . . . 56
A.3 Software, Firmware and Tools Installation . . . . . . . . . . . . . . . . . . . . . . 56
A.4 Software, Firmware and Tools Update . . . . . . . . . . . . . . . . . . . . . . . . . 58
A.5 OFED Installation with Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Appendix B Finding the MAC and Serial Number on the Adapter Card . . . . 61
Appendix C Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Appendix D Avertissements de sécurité d’installation (Warnings in French) 64
Appendix E Sicherheitshinweise (Warnings in German) . . . . . . . . . . . . . . . . 66
Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 68

Rev 1.86 Mellanox Technologies
List of Tables
Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Table 2: MNV101511A-BCIT Mellanox Innova IPsec Passive Cooling Adapter Card. . . . . . 11
Table 3: MNV101512A-BCIT Mellanox Innova IPsec Active Cooling Adapter Card. . . . . . . 11
Table 4: Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Table 5: Documents List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Table 6: mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Table 7: ethtool IPsec Offload Counters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Table 8: MNV101512A-BCIT Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Table 9: MNV101511A-BCIT Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Table 10: Physical and Logical Link Indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Table 11: FPGA Debug LEDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51
Table 12: FPGA Load-Flow Debug LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Rev 1.8 7Mellanox Technologies
List of Figures
Figure 1: Mellanox Innova IPsec EN Adapter Card Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Figure 2: IPsec Solution Layers and Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Figure 3: MNV101511A-BCIT/MNV101512A-BCIT LEDs Placement (Example) . . . . . . . . . . . . . . . . 50
Figure 4: Mechanical Drawing of MNV101511A-BCIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Figure 5: Mechanical Drawing of MNV101512A-BCIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Figure 6: Single-Port Tall Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Figure 7: Single-Port Short Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Figure 8: MNV101511A-BCIT Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Figure 9: MNV101512A-BCIT Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Rev 1.88 Mellanox Technologies
Revision History
This document was printed on August 29, 2019.
Table 1 - Revision History Table
Date Rev Comments/Changes
August
2019
1.8 • Updated Table 4, “Features,” on page 12: removed Hardware-based I/O Vir-
tualization and authentication algorithms.
• Updated Section 1.4, “Operating Systems/Distributions,” on page 14.
• Updated Figure 3, “MNV101511A-BCIT/MNV101512A-BCIT LEDs
Placement (Example),” on page 50.
• Mellanox Innova is a trademark of Mellanox Technologies Ltd.
February
2018
1.7 • Updated Section 5.2.2, “Installing the Kernel and Driver,” on page 34
• Updated Section 5.1.4, “Updating Firmware After Installation,” on page 31
December
2017
1.6 • Updated Table 8, “MNV101512A-BCIT Specifications Table,” on page 48
• Updated Table 9, “MNV101511A-BCIT Specifications Table,” on page 49
November
2017
1.5 • Added Table 3, “MNV101512A-BCIT Mellanox Innova IPsec Active Cool-
ing Adapter Card,” on page 11
• Updated Section 5.1, “Installation via MLNX_OFED,” on page 25
• Updated Section 5.1, “Installation via MLNX_OFED,” on page 25
• Added Table 9, “MNV101511A-BCIT Specifications Table,” on page 49
• Added Figure 5, “Mechanical Drawing of MNV101512A-BCIT,” on
page 53
• Updated Appendix A.3, “Software, Firmware and Tools Installation,” on
page 56
• Updated Appendix A.4, “Software, Firmware and Tools Update,” on
page 58
• Added Appendix A.5, “OFED Installation with Script,” on page 59
June 2017 1.4 • Removed MCX4732A-BCIT from document
• Changed mlx_ipsec to mlx5_core
• Updated “memory” in Section 1.2, “Features and Benefits,” on page 12
• Updated Section 4.2, “IPsec Offload Kernel and Driver,” on page 22
• Updated Section 4.3, “IPsec Offload for DPDK Applications,” on page 24
• Updated Section 5.1, “Installation via MLNX_OFED,” on page 25
• Updated Section 5.3.1, “Loading/Unloading the Module,” on page 37
• Updated Section 5.3.3, “Destroying IPsec Tunnels,” on page 39
• Updated Section 6.2, “mlx_fpga Synopsis,” on page 41
• Updated Table 8, “MNV101512A-BCIT Specifications Table,” on page 48
• Added Appendix A.3, “Software, Firmware and Tools Installation,” on
page 56
• Updated Appendix A.4, “Software, Firmware and Tools Update,” on
page 58

Rev 1.8 9Mellanox Technologies
January
2017
1.3 • Updated Section 5.2.1, “Obtaining the Kernel Modules,” on page 34
• Updated Section 5.3.2, “Setting up an Offloaded IPsec Connection,” on
page 37
• Updated Section 5.3.3, “Destroying IPsec Tunnels,” on page 39
• Updated Section 6.1, “Tool Requirements,” on page 41
• Updated Section 6.2, “mlx_fpga Synopsis,” on page 41
• Added Section 6.3.1.1, “Burning the FPGA’s Flash Device Using the mlx_f-
pga Burning Tool,” on page 42
• Updated Section 6.3.1.2, “Loading Tool,” on page 42
• Updated Chapter 7,“Updating Mellanox Innova IPsec Adapter Card Firm-
ware” on page 44
• Added Figure 7, “Single-Port Short Bracket,” on page 55
• Added Appendix A, “Fast Installation and Update,” on page 56
September
2016
1.2 • Added MNV101511A-BCIT across document:
- Section 1.1, “Product Overview,” on page 11
- Section 9.1, “MNV101512A-BCIT Specifications,” on page 48
- Figure 3, “MNV101511A-BCIT/MNV101512A-BCIT LEDs Placement (Exam-
ple),” on page 50
- Figure 4, “Mechanical Drawing of MNV101511A-BCIT,” on page 52
• Added Chapter 5,“IPsec Offload Software Installation and Operation” on
page 25
• Updated Section 5.2.2, “Installing the Kernel and Driver,” on page 34
• Updated Section 5.3.1, “Loading/Unloading the Module,” on page 37
• Updated Section 5.3.2, “Setting up an Offloaded IPsec Connection,” on
page 37
• Added Section 5.3.3, “Destroying IPsec Tunnels,” on page 39
• Removed Innova IPsec 4 Lx EN Card Drivers
• Updated Chapter 6,“mlx_fpga Tool” on page 41
• Updated Section 8.1, “General,” on page 46
• Updated Section 9.1, “MNV101512A-BCIT Specifications,” on page 48
• Updated Appendix B, “Finding the MAC and Serial Number on the Adapter
Card,” on page 64
July 2016 1.1 • Changed mlx_accel_ipsec to mlx_ipsec.
• Added Section 4.2.2, “mlx5_fpga_tools Module,” on page 23
• Updated Section 5.2.1, “Obtaining the Kernel Modules,” on page 34
• Updated Section 5.2.2, “Installing the Kernel and Driver,” on page 34
• Updated Section 5.2.3, “Installing the Customized iproute2 Utility,” on
page 35
• Added Section 5.3.4, “IPsec Offload Statistics,” on page 39
• Added “Update FPGA Image” on page 43
• Updated Chapter 5.1,“Installation via MLNX_OFED” on page 25
April
2016
1.0 First Release
Table 1 - Revision History Table
Date Rev Comments/Changes

Rev 1.810 Mellanox Technologies
1 Introduction
This is the User Guide for Mellanox Innova™ IPsec adapter card based on the ConnectX®-4 L x
Ethernet (EN) integrated circuit device with an on-board FPGA device.
The Mellanox Innova IPsec EN adapter card provides security acceleration for IPsec-enabled
networks while taking advantage of the ConnectX-4 Lx EN network controller’s best-in-class
performance, unmatched scalability, and efficiency.
The constantly growing demand for security and privacy in modern data centers, private and
public clouds, Web 2.0 infrastructure, and telecommunication systems, requires the use of
security protocols. IPsec is a protocol suite for secure Internet Protocol (IP) communications b y
authenticating and encrypting each IP packet of a communication session. However, the high
computing power required by the IPsec algorithms consumes expensive CPU cycles and limits
network connection performance.
The Mellanox Innova IPsec EN adapter offloads the processing of the IPsec algorithms, frees u p
the CPU, and eases network bottlenecks.
The adapter integrates advanced network capabilities and encryption offloading in one card,
utilizing only a single PCIe slot for both networking and crypto functions.
The Mellanox Innova IPsec EN adapter also brings ConnectX-4 Lx’s industry leading
technologies: hardware support for RDMA over Converged Ethernet, Ethernet stateless offload
engines, overlay networks, GPUDirect® technologies, and more.
This chapter covers the following topics:
• Section 1.1, “Product Overview,” on page 11
• Section 1.2, “Features and Benefits,” on page 12
• Section 1.3, “Block Diagram,” on page 14
•Section 1.4, “Operating Systems/Distributions,” on page 14
• Section 1.5, “Connectivity,” on page 15
• Section 1.6, “Related Documents,” on page 15

Introduction
Rev 1.8 11Mellanox Technologies
1.1 Product Overview
This section provides the ordering part number, port speed, number of ports, and PCI Express
speed for the various models.
Table 2 - MNV101511A-BCIT Mellanox Innova IPsec Passive Cooling Adapter Card
Table 3 - MNV101512A-BCIT Mellanox Innova IPsec Active Cooling Adapter Card
Ordering Part Number (OPN) MNV101511A-BCIT - HHHL card with Xilinx Kintex®
UltraScale™ XCKU060
Data Transmission Rate Ethernet: 10/40Gb/s
Network Connector Types Single-port QSFP
PCI Express (PCIe) SerDes Speed PCIe 3.0 x8 8GT/s
RoHS R6
Adapter IC Part Number MT27711A0-FDCF-BE
Device ID (decimal) 610
Cooling Method Passive Cooling
Ordering Part Number (OPN) MNV101512A-BCIT - HHHL card with Xilinx Kintex®
UltraScale™ XCKU060
Data Transmission Rate Ethernet: 10/40Gb/s
Network Connector Types Single-port QSFP
PCI Express (PCIe) SerDes Speed PCIe 3.0 x8 8GT/s
RoHS R6
Adapter IC Part Number MT27711A0-FDCF-BE
Device ID (decimal) 610
Cooling Method Active Cooling

Rev 1.812 Mellanox Technologies
1.2 Features and Benefits
Table 4 - Featuresa
PCI Express (PCIe) Uses PCIe Gen 3.0 (1.1 and 2.0 compatible) through an x8 edge connector up to
8GT/s
10/40 Gigabit
Ethernet
Mellanox adapters comply with the following IEEE 802.3* standards:
– IEEE Std 802.3ba 40 Gigabit Ethernet
– IEEE Std 802.3ae 10 Gigabit Ethernet
– IEEE Std 802.3ad, Link Aggregation
– IEEE Std 802.1Q, 1P VLAN tags and priority
– IEEE Std 802.1Qau Congestion Notification
– IEEE Std 802.1Qbg
– IEEE P802.1Qaz D0.2 ETS
– IEEE P802.1Qbb D1.0 Priority-based Flow Control
Memory
PCI Express - stores and accesses Ethernet fabric connection information and
packet data
SPI - includes two SPI Flash devices:
• one 16MB SPI Flash device (W25Q128FVSIG by WINBOND-NUVOTON)
for ConnectX-4 Lx device
• one 512Mb SPI flash (MT25QL512ABB8E12 by MICRON TECHNOL-
OGY) for the FPGA device
EEPROM - accessible through the I2C-compatible interface. The EEPROM
capacity is 128Kb.
2GByte DDR4 - PC-1600MT/sec Soldered on board
IPsec Offload
The Mellanox Innova IPsec adapter provides offloading of compute intensive
encryption/decryption, which are used by the IPsec protocol. Support for Linux
and Windows IPsec software interfaces ensures native integration with existing
IPsec applications, with no required changes to the user’s software. IPsec
offloading is handled by the combination of the ConnectX-4 Lx network
controller and an on-board FPGA, providing high performance and flexibility for
future enhancements and customizations. The FPGA is connected to the
ConnectX-4 Lx through a ‘bump-in-the-wire’ topology, hence encryption and
decryption are performed inline with the network flow. This results in lower
latency and additional savings of CPU resources compared to other IPsec
protocol solutions, be it through software or alternative accelerators.
Overlay Networks
In order to better scale their networks, data center operators often create overlay
networks that carry traffic from individual virtual machines over logical tunnels
in encapsulated formats such as NVGRE and VXLAN. While this solves network
scalability issues, it hides the TCP packet from the hardware offloading engines,
placing higher loads on the host CPU. The Mellanox Innova IPsec adapter
effectively addresses this by providing advanced NVGRE, VXLAN and
GENEVE hardware offloading engines that encapsulate and de-capsulate the
overlay protocol headers, enabling the traditional offloads to be performed on the
encapsulated traffic for these and other tunneling protocols (GENEVE, MPLS,
QinQ, and so on). With the Mellanox Innova IPsec adapter, data center operators
can achieve native performance in the new network architecture.

Introduction
Rev 1.8 13Mellanox Technologies
RDMA and RDMA
over Converged
Ethernet (RoCE)
Mellanox Innova IPsec adapter supports RoCE specifications delivering low-
latency and high- performance over Ethernet networks. Leveraging data center
bridging (DCB) capabilities as well as Mellanox Innova IPsec adapter advanced
congestion control hardware mechanisms, RoCE provides efficient low-latency
RDMA services over Layer 2 and Layer 3 networks.
Mellanox
PeerDirect®
Mellanox PeerDirect® communication provides high efficiency RDMA access by
eliminating unnecessary internal data copies between components on the PCIe
bus (for example, from GPU to CPU), and therefore significantly reduces
application run time. Mellanox Innova IPsec adapter advanced acceleration
technology enables higher cluster efficiency and scalability to tens of thousands
of nodes.
CPU offload Adapter functionality enabling reduced CPU overhead allowing more available
CPU for computation tasks.
Quality of Service
(QoS)
Support for port-based Quality of Service enabling various application
requirements for latency and SLA
Storage
Acceleration
A consolidated compute and storage network achieves significant cost-
performance advantages over multi-fabric networks. Standard block and
file access protocols can leverage RDMA for high-performance storage access.
Distributed RAID
Mellanox Innova IPsec adapter delivers advanced Erasure Coding offloading
capability, enabling distributed Redundant Array of Inexpensive Disks (RAID), a
data storage technology that combines multiple disk drive components into a
logical unit for the purposes of data redundancy and performance improvement.
Mellanox Innova IPsec adapter’s Reed-Solomon capability introduces redundant
block calculations, which, together with RDMA, achieves high performance and
reliable storage access.
a. This section describes hardware features and capabilities. Please refer to the driver release notes for fea-
ture availability. See Section 1.6, “Related Documents,” on page 15.
Table 4 - Featuresa

Rev 1.814 Mellanox Technologies
1.3 Block Diagram
Figure 1: Mellanox Innova IPsec EN Adapter Card Block Diagram
1.4 Operating Systems/Distributions1
• RHEL/CentOS
1. Please refer to the driver release notes for feature availability.
ConnectX
DRAM
x8 PCIe Gen3
FPGA
Config. FLASH
x8 PCIe Gen3
JTAG -IF
QSFP
I2C
JTAG/
GPIO
2GB
DDR 4-
1600[MT/Sec]

Introduction
Rev 1.8 15Mellanox Technologies
1.5 Connectivity
• Interoperable with 10Gb and 40Gb Ethernet switches
• Passive copper cable with ESD protection
• Powered connectors for optical and active cable support
1.6 Related Documents
Table 5 - Documents List
Document’s Name Location
Mellanox Firmware Tools (MFT)
User Manual
Document no. 2204UG
User Manual describing the set of MFT firmware management
tools for a single node.
See http://www.mellanox.com/page/management_tools
Mellanox OFED for Linux
User Manual
Document no. 2877
User Manual describing OFED features, performance, tools
content and configuration. See http://www.mellanox.com =>
Products => Software => InfiniBand/VPI Drivers => Mellanox
OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)
Performance Tuning Guidelines
for Mellanox Network Adapters
Document no. 3368
User Manual describes important tuning parameters and
settings that can improve performance for Mellanox drivers.
See http://www.mellanox.com/related-docs/prod_software/
Performance_Tuning_Guide_for_Mellanox_Network_Adapters
.pdf
MellanoxO FED for Linux Driver
Release Notes
Release notes for Mellanox Technologies' MLNX_OFED for
Linux driver kit for Mellanox adapter cards:
http://www.mellanox.com => Products => Software =>
InfiniBand/VPI Drivers => Mellanox OpenFabrics Enterprise
Distribution for Linux (MLNX_OFED) => Release Notes
IEEE Std 802.3 Specification This is the IEEE Ethernet specification
http://standards.ieee.org/getieee802
PCI Express 3.0 Specifications Industry Standard PCI Express 3.0 Base and Card
Electromechanical Specifications.
https://pcisig.com/specifications
IETF IPsec Specifications https://tools.ietf.org/html/rfc4301

Rev 1.816 Mellanox Technologies
2 Interfaces
Each adapter card includes the following interfaces:
• “Ethernet QSFP Interface”
• “PCI Express Interface”
• “LED Interface”
The adapter cards include special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
2.1 Ethernet QSFP Interface
The network port of the Mellanox Innova IPsec adapter is compliant with the IEEE 802.3
Ethernet standards listed in Table 4, “Features,” on page 12. For connecting to an SFP+ interface,
you can use a Mellanox QSA (QSFP to SFP+) adapter module.
2.2 PCI Express Interface
The Mellanox Innova IPsec adapter card supports PCI Express 3.0 (1.1 and 2.0 compatible)
through an x8 edge connector. The device can be either a master initiating the PCI Express bus
operations or a slave responding to PCI bus operations. The following lists PCIe interface
features of the Mellanox Innova IPsec adapter card:
• PCIe Gen 3.0 compliant, 1.1 and 2.0 compatible
• 2.5, 5.0, or 8.0GT/s link rate x8
• Auto-negotiates to x8, x4, x2, or x1
• Support for MSI/MSI-X mechanisms
2.3 LED Interface
For Mellanox Innova IPsec adapter card LED specifications, please refer to Section 9.3,
“Mellanox Innova IPsec EN LEDs,” on page 50.

Hardware Installation
Rev 1.8 17Mellanox Technologies
3 Hardware Installation
3.1 System Requirements
3.1.1 Hardware
A system with a PCI Express x8 slot is required for installing the card.
3.1.2 Operating Systems/Distributions
Please refer to Section 1.4, “Operating Systems/Distributions,” on page 14.
3.2 Safety Precautions
1. Remove any metallic objects from your hands and wrists.
2. Make sure to use only insulated tools.
3. Verify that the system is powered off and is unplugged.
4. It is strongly recommended to use an ESD strap or other antistatic devices.
3.3 Pre-installation Checklist
1. Verify that your system meets the hardware and software requirements stated above.
2. Shut down your system if active.
3. After shutting down the system, turn off power and unplug the cord.
4. Remove the card from its package. Please note that the card must be placed on an antistatic
surface.
5. Check the card for visible signs of damage. Do not attempt to install the card if damaged.
3.4 Bracket Installation Instructions
The card is usually shipped with a tall bracket installed. If this form factor is suitable for your
requirements, you can skip the remainder of this section and move to Section 3.5, “Card Installa-
tion Instructions,” on page 18.
If you need to replace it with the short bracket that is included in the shipping box, please follow
the instructions in this section.
The adapter is being installed in a system that operates with voltages that can be lethal.
Before opening the case of the system, observe the following precautions to avoid injury and
prevent damage to system components.

Rev 1.818 Mellanox Technologies
To replace the bracket you will need the following parts:
• The new bracket of the proper height
• The 2 screws saved from the removal of the bracket
• The 2 fiber washers saved from the removal of the bracket
3.4.1 Removing the Existing Bracket
1. Remove the two screws holding the bracket in place. The bracket comes loose from the card.
2. Save the two screws and the two fiber washers.
3.4.2 Installing the New Bracket
1. Place the bracket onto the card until the screw holes line up.
2. Screw on the bracket using the screws and washers saved from the bracket removal procedure
above.
3. Make sure that the LEDs are aligned onto the bracket holes.
4. Use a torque driver to apply up to 2.9 lbs-in torque on the screws.
3.5 Card Installation Instructions
1. Open the system case.
2. Place the adapter in a standard PCI Express slot.
3. Applying even pressure at both corners of the card, insert the adapter card into the slot until it
is firmly seated. When the adapter is properly seated, the adapter port connectors are aligned
with the slot opening, and the adapter faceplate is visible against the system chassis.
Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more
than three times.
Be careful not to put stress on the LED.
Do not force the bracket onto the card. You may have to gently push the LEDs using a
small screwdriver to align the LEDs with the holes in the bracket.

Hardware Installation
Rev 1.8 19Mellanox Technologies
3.6 Cables and Modules
To obtain the list of supported cables for your adapter, please refer to http://www.mellanox.com/
products/interconnect/cables-configurator.php.
3.6.1 Cable Installation
1. All cables can be inserted or removed with the unit powered on.
2. To insert a cable, press the connector into the port receptacle until the connector is firmly
seated.
a. Support the weight of the cable before connecting the cable to the adapter card. Do this by using a cable
holder or tying the cable to the rack.
b. Determine the correct orientation of the connector to the card before inserting the connector. Do not try and
insert the connector upside down. This may damage the adapter card.
c. Insert the connector into the adapter card. Be careful to insert the connector straight into the cage. Do not
apply any torque, up or down, to the connector cage in the adapter card.
d. Make sure that the connector locks in place.
3. After inserting a cable into a port, the Amber LED indicator will light when the physical con-
nection is established (that is, when the unit is powered on and a cable is plugged into the port
with the other end of the connector plugged into a functioning port). See Section 9.3, “Mella-
nox Innova IPsec EN LEDs,” on page 50.
4. After plugging in a cable, lock the connector using the latching mechanism particular to the
cable vendor. When a logical connection is made, the Green LED will light. When data is
being transferred the Green LED will blink. See Section 9.3, “Mellanox Innova IPsec EN
LEDs,” on page 50.
Do not use excessive force when seating the card, as this may damage the system or the
adapter.
When installing cables make sure that the latches engage.
Always install and remove cables by pushing or pulling the cable and connector in a
straight line with the card.

Rev 1.820 Mellanox Technologies
5. Care should be taken as not to impede the air exhaust flow through the ventilation holes. Use
cable lengths which allow for routing horizontally around to the side of the chassis before
bending upward or downward in the rack.
6. To remove a cable, disengage the locks and slowly pull the connector away from the port
receptacle. LED indicator will turn off when the cable is unseated.
3.7 Identify the Card in Your System
Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-
nox Technologies”:
lspci |grep -i Mellanox
Network controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]