Cisco C880 M4 User manual

Configuration Guide
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 146
Cisco C880 M4 Server with SAP HANA Tailored
Datacenter Integration
Design and Deploy a SAP HANA Single-Node Solution Based on Standalone
Cisco UCS C460 M4 Rack Servers with SLES for SAP 12 SP1
February 2017

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 146
Contents
Prerequisites............................................................................................................................................................4
Architecture Overview.............................................................................................................................................5
Recommended Network Layout............................................................................................................................5
Disk and File System Layout.................................................................................................................................7
Configuration Overview: SAN Environment .......................................................................................................... 8
Configuration Overview: NAS Environment .......................................................................................................... 9
Physical Components of the Cisco C880 M4 Server ..........................................................................................11
Cisco C880 M4 Server Configuration...................................................................................................................13
Set Up the Cisco Integrated Management Controller..........................................................................................13
Connect to the Management Board....................................................................................................................13
Set the Integrated Management Controller IP Address.......................................................................................14
Log in to the Web User Interface..........................................................................................................................15
Update the Firmware ..........................................................................................................................................16
Configure the System Intelligent Platform Management Interface......................................................................18
Configure BIOS Settings.....................................................................................................................................21
C880M4 System Monitoring via SNMP................................................................................................................. 30
iSCSI Boot Configuration...................................................................................................................................... 36
Prepare for iSCSI Configuration.......................................................................................................................... 36
Configure iSCSI Boot..........................................................................................................................................39
SAN Boot Configuration........................................................................................................................................ 56
Overview of Steps for Configuring the SAN Boot Environment...........................................................................56
Notes on SAN Boot System Design.................................................................................................................... 57
Notes on SAN Boot System Installation..............................................................................................................57
Configure the SAN Boot Environment.................................................................................................................58
Notes on SAN Boot System Operation...............................................................................................................71
16-Gbps Fibre Channel Card Settings: UEFI Driver Version 6.11a5 or Later ...................................................72
Set UEFI Driver Version 6.11a5 or Later on the 16-Gbps Fibre Channel Card................................................... 72
Select the Method for Configuring the 16-Gbps Fibre Channel Card.................................................................. 73
Display the Device Manager...............................................................................................................................75
Start the HBA Setup Utility and Select the HBA..................................................................................................76
Acquire an HBA WWN........................................................................................................................................78
Initialize the HBA Settings................................................................................................................................... 79
Enable the BIOS.................................................................................................................................................80
Set the Link Speed and Topology.......................................................................................................................82
Set the Boot Device............................................................................................................................................87
Apply the Settings...............................................................................................................................................96
Record the WWN Information of the Fibre Channel Port ....................................................................................98
16-Gbps Fibre Channel Card Settings: Traditional BIOS Driver Version 6.02x6 or Later.............................. 101
Configure the HBA Setup Utility........................................................................................................................101
Enable the BIOS...............................................................................................................................................103
Set the Link Speed and Topology.....................................................................................................................104
Set the Boot Device..........................................................................................................................................110
Considerations for Linux Installation................................................................................................................. 118
Linux Server OS Installation..............................................................................................................................118
SAN Multipath Boot...........................................................................................................................................118
Prepare for Linux Installation ............................................................................................................................118
Install the OS and Bundled Software................................................................................................................118
Install Emulex OneCommand Manager ............................................................................................................ 119
Install the Multipath Driver.................................................................................................................................119
Restore the Multipath Connection..................................................................................................................... 119
SLES 11 and 12 Installation: iSCSI..................................................................................................................... 121

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 146
Post-Installation Configuration .......................................................................................................................... 123
Update the OS After the Installation.................................................................................................................. 123
Verify the Driver Specification........................................................................................................................... 123
Configure and Tune the OS..............................................................................................................................123
Bond Linux Ethernet Interfaces.........................................................................................................................124
VLAN Configuration ..........................................................................................................................................126
SSH Key Distribution ........................................................................................................................................ 126
sysctl.conf.........................................................................................................................................................126
Network Time Protocol Configuration ...............................................................................................................128
SAP HANA Setup (Network File System)........................................................................................................... 129
SAP HANA Setup (SAN) ...................................................................................................................................... 130
mkfs.xfs............................................................................................................................................................. 130
/etc/fstab ...........................................................................................................................................................130
/etc/after.local.................................................................................................................................................... 130
SAP HANA Installation ........................................................................................................................................ 131
SAP Notes ........................................................................................................................................................131
Linux Notes....................................................................................................................................................... 131
SAP Application Notes......................................................................................................................................132
Notes About Third-Party Software .................................................................................................................... 132
SAP HANA Virtualization Notes........................................................................................................................132
SAP HANA High Availability (Scale-Out Only) .................................................................................................. 133
Configure High Availability................................................................................................................................133
Enable the SAP HANA Storage Connector API................................................................................................ 134
Test the IPMI Connectivity................................................................................................................................134
Autostart SAP HANA ........................................................................................................................................ 135
SAP HANA Failover Test (Scale-Out Only)........................................................................................................ 136
SAP HANA Server Network Configuration........................................................................................................138
SAP HANA Hardware Configuration Check Tool Preparation ......................................................................... 142
OS Validation.................................................................................................................................................... 142
Storage Test .....................................................................................................................................................142
Example (user = <SID>adm) ............................................................................................................................ 145
For More Information........................................................................................................................................... 146

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 146
Prerequisites
The Cisco®solution for SAP HANA requires basic to expert understanding of various aspects of IT infrastructure.
Familiarity with the following configuration and installation skills will help you build and troubleshoot this solution:
●Skills essential for building this solution
◦Cisco Nexus®Family installation and configuration
◦Cisco Unified Computing System™ (Cisco UCS®) Family installation and configuration
◦NetApp installation and configuration (optional)
◦EMC installation and configuration (optional)
◦installation and configuration
◦SUSE Linux Enterprise Server for SAP Applications 11 and 12 installation and configuration
◦Red Hat Enterprise Linux (RHEL) Server for SAP HANA 6.7 installation and configuration
●Skills highly recommended for installing the solution at the customer site
◦SAP HANA installation and configuration
◦Cisco Transport Gateway installation and configuration
◦Cisco Smart Call Home configuration on Cisco Nexus and Cisco UCS platforms
●Optional skills
◦SAP Landscape Transportation (SLT) Replication Server installation and configuration
◦SAP Business Suite configuration (focus on remote functions calls [RFCs] and interfaces)
●Additional useful skills
◦Cisco Catalyst®Family configuration
◦Configuration of Cisco Nexus 2000 Series Fabric Extenders and Cisco Nexus 3000, 5000, 7000, and
9000 Series Switches
◦Configuration of a Fibre Channel SAN switch that is compatible with the Cisco C880 M4 Server (ideally, a
Cisco MDS 9000 Family switch)
If you find any procedures that do not work as documented here, please collect as much information as you can
and open a case at support.cisco.com:
Product: Cisco C880 M4 Server
Project: CSC.datacenter
Solution: HANA-Solution
Component: Documentation

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 146
Architecture Overview
The main requirements for a SAP HANA solution are defined by SAP. The hardware partner has only limited
possibilities for altering the infrastructure design.
SAP has defined the following requirements:
●Intel®Xeon®processor E7-8880 v3 CPU
●512 GB per CPU socket for analytics (768 GB for SAP S4/HANA)
●10 Gigabit Ethernet redundant storage connectivity to network-attached storage (NAS)
●8-Gbps (or greater) Fibre Channel redundant storage connectivity to SAN
●10 Gigabit Ethernet redundant SAP HANA connectivity between SAP HANA nodes
●1 and 10 Gigabit Ethernet redundant uplink
●SUSE Linux Enterprise Server for SAP applications 11 SP3, 4, and 12 (soon SP1 as well)
●RHEL Server for SAP HANA 6.7 (soon 7.0 as well)
The architecture also includes the following components:
●Solution management components (optional)
◦Cisco 2911 Integrated Services Router (ISR; optional)
●Connectivity components (optional)
◦Cisco Nexus 2000 Series for Small Computer System Interface over IP (iSCSI) boot
◦Cisco Nexus 9000 Series for internal and external communication
◦Cisco MDS 9000 Family SAN connectivity
●Computing components
◦Cisco C880 M4 Server
◦4 x 16-Gbps Fibre Channel
◦8 x 10 Gigabit Ethernet
◦4 x 1 Gigabit Ethernet
●Storage components
◦This document describes how to use existing customer storage to connect the Cisco C880 M4 to it.
Recommended Network Layout
Figure 1 shows a possible network configuration for an SAP Business Warehouse (BW) on SAP HANA system
running on four hosts (one master, two workers, and one standby) that are connected through an internode
network (red). Two types of clients are attached: SAP BW application servers (light blue) and other SAP HANA
clients (dark blue; for example, SAP HANA Studio). The data is persistently stored on enterprise storage (purple)
and backed up to a separate storage location (green). To make this setup highly available, a standby host has
been defined; moreover, the data is replicated to another data center using SAP HANA system replication (yellow).
In addition, external administration tools can be used because a separate, low-bandwidth administrative network
(black) is configured.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 146
Figure 1. SAP HANA Network Layout
Source: SAP HANA TDI Network Guide (SAP_TDI_Network_Guide)

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 146
Disk and File System Layout
Rules from SAP and customer expectations determined the disk and file system layout. This layout must be
implemented without any changes. Any change in the disk and file system layout will automatically invalidate
certification, and SAP will deny any support. Neither local SAP consultants nor customers can change these rules.
Please report any attempt to do so to the SAP HANA product management team.
/usr/sap/<SID>
Node-specific volume for storing local HANA information
/hana/shared/<SID>
Shared volume for storing the HANA executable files
/hana/data/<SID>/mnt00[001-999]
Data volumes
/hana/log/<SID>/mnt00[001-999]
Log volumes
Figure 2 shows the layout.
Figure 2. Disk and File System Layout

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 146
Configuration Overview: SAN Environment
Figure 3 shows how to connect the Cisco C880 M4 in a SAN environment. The base unit will be delivered with four
16-Gbps Fibre Channel interfaces, which can be used for OS SAN boot and for SAP HANA shared, data, and log
volumes.
The network and SAN connectivity shown in this figure can be changed to meet customer requirements, but it must
follow the SAP HANA Tailored Datacenter Integration (TDI) rules.
Figure 3. Sample SAN Configuration

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 146
Configuration Overview: NAS Environment
Figure 4 shows how to connect the Cisco C880 M4 in an NAS environment. The base unit will be delivered with
eight 10 Gigabit Ethernet interfaces, which can be used for SAP HANA shared, data, and log volumes.
The recommended approach is to use two of the four 1-Gbps Ethernet interfaces for iSCSI boot of the OS.
The network connectivity shown in this figure can be changed to meet customer requirements, but it must follow
the SAP HANA TDI rules.
Figure 4. Sample NAS Configuration

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 146
SAP HANA TDI Versus SAP HANA Appliance
SAP HANA Tailored Datacenter Integration, or SAP HANA TDI, allows customers to use existing hardware and
infrastructure components for their SAP HANA deployments.
Although SAP HANA TDI gives customers more flexibility, saving IT budget and existing investments, customers
choosing this approach also have increased responsibility. With the SAP HANA TDI approach, only hardware is
delivered. Finding a certified SAP HANA technician to install the SAP HANA software (including the Linux OS) is
the customer’s responsibility.
Customers choosing the SAP HANA TDI delivery model are also responsible for the OS provider’s service contract
and for end-solution validation, support, and security. Unlike the SAP HANA appliance approach, in which SAP and
its hardware partners validate the solution and provide ongoing maintenance, with the TDI approach the customer
is responsible for negotiating with all the OS and hardware partners to obtain individual agreements for solution
maintenance and support.
Several tools and services are available for SAP HANA TDI customers to help with end-solution validation, go-live
testing, and ongoing support:
●Hardware Configuration Check Tool for SAP HANA (documentation attached to SAP note 1943937)
provides self-managed tests for customers to determine whether the hardware that they intend to use meets
the minimum performance criteria required to run SAP HANA in a production environment.
●SAP provides go-live-checks and other support services for SAP HANA customers as part of the SAP
support license (see SAP IT Planning for details).
●Cisco Advanced Service offers support to customers to set up an environment that meets all the SAP rules
to successfully pass the Hardware Configuration Check Tool test and help ensure a successful project
runtime environment.
Cisco Offering to Support SAP HANA TDI Customers
The Cisco Technical Assistance Center (TAC) offers a Cisco Solution Support (SSPT) contract through which the
customer can purchase appliance-like support. Cisco SSPT for SAP HANA Infrastructure helps you resolve SAP
HANA infrastructure problems faster by adding a solution-level perspective to your device-level service contracts.
This service covers groups of Cisco UCS blade and rack server deployments as well as complementary Cisco and
technology partner products.
For a detailed description, see http://www.cisco.com/web/services/portfolio/solutions-support/documents/sspt-
saphana-defintion.pdf.
Also read Zora Caklovic’s blog at https://blogs.saphana.com/2015/02/18/sap-hana-tailored-data-center-integration-
tdi-overview/.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 146
Physical Components of the Cisco C880 M4 Server
This section shows the components of the Cisco C880 M4.
Figure 5 shows the front of the Cisco C880 M4.
Figure 5. Cisco C880 M4: Front View of Connectors and System Boards

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 146
Figure 6 shows the rear of the Cisco C880 M4. Make sure that the power supply units (PSUs) and fans are placed
in the slots as shown in this figure.
Figure 6. Cisco C880 M4: Rear View of Connectors
Figure 7 shows the management board (MMB) connectors on the Cisco C880 M4.
Figure 7. Cisco C880 M4: MMB 0 Connectors

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 146
Cisco C880 M4 Server Configuration
This section describes how to configure the Cisco C880 M4 Server.
Set Up the Cisco Integrated Management Controller
First you need to configure the onboard Cisco Integrated Management Controller (IMC).
1. Connect an RS-232 cable to the communications (COM) port.
2. Connect the user ports (IMC ports) to the customer management network and VLAN to access the IMC
over your web browser (Figure 8). These two ports function as clustered ports.
Figure 8. Cisco C880 M4 MMB Connections
MMB Connectors View
RS-232 Cable
RJ-45 Connection
Customer
Management Network
CIMC Ports
100 MB/S
Note: After you power on the controller, wait for the ready (RDY) and active (ACT) LEDs to display a steady
green light before performing the initial setup.
Connect to the Management Board
After you turn on the power, the MMB initialization process starts. During the initialization process, the MMB RDY
LED flashes and the alarm LED of the unit in the front turns on.
After the initialization process is complete, the MMB RDY LED stops flashing and changes to a steady green light.
1. Verify that the COM port of the MMB is connected to the technician’s system.
2. Set up the IP address for the IMC. Open a new serial connection using putty with the settings shown in
Table 1.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 146
Table 1. Cisco IMC Connection Settings
Item
Value
Baud rate
19200
Data bit
8
Parity bit
None
Stop bit
1
Flow control
None
Emulation
VT100
Note: The MMB initialization process takes about 6 minutes to complete.
After the serial connection is opened, you will see an initial message and login prompt on the terminal software
screen. Enter the login name as Administrator. This is the preset default administrator user account name. After
entering this user name, you are asked to change your password immediately.
3. Enter a new password. Then reenter the same password.
Cisco-1541432026 login: Administrator
You are required to change the password immediately (root enforced)
New password:
Retype New password:
Administrator >
Note: To see more commands and command syntax, enter the help command.
Set the Integrated Management Controller IP Address
Now set the IMC IP address.
1. Verify that the IMC ports are connected to the management network as shown in Figure 7.
2. After resetting the password as in the previous section, enter the commands shown here to set the IMC IP
address and enable HTTP and Telnet access to it.
Administrator> set ip <ip address> <netmask>
Administrator> set gateway <ip address>
Administrator> set http enable
Administrator> set telnet enable
3. The default HTTP port is 8081. You can change this port if required by entering the following command:
Administrator> set http_port 80

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 146
Log in to the Web User Interface
Now log in to the web user interface. Table 2 shows settings.
Table 2. Settings to Log in to Web User Interface
Item
Value
URL
http://<Integrated Management controller IP>:8081
User name
Administrator
Password
Administrator’s password
Figure 9 shows the login screen.
Figure 9. Cisco IMC Web User Interface Login Screen
Figure 10 shows the system status and health screen that appears after you log in.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 146
Figure 10. Cisco C880 M4 System Status and Health Page After Initial Login
Update the Firmware
To update the server firmware, you first need to download the file.
1. Download the server firmware from http://www.cisco.com/. Choose Support > Products and select the
Cisco C880 M4 Server (Figure 11).
Figure 11. Downloading Server Firmware
2. Choose the firmware according to the processor you are using:
●Ivy-Bridge
◦Intel e7-88x0v2 firmware 1.0.x (x=4 in August 2016)
●Haswell
◦Intel e7-88x0v3 firmware 1.2.x (x=4 in August 2016)
3. Unzip the downloaded C880M4-1.0.4.zip or C880M4-1.2.4.zip archive into a new directory on the system
you use to start the MMB session.
4. Now perform the update. Log in to the system MMB through a web browser (Figure 12).

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 146
Figure 12. Logging in to the MMB
5. Choose Maintenance > Firmware Update and browse to the firmware file (Figure 13).
Figure 13. Browsing to the Firmware File
6. Click Update.
7. Wait for the firmware version check window to appear (this process takes about 5 minutes).
8. Verify the firmware version. Then click OK
After about 1 minute, the progress bar appears. The time needed to update the firmware in a single MMB is
about 70 minutes.
9. The web user interface session is disconnected during the firmware update process. For a single MMB
configuration, log in to the web user interface again after about 5 minutes.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 18 of 146
Configure the System Intelligent Platform Management Interface
By default, the Intelligent Platform Management Interface (IPMI) is not activated on the Cisco C880 M4 (Figure 14).
If you want to use this function, you must enable it by configuring it at the command line interface (CLI).
Note: IPMI is not supported for the MMB IP address. Only the baseboard management controller (BMC) IP
address can be used for the IPMI.
Figure 14.
You can check whether IPMI is enabled by entering the following command:
mgmtsrv01:/tftpboot # ipmitool -I lanplus -U admin -P admin -H 172.23.1.101
power status
Error: Unable to establish IPMI v2 / RMCP+ session
mgmtsrv01:/tftpboot #
To configure IPMI, follow these steps:
1. Set up the BMC IP address by configuring the console redirection IP address (Figure 15).

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 19 of 146
Figure 15. Configuring Console Redirection IP Address
2. Create the BMC user admin with password admin. The BMC user is disabled by default.
mgmtsrv01:/tftpboot # telnet 172.23.1.100
Telnet escape character is '\337'.
Trying 172.23.1.100...
Connected to 172.23.1.100.
login: Administrator
Password: ***
Last login: Tue Aug 2 12:00:37 from 172.25.1.6
Administrator > set bmc user admin admin admin quiet
Connection closed.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 20 of 146
3. Test the function.
mgmtsrv01:/tftpboot # ipmitool -I lanplus -U admin -P admin -H 172.23.1.101
power status
Chassis Power is off
mgmtsrv01:/tftpboot # ipmitool -I lanplus -U admin -P admin -H 172.23.1.101
power on
Chassis Power Control: Up/On
mgmtsrv01:/tftpboot # ipmitool -I lanplus -U admin -P admin -H 172.23.1.101
power status
Chassis Power is on
mgmtsrv01:/tftpboot #
Configure Clustering with IPMI as Fencing Device
If you don’t have a STONITH Block Devices (SBD) option to configure because of legging on shares Fibre Channel
disks of a RHEL 6 installation, IPMI is also an valid option which can be configured.
The STONITH configuration with IPMI for SLES is shown here.
Note: The server01-ipmi and server02-ipmi parameters refer to the BMC IP address. Add those entries in the
/etc/hosts file of both nodes.
server01:~ # crm configure
crm(live)configure# primitive STONITH-Server01 stonith:external/ipmi op monitor
interval="0" timeout="60s" op monitor interval="300s" timeout="60s" on-
fail="restart" op start interval="0" timeout="60s" on-fail="restart" params
hostname="server01" ipaddr="server01-ipmi" userid="admin" passwd="admin"
interface="lanplus"
crm(live)configure# primitive STONITH-Server02 stonith:external/ipmi op monitor
interval="0" timeout="60s" op monitor interval="300s" timeout="60s" on-
fail="restart" op start interval="0" timeout="60s" on-fail="restart" params
hostname="server11" ipaddr="server02-ipmi" userid="admin" passwd="admin"
interface="lanplus"
crm(live)configure# location LOC_STONITH_Server01 STONITH-Server01 -inf:
server01
crm(live)configure# location LOC_STONITH_Server02 STONITH-Server02 -inf:
server02