Fujitsu PRIMEQUEST 1000 Series User manual

PRIMEQUEST 1000 Series
ServerView Mission Critical Option
Version 1.2
User Manual
September 2011 Edition

© Copyright Fujitsu Limited 2011
All hardware and software names used are trademarks of their respective manufacturers.
All rights, including rights of translation, reproduction by printing, copying or similar
methods, in part or in whole, are reserved.
Offenders will be liable for damages.
All rights, including rights created by patent grant or registration of a utility model or
design, are reserved.
Delivery subject to availability. Right of technical modification reserved.

SVmco User Guide
3
Preface
Abbreviations
This manual uses the following product name abbreviations.
Formal product name Abbreviation
PRIMEQUEST
1400S/1400E/1400L/1800E/1800L PRIMEQUEST1000
PRIMEQUEST
1400S2/1400E2/1400L2/1800E2/1800L2 PRIMEQUEST1000x２

SVmco User Guidel
4
Contents
1 Product OverView .........................................................................................1
1.1 Monitoring mechanism on PRIMEQUEST.......................................................1
1.1.1 Basic Architecture ...........................................................................................1
1.1.2 ServerView Suite.............................................................................................2
1.2 ServerView Mission Critical Option (SVmco) ..................................................3
1.2.1 Function of SVmco..........................................................................................3
1.2.2 Monitoring mechanism for hardware failure ....................................................4
2 Installation Steps...........................................................................................5
2.1 Configuring SVmco (Linux: Red Hat Enterprise Linux) ...................................5
2.1.1 Configuring the PSA-to-MMB communication LAN.........................................8
2.1.2 Confirming management LAN settings..........................................................15
2.1.3 Confirming SELinux function settings............................................................15
2.1.4 Checking the firewall function (opening ports)...............................................16
2.1.5 Setting the management LAN IP address.....................................................22
2.1.6 Setting the destinations of traps from a partition...........................................25
2.1.7 Configuring SNMP to use duplicate disks .....................................................30
2.1.8 Installing a SVmco update.............................................................................34
2.1.9 Uninstalling SVmco.......................................................................................36
2.2 Configuring SVmco (Windows Server 2008).................................................37
2.2.1 Configuring the PSA-to-MMB communication LAN.......................................42
2.2.2 Confirming management LAN settings..........................................................46
2.2.3 Installing the PSHED Plugin driver................................................................47
2.2.4 Configuring the Windows Firewall.................................................................49
2.2.5 Setting the management LAN IP address.....................................................58
2.2.6 Setting the destinations of traps from a partition...........................................60
2.2.7 Setting the Watchdog Timer for monitoring after a STOP error (fatal system
error) 62
2.2.8 Installing a SVmco update.............................................................................63
2.2.9 Uninstalling SVmco.......................................................................................65
2.3 Configuring SVmco (VMware).......................................................................66
2.3.1 Installing SVmco............................................................................................66
2.3.2 Settings after SVmco Installation...................................................................67
2.3.3 Configuring the network.................................................................................68
2.3.4 Configuring the firewall..................................................................................79
2.3.5 Configuring the default gateway....................................................................80

SVmco User Guide
5
2.3.6 Retrieving VMware SVmco maintenance information...................................82
2.3.7 Notes on VMware SVmco.............................................................................83
3 SVmco CLI (Command Line Interface) Operations..................................84
3.1 CLI command list ..........................................................................................84
3.2 Disk Management Command (diskctrl).........................................................86
3.3 SVmco Start/Stop Command (y30SVmco) .................................................101
3.4 Command for Collecting SVmco Data for Investigation (getosvmco)..........103
3.5 SNMP Security Setting Command (setsnmpsec)........................................105
3.6 PCI Card Operation Command (fjpciswap).................................................107
3.7 Firewall Setting Command for the Management LAN Interface (setmlanfw.sh)122
3.8 PSA-MMB Communication LAN Setting Command (setpsalan).................131
４．Hot Replacement of Hard Disks ...........................................................................133
４．１ Overview of Hard Disk Hot Replacement............................................................133
４．２．Adding, Removing, and Replacing Hard Disks.................................................137
４．２．１．Addition procedure ....................................................................................137
４．２．２．Removal procedure...................................................................................139
４．２．３．Replacement procedure (for hard disk failures not causing non-
responsiveness)..............................................................................................................142
４．２．４．Replacement procedure (for hard disk failures causing non-responsiveness)146
４．３．Replacing Hard Disks in a Hardware RAID Configuration................................149
４．３．１．Hot replacement of a faulty hard disk........................................................150
４．３．２．Hard disk preventive replacement.............................................................151
４．３．３．Hard Disk Replacement at Multiple Deadlock Occurrence........................155
５．PCI Card Hot Maintenance in Windows...............................................................157
５．１．Overview of Hot Maintenance...........................................................................157
５．１．１．Overall flow................................................................................................158
５．２．Common Hot Plugging Procedure for PCI Cards.............................................159
５．２．１．Replacement procedure............................................................................160
５．２．２．Addition procedure ....................................................................................162
５．２．３．About removal ...........................................................................................164
５．３．NIC Hot Plugging..............................................................................................164
５．３．１．Hot plugging a NIC incorporated into teaming...........................................164
５．３．２．Hot plugging a non-redundant NIC............................................................174
５．３．３．NIC addition procedure..............................................................................178
５．４． FC Card Hot Plugging......................................................................................178
５．４．１．Hot plugging an FC card incorporated with the ETERNUS multipath driver179
５．４．２．FC card addition procedure.......................................................................185
５．５．Hot Replacement Procedure for iSCSI.............................................................186

SVmco User Guidel
6
５．５．１．Confirming the incorporation of a card with MPD ......................................186
５．５．２．Disconnecting MPD...................................................................................199
５．５．３．Incorporating a card with MPD ..................................................................203
６．Manual SVmco Installation and Uninstallation ...................................................204
６．１．Manually Installing SVmco (Linux: Red Hat Enterprise Linux)..........................204
６．１．１．Installation flow..........................................................................................205
６．１．２．Checks before SVmco installation.............................................................205
６．１．３．Installing SVmco........................................................................................206
６．１．４．Automatic configuration during SVmco installation....................................207
６．１．５．Restarting the partition...............................................................................208
６．１．６．Settings after SVmco installation...............................................................208
６．１．７．Installing a SVmco update.........................................................................215
６．１．８．Uninstalling SVmco....................................................................................215
６．２．Manually Installing SVmco (Windows Server 2008) .........................................215
６．２．１．Installation flow..........................................................................................216
６．２．２．Checks before SVmco installation.............................................................217
６．２．３．Installing SVmco........................................................................................219
６．２．４．Automatic configuration during SVmco installation....................................221
６．２．５．Settings after SVmco installation...............................................................222
７．Messages................................................................................................................224
７．１．SVmco Messages.............................................................................................225
７．２．SVagent Messages...........................................................................................305
７．２．１．S.M.A.R.T. messages................................................................................306
７．２．２．RAS Support Service messages ...............................................................307
７．２．３．SVAgents external storage (hard disk cabinet) unit-related messages.....310
７．２．４．LAN-related messages [Linux]...................................................................316
７．２．５．Fibre Channel messages [Linux]...............................................................409
７．２．６．FCoE messages [Linux].............................................................................438
７．２．７．SCSI messages [Linux] .............................................................................465
７．２．８．SCSI common layer messages [Linux]......................................................559
７．２．９．ETERNUS multipath driver messages [Linux]...........................................667
７．２．１０．GDS messages [Linux] ..........................................................................667
７．２．１１．GLS messages [Linux]...........................................................................676
７．２．１２．ServerView RAID messages [Linux]......................................................686
７．２．１３．LAN-related messages [Windows].........................................................764
７．２．１４．Fibre Channel messages [Windows]......................................................802
７．２．１５．SCSI/SAS messages [Windows]............................................................804
７．２．１６．Disk messages [Windows].....................................................................808
７．２．１７．ETERNUS multipath driver messages [Windows] .................................811

SVmco User Guide
7
７．２．１８．GLS messages [Windows].....................................................................816
７．２．１９．ServerView RAID-related messages [Windows]....................................818
７．２．２０．STOP ERROR message [Windows]......................................................896

SVmco User Guide
1
1 Product OverView
1.1 Monitoring mechanism on PRIMEQUEST
1.1.1 Basic Architecture
The MMB(Management Board) manages the hardware for the PRIMEQUEST
1000 series.
The MMB manages the mounted components (SB, power supply unit, fan, and
LAN) and partitions in the PRIMEQUEST 1000 series server, and controls the
system clock, management LAN, and PCI_Box.
It also sets various operations, such as video redirection, text console redirection,
and Memory Mirror mode. These functions can be operated from the MMB Web-
UI.
ServerView Agent (SVagent), ServerView RAID (SV RAID) and ServerView
Mission Critical Option (SVmco) is software for monitoring hardware errors on
partitions and managing configurations. It runs as an application on the
operating system on each partition.
The MMB and ServerView Suite (SVS) can be operated from the MMB Web-UI
and SVOM Web-UI on a general-purpose PC connected to the management
LAN. No special console is needed.
All the operations and settings for system management are done through the
MMB and SVOM Web-UI. All event monitoring in PRIMEQEUST system (include
detected by MMB) can be monitored by SVOM Web-UI.
Note:
Following application software is changed to install on PRIMEQUEST1000 and
PRIMEQUEST1000x2 (or later).
PRIMEQUEST1000: PRIMEQUEST Server Agent (PSA) *1
PRIMEQUEST1000x2 (or later): SVmco

SVmco User Guidel
2
When these software is intalled by SVIM, SVIM judge machine type and install only
necessary software.
*1: PSA is system mangement software only PRIMEQUEST1000.
For detail, please refer PRIMEQUEST 1000 Series General Description Manual (C122-
B022-03EN).
1.1.2 ServerView Suite
The role for each ServerView components in partition is as below.
ServerView Operation Manager (SVOM)
To display hardware configuration and abnormal event on each partition.
ServerView Agents (SVagent)
To collect hardware configuration and detect abnormal event in partition.
ServerView RAID (SV RAID)
To collect hardware configuration and detect abnormal event for disk device and
SAS/RAID card.
ServerView Mission Critical Option (SVmco)
To notify event detected by SVagent to MMB and provide
PRIMECLUSTER/PRIMESOFT linkage function (onlu Linux).
When detect hardware problem, the following actions are executed according to event.
- REMCS report: sent from MMB
(REMCS from SVOM can not be used in case of PRIMEQUEST.)
- Mail report: sent from MMB or SVOM.
- Trap report: sent from SVagent orSV RAID
Trap report from MMB is not supported from PQ1000x2 (or later).
- Syslog or EventLog output: Output log by SVagent or SV RAID.

SVmco User Guide
3
1.2 ServerView Mission Critical Option
(SVmco)
This section provide an overview of the SVmco function.
1.2.1 Function of SVmco
SVmco(ServerView Mission Critical Option) is a basic system application
for maitenance of PRIMEQUEST server system.
SVmco cooperation with ServerView Agents(SVagent) and Management Board (MMB),
detect hardware problem on partition and send report.
SVmco's main functions are as follows:
- MMB(Management Board) linkage function
SVmco communicate with MMB, and send hardware abnormal information to MMB.
Then MMB send REMCS/Mail report.
- DISK maintenance operation
SVmco provides the command for hot-plugging of disk drives.
- REMCS
REMCS is Fujitsu's remote maintenance service.
SVmco is positioned as a REMCS agent in the partition and collects hardware
information in each partition.
- PRIMECLUSTER/PRIMESOFT integration
SVmco enables the linkage with PRIMECLUSTER/PRIMESOFT (clustering system).

SVmco User Guidel
4
1.2.2 Monitoring mechanism for hardware
failure
SVagent monitors the log output to the system log (Linux and VMware) and event log
(Windows), filters the events that have occurred according to the filter definition in
SVagent, and SVagent and SVmco takes the required action (e.g., e-mail notification,
REMCS notification, TRAP notification, log output) depending on the result.
To issue reports, the monitoring mechanism uses a dedicated LAN between SVmco and
the MMB to link with the MMB. The reports are sent through the MMB.
Notes
For VMware, the hardware messages below, which are output to the Console OS system
logs, are monitored. The relevant sections of 7.2. SVagent Messages are indicated in
parentheses. See these sections for details on messages.
- LAN ("7.2.4 LAN-related messages [Linux]")
- FC ("7.2.5 Fibre Channel messages [Linux]")
- SCSI ("7.2.7 SCSI messages [Linux]")
However, the driver messages of the SCSI common layer and those specific to VMware
and different from those of Linux are not monitored. Also, vmkernel or vmkwarning is not
monitored.

SVmco User Guide
5
2 Installation Steps
2.1 Configuring SVmco (Linux: Red Hat
Enterprise Linux)
This section describes confirmation of the required settings for SVmco operation
after operating system installation, and corresponding features about the
settings.
Confirmation of the required settings for SVmco operation, and corresponding features
about the settings
Required/
As
needed
Setting item
Automatic
setting (*1)/
Manual
setting (*2)
See
Configuring the
PSA-to-MMB
communication
LAN
Automatic
setting
2.1.1
Configuring the
PSA-to-MMB
communication
LAN
Confirming
management LAN
settings
Manual
setting
2.1.2 Confirming
management
LAN settings
Confirming
SELinux function
settings
Automatic
setting
2.1.3 Confirming
SELinux
function settings
Required
Checking the
firewall function
(opening ports)
Automatic
setting 2.1.4 Checking the firewall
function
(opening ports)

SVmco User Guidel
6
Required/
As
needed
Setting item
Automatic
setting (*1)/
Manual
setting (*2)
See
Setting the
management LAN
IP address
Manual
setting 2.1.5 Setting the management
LAN IP address
Setting the
destinations of
traps from a
partition
Manual
setting 2.1.6 Setting the destinations of
traps from a
partition
Configuring SNMP
to use duplicate
disks
Manual
setting 2.1.7 Configuring
SNMP to use
duplicate disks
Installing a SVmco
update Not applicable 2.1.8 Installing a SVmco update
As
needed
Uninstalling
SVmco (*3) Not applicable 2.1.9 Uninstalling
SVmco
*1 Automatic setting: Values are automatically set during SVmco installation.
You may need to change an automatically set value. See the section listed in the
table.
*2 Manual setting: Values are not automatically set during SVmco installation.
Make settings as described in the section referred to in the table.
*3 To operate the PRIMEQUEST 1000x2 series server, you need to first install
SVmco. If you uninstall SVmco, the following restrictions apply.
Even under an REMCS agreement, no software errors are reported.

SVmco User Guide
7
Hot maintenance of hard disks is disabled. The partition must be
stopped for maintenance.
PRIMECLUSTER linkage is disabled.
Remarks
SVIM installs SVmco as it installs the operating system. For details on
SVIM, see the ServerView Suite ServerView Installation Manager.
For details on how to manually install SVmco, see 6. Manual SVmco
Installation and Uninstallation.
The following table lists settings for SVmco operation. The installer
automatically adds or updates these settings during SVmco installation.
Settings automatically added/changed during SVmco installation
Target Action Remarks
snmpd.conf file Add setting
snmptrapd.conf file Add setting
snmptrapd start option Change
snmpd start option Change
Dedicated PSA-to-
MMB communication
LAN interface
Set IP
address
Referring to 2.1.1
Configuring the
PSA-to-MMB
communication
LAN, change
settings as needed.
iptables setting Add setting
SELinux configuration Change If SELinux is
disabled, do not

SVmco User Guidel
8
Target Action Remarks
file
(/etc/selinux/conf)
change the settings.
Note
If you change the IP address of the dedicated PSA-to-MMB communication LAN
on the MMB side or partition side, you need to then restart SVmco. Otherwise,
SVmco would not be able to report any detected errors.
2.1.1 Configuring the PSA-to-MMB
communication LAN
This section describes how to confirm the settings for the PSA-to-MMB
communication LAN.
Communication between SVmco and the MMB requires an active NIC (network
interface card) connected to the PSA-to-MMB communication LAN.
The PSA-to-MMB communication LAN is usually set with the following values during
SVmco installation.
<IP address>
172.30.0.<partition ID + 2>/24
Example: Partition ID of 2
172.30.0.4/24

SVmco User Guide
9
<Communication settings>
Auto Negotiation off
Speed 100 Mbps
Duplex full
In the following cases, change the IP address setting.
1. IP address which was automatically set has the same IP address and
subnet address as a production network or management network.
2. You are configuring a new partition using a duplicate disk from another
partition.
3. The partition ID changes because a disk installed in a partition was
moved to another partition.
Change the IP address setting by using the following procedure. If you are
changing the setting because of the above reason 1, you need to change the IP
addresses of all partitions in the cabinet and the IP address of the PSA-to-MMB
communication LAN on the MMB side to IP addresses in the same subnet.
Change the IP addresses on the MMB side from the MMB Web-UI.
Notes
Do not change the automatically set communication values. Otherwise,
PSA-MMB communication may be disabled, making SVmco functions
unavailable.

SVmco User Guidel
10
Change PSA-to-MMB LAN settings only after SVmco installation. If
changed before SVmco installation, the settings are overwritten by the
installation.
If [Interface] of [MMB-PSA IP Address] in the [Network Configuration] -
[Network Interface] window of the MMB Web-UI is Disable, you can use
event notification function of SVmco. After changing it from Disable to
Enable, the OS must be restarted.
Confirming the NIC of the PSA-to-MMB communication LAN
Confirm the interface name is assigned to the NIC of the PSA-to-MMB
communication LAN by using the following procedure.
Operations
1. Execute the ifconfig command to confirm the interface name. This
command lists the network interfaces recognized by the system.
Syntax
/sbin/ifconfig -a
Example of output
eth0, eth1, and lo shown on the left side are the interface names.
# /sbin/ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:D0:B7:53:89:C3
inet addr:10.24.17.149 Bcast:10.24.17.255

SVmco User Guide
11
Mask:255.255.255.0
inet6 addr: fe80::2d0:b7ff:fe53:89c3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500
Metric:1
RX packets:1107704 errors:0 dropped:0 overruns:0 frame:0
TX packets:2653820 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:390009908 (371.9 MiB) TX bytes:809006934
(771.5 MiB)
eth1 Link encap:Ethernet HWaddr 00:0E:0C:21:83:97
inet addr:192.168.0.162 Bcast:10.24.17.255
Mask:255.255.255.0
inet6 addr: fe80::20e:cff:fe21:8397/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500
Metric:1
RX packets:1538726 errors:0 dropped:0 overruns:0 frame:0
TX packets:356 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:341051195 (325.2 MiB) TX bytes:22862 (22.3 KiB)
Base address:0x5cc0 Memory:fbfe0000-fc000000
lo Link encap:Local Loopback

SVmco User Guidel
12
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3865 errors:0 dropped:0 overruns:0 frame:0
2. Execute the ethtool command to find the NIC of the PSA-to-MMB
communication LAN. Enter the command as shown below for each
interface displayed in step 1. Then, check the results.
The NIC of the PSA-to-MMB communication LAN is bus-info
(SEG:BUS:DEV.FUNC number) 0000:00:19:00.0.
Syntax
/sbin/ethtool -i <interface name>
Example of output
The execution results for eth0 in the example show the matching NIC for
the PSA-to-MMB communication LAN.
# /sbin/ethtool -i eth0
driver: e100
version: 3.0.27-k2-NAPI
firmware-version: N/A
bus-info: 0000:00:19.0 (Corresponds to NIC of PSA-to-MMB
communication LAN)