RES LaPalma User manual

LaPalma User's Guide
Copyright © 2018
Table of contents
1 . Introduction.....................................................................................................................................2
2 . System Overview............................................................................................................................2
3 . Online Document tion....................................................................................................................2
3.1 . M n p ges................................................................................................................................2
4 . Connecting to L P lm ...................................................................................................................2
4.1 . Login node...............................................................................................................................3
4.2 . Tr nsferring files......................................................................................................................3
4.3 . Gr phic l pplic tions.............................................................................................................4
5 . File Systems....................................................................................................................................4
5.1 . Root Filesystem.......................................................................................................................5
5.2 . Lustre Filesystem.....................................................................................................................5
5.3 . Loc l H rd Drive.....................................................................................................................6
6 . Running Jobs...................................................................................................................................6
6.1 . Queues (QOS)..........................................................................................................................6
6.2 . Submitting jobs........................................................................................................................7
6.2.1 . SLURM comm nds..........................................................................................................7
6.2.2 . Job directives....................................................................................................................7
6.2.3 . Ex mples...........................................................................................................................9
7 . Softw re Environment..................................................................................................................10
7.1 . Modules..................................................................................................................................10
7.2 . C Compilers............................................................................................................................11
7.2.1 . Distributed Memory P r llelism.....................................................................................11
7.2.2 . Sh red Memory P r llelism............................................................................................11
7.2.3 . Autom tic P r lleliz tion................................................................................................12
7.2.4 . 64 bit ddressing.............................................................................................................12
7.2.5 . Optimiz tion...................................................................................................................12
7.3 . FORTRAN Compilers...........................................................................................................12
7.3.1 . Distributed Memory P r llelism.....................................................................................12
7.3.2 . Sh red Memory P r llelism............................................................................................12
7.3.3 . 64 bit Addressing............................................................................................................12
7.4 . Optimiz tion..........................................................................................................................13
7.5 . Debuggers..............................................................................................................................13
7.6 . Softw re in L P lm ...............................................................................................................13
8 . Getting help...................................................................................................................................13
9 . FAQ's............................................................................................................................................13
10 . Acknowledgements.....................................................................................................................14
APPENDIX..........................................................................................................................................16
A. SSH................................................................................................................................................16
A.1. Gener ting n SSH Key p ir on Linux...................................................................................16
A.1.1. Using the ssh- gent in Linux...........................................................................................17
A.1.1.1. Using the ssh- gent in Termin l Session.............................................................17
A.1.1.2. Using ssh- gent in n X Session.............................................................................17
A.2. Gener ting n SSH Key p ir on Windows.............................................................................17
A.2.1. Using the ssh- gent in Windows......................................................................................21
1

1 . Introduction
This user's guide for the L P lm Supercomputer (v3) is intended to provide the minimum mount of
inform tion needed by new user on this system. As such, it ssumes th t the user is f mili r with m ny of the
st nd rd spects of supercomputing s the Unix oper ting system.
We hope you c n find most of the inform tion you need to use our computing resources: from pplic tions nd
libr ries to technic l document tion bout L P lm ; how to include references in public tions nd so on.
Ple se re d c refully this document nd if ny doubt rises do not hesit te to cont ct our support group t
[email protected]
2 . System Overview
L P lm comprises 252 IBM dx360 M4 compute nodes. Every node h s sixteen E5-2670 cores t 2.6 GHz,
running Linux oper ting system with 32 GB of memory RAM nd 500GB of loc l disk stor ge.
Two Bull R423 servers re connected to p ir of Net pp E5600 stor ge systems providing tot l mount of
346 TB of disk stor ge ccessible from every node through Lustre P r llel File System.
The networks th t interconnect the L P lm re:
• Infinib nd Network: High b ndwidth network used by p r llel pplic tions communic tions.
• Gig bit Network: Ethernet network used by the nodes to mount remotely their root file system from
the servers nd the network over which Lustre works.
3 . Online Documentation
3.1 . Man pages
Inform tion bout most comm nds nd softw re inst lled is v il ble vi the st nd rd UNIX man comm nd.
For ex mple, to re d bout comm nd-n me just type on shell inside L P lm :
usertest@login1:~> man command-name
which displ ys inform tion bout th t comm nd to the st nd rd output.
If you don't know the ex ct n me of the comm nd you w nt but you know the subject m tter, you c n use the
-k fl g. For ex mple:
usertest@login1:~> man -k compiler
This will print out list of ll comm nds whose m n-p ge definition includes the word 'compiler'. Then you
could execute the ex ct m n comm nd line to know bout the ex ct comm nd you were looking for.
Just to know more bout the man comm nd itself, you could lso type:
usertest@login1:~> man man
4 . onnecting to LaPalma
You must use Secure Shell (ssh) tools to login into or tr nsfer file into L P lm . We do not ccept incoming
connections from protocols s telnet, ftp, rlogin, rcp, or rsh comm nds. Once you re logged into
L P lm you c nnot m ke outgoing connections for security re sons.
To get more inform tion bout the secure shell version supported nd how to get ssh for your system
(including windows systems) see Appendix A.
2

L P lm does not support uthentic tion b sed on user nd p ssword, but key-b sed uthentic tion
mech nism. In order to get ccess to L P lm you h ve to provide us your public ssh key vi em il
(res_support@i c.es). T ke look t Appendix B for gener ting you own public/priv te key p ir.
Once you h ve provided your public ssh key you c n get into L P lm system, connecting to the login node:
lapalma1.iac.es
Here you h ve n ex mple of logging into L P lm from UNIX environment:
localsystem$ ssh -l usertest lapalma1.iac.es
+----------------------------------------------------------------------+
| |
| Welcome to aPalma |
| |
| |
| |
| * Please contact [email protected] for questions at any time |
| |
| * User Guide located at /storage/ aPalmaUserGuide.pdf |
| |
+----------------------------------------------------------------------+
usertest@login1:~>
If you re on Windows system, you need to downlo d nd inst ll Secure Shell client to perform the
connection to the m chine (See ppendix A for more inform tion).
Most of these pplic tions re gr phic l nd you will h ve to fill some inform tion in some of the fields
offered, in the field 'Host n me' or 'Remote Host n me' you will need to introduce: lapalma1.iac.es. After this
procedure you m y be logged into L P lm .
The first time th t you connect to the L P lm system secure shell needs to interch nge some initi l
inform tion to est blish the communic tion. This inform tion consists of the ccept nce of the RSA key of the
remote host, you must nswer 'yes' or 'no' to confirm the ccept nce of this key.
If you c nnot get ccess to the system fter following this procedure, first consult Appendix A for n extended
inform tion bout Secure Shell, or you c n cont ct us, (see section 9 to know how to cont ct us).
4.1 . Login node
Once you re connected into the m chine, you will be presented with UNIX shell prompt nd you will
norm lly be in your home ($HOME) directory. If you re new to UNIX, you will h ve to le rn the b sics
before you could do nything useful.
The m chine in which you will be logged in will be L P lm (login1). This m chine cts s front ends, nd
re used typic lly for editing, compiling, prep r tion/submission of b tch executions nd s g tew y for
copying d t inside or outside L P lm .
It is not permitted the execution of cpu-bound progr ms on this node, if some compil tion needs much more
cputime th n the permitted, this needs to be done through the b tch queue system.
It is not possible to connect directly to the compute nodes from the login nodes, ll resource lloc tion is done
by the b tch queue system.
4.2 . Transferring files
For tr nsferring files to L P lm from Unix systems, you c n use secure copy (scp) or secure ftp (sftp),
both tools h ve the s me synt x s the old nd insecure tools such s rcp (remote copy) nd ftp.
As h ve been s id before no connections re llowed from inside L P lm to the outside world so ll scp nd
sftp comm nds h ve to be executed from your loc l m chines nd not inside L P lm .
3

Here there re some ex mples of e ch of this tools tr nsferring files to L P lm :
localsystem$ scp localfile [email protected]
localsystem$ sftp [email protected]
sftp> put localfile
These re the w ys to retrieve files from L P lm to your loc l m chine:
localsystem$ scp [email protected]:remotefile localdir
localsystem$ sftp [email protected]
sftp> get remotefile
On Windows system, most of the secure shell clients comes with tool to m ke secure copies or secure ftp's.
There re sever l tools th t ccomplishes the requirements, ple se refer to the Appendix A, where you will
find the most common ones nd ex mples of use.
4.3 . Graphical applications
You could execute gr phic l pplic tions from the login node, to do th t the only w y is tunnelling ll the
gr phic l tr ffic through the Secure shell connection est blished.
You will need to h ve n Xserver running on your loc l m chine to be ble to show the gr phic l inform tion.
Most of the UNIX fl vors h ve n X server inst lled by def ult. In Windows environment, you will prob bly
need to downlo d nd inst ll some type of X server emul tor (see ppendix A).
The second step in order to be ble to execute gr phic l pplic tions is to en ble in your secure shell
connection the forw rding of the gr phic l inform tion through the secure ch nnel cre ted. This is norm lly
done dding the -X fl g to your norm l ssh comm nd used to connect to L P lm .
Here you h ve n ex mple:
localsystem$ ssh -X -l usertest lapalma1.iac.es
+----------------------------------------------------------------------+
| |
| Welcome to aPalma |
| |
| |
| * Please contact [email protected] for questions at any time |
| |
| * User Guide located at /storage/ aPalmaUserGuide.pdf |
| |
+----------------------------------------------------------------------+
usertest@login1:~>
For Windows systems, you will h ve to en ble the 'X11 forw rding', th t option norm lly resides on the
'Tunneling' or 'Connection' menu of the client configur tion window (see ppendix A for further det ils).
5 . File Systems
IMPORTANT: It is your responsibility s user of the L P lm system to b ckup ll your critic l d t . NO
backup of user data will be done in any of the filesystems of LaPalma.
E ch user h s sever l re s of disk sp ce for storing files. These re s m y h ve size or time limits, ple se
re d c refully ll this section to know bout the policy of us ge of e ch of these filesystems.
There re 3 different types of stor ge v il ble inside node:
4

Root filesystem: Is the filesystem where the oper ting system resides
Lustre filesystems: Lustre is distributed networked filesystem which c n be ccessed from ll the
nodes
Local hard drive: Every node h s n intern l h rd drive
Let's see them in det il.
5.1 . Root Filesystem
The root file system, where the oper ting system is stored doesn't reside in the node, this is NFS filesystem
mounted from Network Att ched Stor ge (NAS).
As this is remote filesystem only d t from the oper ting system h s to reside in this filesystem. It is NOT
permitted the use of /tmp for tempor ry user d t . The loc l h rd drive c n be used for this purpose s you
could re d in section 5.3.
Furthermore, the environment v ri ble $TMPDIR is lre dy configured to force the norm l pplic tions to use
the loc l h rd drive to store their tempor ry files.
5.2 . Lustre Filesystem
Lustre is n open-source, p r llel file system th t c n provide f st, reli ble d t ccess from ll nodes of the
cluster to glob l filesystem, with rem rk ble sc le c p city nd perform nce. Lustre llows p r llel
pplic tions simult neous ccess to set of files (even single file) from ny node th t h s the Lustre file
system mounted while providing high level of control over ll file system oper tions. These filesystems re
the recommended to use with most jobs, bec use Lustre provides high perform nce I/O by "striping" blocks of
d t from individu l files cross multiple disks on multiple stor ge devices nd re ding/writing these blocks in
p r llel. In ddition, Lustre c n re d or write l rge blocks of d t in single I/O oper tion, thereby minimizing
overhe d.
Even though there is only one Lustre filesystem mounted on L P lm , there re different loc tions for
different purposes:
storage home: This loc tion h s the home directories of ll the users. When you log into L P lm you st rt in
your home directory by def ult. Every user will h ve their own home directory to store the execut bles, own
developed sources nd their person l d t .
storage projects: In ddition to the home directory, there is directory in /storage/projects for e ch group of
users of L P lm . For inst nce, the group i c01 will h ve /storage/projects/iac01 directory re dy to use.
This sp ce is intended to store d t th t needs to be sh red between the users of the s me group or project.
All the users of the s me project will sh re their common /storage/projects sp ce nd it is responsibility of
e ch project m n ger to determine nd coordin te the better use of this sp ce, nd how it is distributed or
sh red between their users.
storage scratch: E ch L P lm user will h ve directory over /storage/scratch, you must use this sp ce to
store tempor ry files of your jobs during its execution.
The previouse three loc tions sh re the s me quot in order to limit the mount of d t th t c n be s ved by
e ch group. Since the loc tions /stor ge/home, /stor ge/projects nd /stor ge/scr tch re in the s me
filesystem, the quot ssigned is the sum of “Disk Projects” nd “Disk Scr tch” est blished by the ccess
committee.
The quot nd the us ge of sp ce c n be consulted vi the quot comm nd:
usertest@login1:~> lfs quota -g <GROUP> /storage
For ex mple, if your group h s been gr nted the following resources:
Disk Projects: 1000GB
5

Disk Scr ch: 500GB
The comm nd quot will report the sum of the two v lues:
usertest@login1:~> lfs quota -g usergroup /storage
Disk quotas for grp usergroup (gid 666):
Filesystem kbytes quota limit grace files quota limit grace
/storage/ 5000 1500000000 1500000000 - 700 10000 100000 -
The mount of files is limited s well. By def ult the quot for files is set to 100000 file.
If you need more disk sp ce or number of files, the responsible of your project h s to m ke request for this
extr sp ce needed, specifying the requested sp ce nd the re sons why it is needed. The request c n be sent
by em il or ny other w y of cont ct to the user support te m s it is expl ined in section 9 of this document.
storage apps: Over this loc tion will reside the pplic tions nd libr ries th t h ve lre dy been inst lled on
L P lm . T ke look t the directories or to section 7 of this document to know the pplic tions v il ble for
gener l use. Before inst lling ny pplic tion th t is needed by your project, first check if this pplic tion is
lre dy inst lled on the system. If some pplic tion th t you need is not on the system, you will h ve to sk
our user support te m to inst ll it. Check section 9 how to cont ct us. If it is gener l pplic tion with no
restrictions in its use, this will be in st lled over public directory, th t is over /storage/apps so ll users on
L P lm could m ke use of it. If the pplic tion needs some type of license nd its use must be restricted,
priv te directory over /storage/apps will be cre ted, so only the required users of L P lm could m ke use of
this pplic tion.
All pplic tions inst lled on /storage/apps will be inst lled, controlled nd supervised by the user support
te m. This doesn't me n th t the users could not help in this t sk, both c n work together to get the best result.
The user support c n provide his wide experience in compiling nd optimizing pplic tions in the L P lm
pl tform nd the users c n provide his knowledge of the pplic tion to be inst lled. All th t gener l
pplic tions th t h ve been modified in some w y from its norm l beh vior by the project users' for their own
study, nd m y not be suit ble for gener l use, must be inst lled over /storage/projects or /storage/home
depending on the us ge scope of the pplic tion, but not over /storage/apps.
5.3 . Local Hard Drive
Every node h s loc l h rd drive th t c n be used s loc l scr tch sp ce to store tempor ry files during
executions of one of your jobs. This sp ce is mounted over /scr tch directory. The mount of sp ce within
the /scr tch filesystem v ries from node to node (depending on the tot l mount of disk sp ce v il ble). All
d t stored in these loc l h rd drives t the compute nodes will not be v il ble from the login nodes. Loc l
h rd drive d t is not utom tic lly removed, so e ch job should h ve to remove its d t when finishes.
6 . Running Jobs
Slurm is the utility used t L P lm for b tch processing support, so ll jobs must be run through it. This
document provides inform tion for getting st rted with job execution t L P lm .
In order to keep the login nodes in proper lo d, 10 minutes limit tion in the cpu time is set for processes
running inter ctively in these nodes. Any execution t king more th n this limit should be c rried out through
the queue system.
6.1 . Queues (QOS)
The user's limits re ssigned utom tic lly to e ch p rticul r user (depending on the resources gr nted by the
Access Committee). Anyw y you re llowed to use the speci l queue: "debug" in order to perform some f st
short tests. To use the "debug" queue you need to include the #SBATCH --qos=debug directive.
Table 1. Queues
Queues Max CPUs Wall time limit
class_a 2400 72 hours
6

class_b 1200 48 hours
class_c 1200 24 housr
debug 64 10 min
interactive 1 1 hour
The specific limits ssigned to e ch user depends on the priority gr nted by the ccess committee.
Users gr nted with "high priority hours" will h ve ccess to m ximum of 4032 CPUs nd m ximum
w ll_clock_limit of 72 hours. For users with "low priority hours" the limits re 1200 CPUs nd 24 hours. If
you need to incre se these limits ple se cont ct the support group.
class_a, class_b, class_c Queues ssigned by the ccess committee nd where norm l jobs will be
executed, no speci l directive is needed to use these queues.
debug: This queue is reserved for testing the pplic tions before submitting them to the 'production'
queues. Only one job per user is llowed to run simult neously in this queue, nd the execution time
will be limited to 10 minutes. The m ximum number of nodes per pplic tion is 32. Only limited
number of jobs m y be running t the s me time in this queue. To use this queue dd the directive
#SBATCH --qos=debug
interactive: Jobs submitted to this queue will run in the inter ctive (login) node. It is intended to run
GUI pplic tions th t m y exceed the inter ctive cpu time limit. Note th t only sequenti l jobs re
llowed. To use this queue l unch the following comm nd from login1: salloc -p
interactive
6.2 . Submitting jobs
A job is the execution unit for the SLURM. A job is defined by text file cont ining set of directives
describing the job, nd the comm nds to execute.
6.2.1 . SLURM commands
These re the b sic comm nds to submit jobs:
sbatch <job_script>
submits “job script” to the queue system (see below for job script directives).
squeue
shows ll the jobs submitted.
scancel <job_id>
remove his/her job from the queue system, c ncelling the execution of the processes, if they were
lre dy running.
scontrol show job <job_id>
obt ins det iled inform tion bout specific job, including the ssigned nodes nd the possible
re sons preventing the job from running.
scontrol hold <job_id>
sets lock to the specified job. To rele se job, the s me comm nd must be run with -r option.
7

6.2.2 . Job directives
A job must cont in series of directives to inform the b tch system bout the ch r cteristics of the job.
These directives ppe r s comments in the job script, with the following synt x:
#SBATCH –-directive=<value>
Some come directives h ve shorter version, you c n use both forms:
#SBATCH -d <value>
Addition lly, the job script m y cont in set of comm nds to execute. If not, n extern l script must be
provided with the 'execut ble' directive. Here you m y find the most common directives:
#SBATCH -J <name_of_job>
The job n me th t will be ppe r when the comm nd squeue is executed. This n me is est blished by the user
nd it is different from the job_id ( ssigned by the b tch system)
#SBATCH --qos <queue_name>
The queue where the job is to be submitted. Let this field empty unless you need to use “debug” queue.
#SBATCH -t <wall_clock_limit>
The limit of w ll clock time. This is mandatory field nd you must set it to v lue gre ter th n the re l
execution time for your pplic tion nd sm ller th n the time limits gr nted to the user. Notice th t your job
will be killed fter the el psed period. Shorter limits re likely to reduce the w iting time. Form t is
HH:MM:SS or DD-HH:MM:SS
#SBATCH -D <pathname>
The working directory of your job (i.e. where the job will run). If not specified, it is the current working
directory t the time the job w s submitted.
#SBATCH -o <output_file>
The n me of the file to collect the st nd rd output (stdout) of the job. It is recommended to use %j in the n me
of the file, then slurm will utom tic lly include the job Id to void overwriting the output files if you submit
sever l jobs.
#SBATCH -e <error_file>
The n me of the file to collect the stderr output of the job. It is recommended to use %j in the n me of the file,
then slurm will utom tic lly include the job Id to void overwriting the error files if you submit sever l jobs.
#SBATCH -N <number_nodes>
The number of nodes you re sking for. Be r in mind th t e ch node h s 16 cores, so by def ult e ch node
will execute 16 t sks, one per core.
#SBATCH --cpus-per-task=<number>
The number of cpus lloc ted for e ch t sk. This is useful for hybrid MPI+OpenMP pplic tions, where e ch
8

process will sp wn number of thre ds. The number of cpus per t sk must be n integer between 1 nd 16,
since e ch node h s 16 cores (one for e ch thre d).
#SBATCH --tasks_per_node=<number>
The number of t sks lloc ted in e ch node. When n pplic tion uses more th n 1.7 GB of memory
per process, it is not possible to h ve 16 processes in the s me node nd its 32 GB of memory. It c n be
combined with the --cpus-per-task to lloc te the nodes exclusively. The number of t sks per node
must be n integer between 1 nd 16 (be r in mind th t some cores will st y idle when setting number lower
th n 16, so if it is not possible for you to use ll 16 v il ble cores, try to minimize the number of cores th t
will be w sted).
#SBATCH –-array=ini:end:step
This will en ble Array Jobs. Arr y jobs nd t sk gener tion c n be used to run pplic tions over different
inputs like you could lso do with GREASY in p st versions of L P lm or M reNostrum. It will cre te s
m ny jobs s you specify from ini to end nd the step (step by def ult is 1). You c n get the current t sk using
$SLURM_ARRAY_TASK_ID.
There re lso few SLURM environment v ri bles you c n use in your scripts:
Variable Meaning
SLURM_JOBID Specifies the job ID of the executing job
SLURM_NPROCS Specifies the tot l number of processes in the job
SLURM_NNODES Is the ctu l number of nodes ssigned to run your job
SLURM_PROCID Specifies the MPI r nk (or rel tive process ID) for the current process. The r nge is
from 0-(SLURM_NPROCS-1)
SLURM_NODEID Specifies rel tive node ID of the current job. The r nge is from 0-
(SLURM_NNODES-1)
SLURM_LOCALID Specifies the node-loc l t sk ID for the process within job
SLURM_NODELIST Specifies the list of nodes on which the job is ctu lly running
SLURM_ARRAY_TASK_ID T sk ID inside the rr y job
SLURM_ARRAY_JOB_ID Job ID (it will be the s me for ll rr y jobs, the s me s the SLURM_JOBID of the
first t sk).
6.2.3 . Examples
Example for a sequential job:
#!/bin/bash
#
#SBATCH -J test_mpi
#SBATCH -n 1
#SBATCH -t 00:02:00
#SBATCH -o test_serial-%j.out
#SBATCH -e test_serial-%j.err
#SBATCH -D .
./serial_binary
The job would be submitted using:
usertest@login1:~/Slurm/Serial> sbatch slurm_serial.cmd
9

Example for a parallel (MPI) job with 64 tasks (4 nodes):
#!/bin/bash
#
#SBATCH -J test_mpi
#SBATCH -N 4
#SBATCH -t 00:30:00
#SBATCH -o test_mpi-%j.out
#SBATCH -e test_mpi-%j.err
#SBATCH -D .
module purge
module load gcc
module load openmpi
mpirun ./mpi_binary
7 . Software Environment
7.1 . Modules
All inst lled softw re (compilers, pplic tions, numeric l nd gr phic l libr ries, tools, etc.) c n be found t
/stor ge/ pps/ or directly / pps. There is directory for e ch pplic tion (upperc se letters) nd then subdirectory with
the inst lled versions. To use ny of this softw re remember to previously lo d the environment with modules.
•Get help
% module help
•List ll v il ble softw re
% module avail
•Lo d specific softw re (def ult version)
% module load gcc
•Lo d specific softw re nd version
% module load gcc/7.2.0
•List ll lo ded softw re
% module list
•Unlo d specific softw re
% module unload gcc
•Unlo d ll softw re
% module purge
•Ch nge version of softw re
% module load gcc/4.8.5
% module switch gcc/4.8.5
7.2 .C Compilers
In L P lm you c n find the next C/C++ compilers :
gcc /g++ -> GNU Compilers for C/C++, versions 4.8.5 nd 7.2.0 (by def ult). You c n choose the compiler
you w nt to use with modules.
% module load gcc
% module load gcc/7.2.0
10