WR Cluster Usage

Content

• Accounts
• File Systems and I/O
• Software Packages
• Usage
• Examples
• Available Modules
• Initial Module Environment Setup
• Using the Batch System
• Usual Steps
• 1) Specify What Should Be Done
• 2) Specify Which Resources You Need
• 3) Submit the Job
• 4) Check Job Status
• 5) Get Results
• Partitions, Limits and Job Priorities
• Environment Variables and Modules
• Special Requests
• Temporary Files
• Information about Job Runs
• GPU Nodes
• Software Development
• Interactive Development
• Compiler
• Base Software
• Parallel Programming
• Programming Tools
• Resource Requirements
• Usage Examples
• ssh Setup
• Applications
• Graphical X11 Applications
• FAQ on Problems

Accounts

File Systems and I/O

The /tmp directory is on all nodes guaranteed to be located on a fast node-local filesystem. The environment variable $TMPDIR within a batch job contains a name to a job-private fast local directory somewhere in /tmp on a node. This directory is generated on each job start with a job-specific name and removed an job termination. See additional description here. If possible, use this dynamically set environment variable to write to and read from temporary files used only in one job run.

The /scratch directory can be used for larger amounts of data that needs to be available longer than a batch job run and/or needs to be shared between nodes. The /scratch directory is shared between all nodes and access to it is slow.

The /scratch2 directory is reserved for users with high I/O demands. It is shared between all nodes, has a medium capacity and has fast access from most nodes (10x higher than /scratch). Get in contact with us if you have high I/O demands.

Please be aware that data on /tmp filessystems may be deleted without any notice after a certain period of time. And be aware that there is no backup for the scratch file systems!

mount point	purpose	located	shared on all nodes	daily backup	capacity	access time	default soft quota
/	operating system	local	no	no	-	-	-
/tmp	node-local temporary user data	local	no	no	small	fast	10 GB
/usr/local	application software	remote server	yes	yes	-	-	-
/home	user data	remote server	yes	yes	medium	medium	50 GB
/scratch	user data	remote server	yes	no	large	slow	5 TB
/scratch2	user data	remote server	yes	no	large	fast	on request

We have established quotas on file systems. Users can ask for their own quota with the command quota -s --show-mntpoint. The maximum number of files is per default restricted to 1 Mio. / 2 Mio. (soft / hard limit) files per file system. For the /scratch filesystem, the numbers are 5 Mio. / 6 Mio.

Beside the standard solutions for most users, we have individual nodes with special local I/O features. If you have special, fast or large I/O demands, contact the system administrator to find the best individual solution.

File and Directory Names

Software Packages

Usage

• module avail shows a list of available modules
• module whatis shows a verbose information on a module
• module load loads a named module
• module list show all currently loaded modules
• module unload removes a named module

Example: Instead of

user@wr0: module load gcc/10.1.0

user@wr0: module load gcc

Examples

user@wr0: module avail

---------------------------------------------- /usr/local/modules/modulesfiles ----------------------------------------------
amd/default            gcc/8.1.0              intel-mpi/2018         metis/5.1.0-32         pin/default
aocc/2.1.0             gcc/8.2.0              intel-mpi/2019         metis/5.1.0-64         python/2.7.15
aocc/2.2.0             gcc/9.1.0              intel-mpi/2020         mpitools/default       python/2.7.15-dg
aocc/default           gcc/default            intel-mpi/default      nvtop/default          python/default
atop/2.3.0             gnuplot/5.2.3          intel-tools/2018       octave/4.4.0           python3/3.6.5
atop/2.4.0             gnuplot/default        intel-tools/2019       octave/default         python3/3.7.0
atop/2.5.0             hwloc/1.11.10          intel-tools/2020       ompp/0.8.5             python3/3.8.1
atop/default           hwloc/1.11.11          intel-tools/default    ompp/default           python3/3.9.0
boost/1.72.0           hwloc/1.11.13          java/10.0.1            openmpi/gnu            python3/default
cmake/3.11.1           hwloc/2.0.1            java/14.0.1            openmpi/intel          sage/8.2
cuda/10.2              hwloc/2.0.4            java/default           papi/5.6.0             sage/default
cuda/9.2               hwloc/2.1.0            libFHBRS/3.1           papi/5.7.0             slurm/18.08.3
cuda/default           hwloc/2.2.0            libFHBRS/default       papi/6.0.0             slurm/19.05.5
dinero4/4.7            hwloc/2.3.0            likwid/4.3.2           papi/default           slurm/default
dinero4/default        hwloc/default          likwid/5.0.1           pgi/18.7               texlive/2018
ffmpeg/4.0             intel-compiler/2018    likwid/default         pgi/19.4               texlive/default
ffmpeg/default         intel-compiler/2019    matlab/default         pgi/20.1               valgrind/3.13.0
gcc/10.1.0             intel-compiler/2020    matlab/R2018a          pgi/default            valgrind/default
gcc/7.3.0              intel-compiler/default matlab/R2019b          pin/3.7

---------------------------------------------- /usr/share/Modules/modulefiles -----------------------------------------------
dot         module-git  module-info modules     null        use.own

----------------------------------------------------- /etc/modulefiles ------------------------------------------------------
mpi/compat-openmpi16-x86_64  mpi/mvapich2-2.0-psm-x86_64  mpi/mvapich2-2.2-x86_64      mpi/openmpi-x86_64
mpi/mpich-3.0-x86_64         mpi/mvapich2-2.0-x86_64      mpi/mvapich2-psm-x86_64
mpi/mpich-3.2-x86_64         mpi/mvapich2-2.2-psm2-x86_64 mpi/mvapich2-x86_64
mpi/mpich-x86_64             mpi/mvapich2-2.2-psm-x86_64  mpi/openmpi3-x86_64

user@wr0: module whatis gcc
gcc                  : GNU compiler suite version 10.1.0

# check current compiler version (system default without loading a module)
user@wr0: gcc --version
nt color=#FF0000>gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28

# load default version
user@wr0: module load gcc
user@wr0: gcc --version
gcc (GCC) 10.1.0

# unload default version
user@wr0: module unload gcc
user@wr0: gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)

Available Modules

Initial Module Enviroment Setup

module load intel-compiler openmpi/intel

Using the Batch System

Usual Steps

1) Specify What Should Be Done

Sequential Job

#!/bin/sh
# start sequential program
./test_sequential.exe
# change directory and execute another sequential program
cd subdir
./another_program.exe

OpenMP Job

#!/bin/sh
# set the number of threads
export OMP_NUM_THREADS=16
# start OpenMP program
./test_openmp.exe

MPI Job

#!/bin/sh
# load the OpenMPI environment
module load openmpi/gnu

# start here your MPI program
mpirun ./test_mpi.exe

2) Specify Which Resources You Need

 
#!/bin/bash
#SBATCH --partition=any          # partition (queue)
#SBATCH --nodes=4                # number of nodes
#SBATCH --ntasks-per-node=32     # number of tasks per node
#SBATCH --mem=4G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation (format D-HH:MM:SS; first parts optional)
#SBATCH --output=slurm.%j.out    # filename for STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # filename for STDERR

# here comes the part with the description of the computational work, for example:
# load the OpenMPI environment
module load openmpi/gnu

# start here your MPI program
mpirun ./test_mpi.exe

• In the first line, a partition with the name any is requested. A partition is a class of hardware nodes. For most partitions it is valid, that all nodes have the same or similar hardware properties.
• In the second line, 4 nodes of that partition) are requested.
• In the following line, 32 cores per node are requested.
• The option --mem=4G asks for 4GB of main memory on each of the nodes.
• The option --time=2:00 asks for 2 minutes of usage for the requested resources.
• The last options specify two filenames where the job output whould be written to (stdout and stderr, respectively).

 
#!/bin/bash
#SBATCH --partition=any          # partition (queue)
#SBATCH --tasks=80               # number of tasks     <---------- this is different to above
#SBATCH --mem=4G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)
#SBATCH --output=slurm.%j.out    # filename for STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # filename for STDERR

# here comes the part with the description of the computational work, for example:
# load the OpenMPI environment
module load openmpi/gnu

# start here your MPI program
mpirun ./test_mpi.exe

3) Submit the Job

user@wr0: sbatch jobscript.sh

Please be aware, that all loaded modules of your interactive session (where you execute the sbatch command) are as well loaded when starting your submitted batch job. This may lead to different behaviour of a batch job for interactive sessions with differently loaded modules!

4) Check Job Status

1. You can view the batch status of all batch jobs in a web brower ( link). The page gets updated periodically.
2. You can show the status of all of your non-finished jobs in a shell window with the squeue command.

   
   user@wr0: squeue
                JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                   55       any test2.sh user     PD       0:00      2 (Resources)
                   56       any test3.sh user     PD       0:00      2 (Priority)
                   54       any test1.sh user      R       0:08      2 wr[50,51]
   

   In the example the user has 3 jobs submitted that are either running or still waiting. The column ST marks the job state (R=running, PD=waiting).

5) Get Results

user@wr0: ls -l
-rw------- 1 user fb02    316 Mar  9 07:27 slurm.51.out
-rw------- 1 user fb02  11484 Mar  9 07:27 slurm.52.err

Selected Batch System Commands

Partitions, Resource Limits and Job Priorities

Resource Limits

A list of the most important queues is:

queue name	maximum time per job	usable memory	default virt.memory/process	nodes used
any	72 hours	(dependent on node)	1 GB	any node
hpc	72 hours	(dependent on node)	1 GB	wr20-wr42, wr50-wr99
hpc3	72 hours	185 GB	1 GB	wr50-wr99
hpc2	72 hours	120 GB	1 GB	wr28-wr42
hpc1	72 hours	120 GB	1 GB	wr20-wr27
gpu	72 hours	185 GB	1 GB	wr12,wr15-wr19
gpu4	72 hours	185 GB	1 GB	wr15
wr14	72 hours	120 GB	1 GB	wr14
wr13	72 hours	85 GB	1 GB	wr13
wr43	72 hours	750 GB	1 GB	wr43
wr44	72 hours	1 TB	1 GB	wr44

Job Priorities and Job Scheduling

Environment Variables and Modules

variable name	purpose	example
$SLURM_SUBMIT_DIR	working directory where the job was submitted	/home/user/testdir
$SLURM_JOB_ID	Job ID given to the job	65
$SLURM_JOB_NAME	Job name given to the job	testjob
$SLURM_JOB_NUM_NODES	number of nodes assigned to this job	2
$SLURM_JOB_CPUS_PER_NODE	number of cores per node assigned to this job	32(x5) (32 cores, on 5 nodes)
$SLURM_JOB_NODELIST	node names of assigned nodes	wr[50,51]

Special Requests

Hyperthreading

If you do not want to use Hyperthreads (i.e. only real cores), specify in your job request additionally: #SBATCH --ntasks-per-core=1
Example: 1 hpc3-node is requested that has 32 cores / 64 hardware threads. The program starts with 32 OpenMP (software) threads spread over all cores and not using hHyperthreading.

 
#!/bin/bash
#SBATCH --partition=hpc3         # partition
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-core=1      # use only real cores
#SBATCH --time=2:00              # total runtime of job allocation

export OMP_NUM_THREADS=32
./test_openmp.exe

Hybrid Programming Models

If you want to use hybrid programming models (e.g. MPI+OpenMP), you can influence the mapping of MPI processes to the requested hardware in several ways, including

• #SBATCH --cpus-per-task=X to reserve X CPUs per (MPI-) task
• #SBATCH --ntasks-per-node=X to spread equally tasks over nodes (X tasks per node in the example)
  Example: 4 nodes with 32 cores are requested. The MPI program starts with 4*32=128 MPI processes spread over the 4 nodes.

   
  #!/bin/bash
  #SBATCH --partition=any          # partition
  #SBATCH --nodes=4                # number of nodes
  #SBATCH --ntasks-per-node=32     # number of cores per node
  #SBATCH --time=2:00              # total runtime of job allocation
  
  module load openmpi/gnu
  mpirun ./test_mpi.exe
  

Use Specific Nodes

If you want to use specific nodes (e.g. wr73), this can be specified in the resource spcification part of the batch job.

• #SBATCH --nodelist=wr73 to ask for node wr73

Temporary Files

char *basedir = getenv("TMPDIR");
if(basedir != NULL)
  {
    char filename = "test.dat";
    char allname[1024];
    sprintf(allname, "%s/%s", basedir, filename);
    FILE *f = fopen(all, "w");
  }

Information about Job Runs

user@wr0: sstat --format=jobid,maxvmsize,MaxDiskRead 123456.batch
       JobID  MaxVMSize  MaxDiskRead
------------ ---------- ------------
123456.batch  47654040K     39789920

If you need such information for already finishedy jobs, use the command sacct.
Example:

user@wr0: sacct -j 123456.batch --format="jobid,CPUTime,MaxVMSize,MaxDiskRead"
       JobID    CPUTime  MaxVMSize  MaxDiskRead
------------ ---------- ---------- ------------
123456.batch   01:37:04  24173824K      828.59M

GPU Nodes

GPU nodes are in the following queues:

batch queue	number of nodes	GPU cards	with tensor cores	CUDA compute capability
gpu	6	Nvidia V100	yes	7.0
gpu4	1	4x Nvidia V100	yes	7.0
wr14	1	Nvidia K80	no	3.7
hpc1	8	Nvidia K20m	no	3.5

With CUDA 11.x, the compute capabilities less than 5.2 by default are no longer supported (marked as deprecated). If you want to compile CUDA code that runs on all of our GPU platforms, either work with a Cuda version below 11 (e.g. module load cuda/10.2) or with Cuda 11.2, compile the code explicitly for all of our platforms with:

user@wr0: nvcc  -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_70,code=sm_70 ...

To optimize the throughput on our GPU nodes and to minimize waiting times for alle GPU users, please follow the rules:

• If you don't need tensor cores (e.g. you do not train nets), consider to use any of the non-tensor core GPUs.
• If you can not utilize more than 1 GPU in parallel, don't use the queue gpu4.
• If you can utilize more than 1 GPU in parallel, use the queue gpu4.

user@wr14: nvidia-smi -q -d ACCOUNTING

==============NVSMI LOG==============

Timestamp                           : Fri Nov 20 16:54:34 2020
Driver Version                      : 440.33.01
CUDA Version                        : 10.2

Attached GPUs                       : 2
GPU 00000000:84:00.0
    Accounting Mode                 : Enabled
    Accounting Mode Buffer Size     : 4000
    Accounted Processes
        Process ID                  : 48147
            GPU Utilization         : 85 %
            Memory Utilization      : 81 %
            Max memory usage        : 171 MiB
            Time                    : 6936 ms
            Is Running              : 0

GPU 00000000:85:00.0
    Accounting Mode                 : Enabled
    Accounting Mode Buffer Size     : 4000
    Accounted Processes             : None

Software Development

Interactice Development

• wr0 for all development where you do not need special hardware (e.g. accelerator) or want to use MPI
• wr14 for GPU development, i.e., CUDA, OpenCL, and MPI tests with small data sets and a small number of MPI processes. Do a ssh -Y wr14 to work interactively on wr14

Compiler

Examples:

• Compile a C file and generate optimized code cc -O2 t.c
• Compile a Fortran file and generate optimized code that is executable on all nodes: module load intel-compiler; ifort -O2 t.f

Base Software

Intel MKL

CC      = icx
CFLAGS  = -mkl
LDLIBS  = -mkl

export MKL_NUM_THREADS=1

Parallel Programming

OpenMP

Example: Compile and run an OpenMP C file:

module load intel-compiler
icx -qopenmp -O2 t.c
export OMP_NUM_THREADS=8
./a.out

MPI

Which MPI-compilers are used can be influenced through the module command: with module load openmpi/gnu you can use the GNU compiler environment (gcc, g++, gfortran), and with module load openmpi/intel you can use the Intel compiler environment (icc, icpc, ifort). Be aware that using module load openmpi/intel the MPI compiler names mpicc etc. are mapped to the GNU compilers. To use an Intel compiler you need to specify Intel's own names for that, i.e., mpiicx, mpiicpx, mpiifort.

All options discussed in the compiler section also apply here, e.g. optimization.

Example: Compile a MPI C file and generate optimised code:

module load openmpi/intel
mpiicx -O2 t.c

The MPI implementation we use (OpenMPI) has options to influence the communication medium used. Within one node, MPI processes can communicate through shared memory, Omni-Path, or Ethernet with TCP/IP. Between nodes, Omni-Path or Ethernet with TCP/IP is possible. OpenMPI ususally chooses the most appropriate medium which means you don't need to specify anything. But if you want to choose a specific and applicable medium you may specify this in the call to mpirun through the --mca btl specifier: mpirun --mca btl communication-channels ... where communication-channels is a list of comma separated communication mediums. Possible values are: sm for shared memory, openib for Omni-Path / Infiniband, and tcp for Ethernet. The last specifier must be self.

mpirun --mca btl tcp,self -np 4 mpi.exe

OpenCL and CUDA

module load opencl
cc opencltest.c -lOpenCL
./a.out

# defines
CC              = cc
CUDA_CC         = nvcc
LDLIBS          = -lcudart

# default rules based on suffices
#       C
%.o: %.c
        $(CC) -c $(CFLAGS) -o $@ $<

#       CUDA
%.o: %.cu
        $(CUDA_CC) -c $(CUDA_CFLAGS) -o $@ $<

myprogram.exe: myprogram.o kernel.o
        $(CC) -o $@ $^ $(LDLIBS)

OpenACC

module load pgi
pgcc -acc -ta=nvidia,cc3.5,nodebug openacctest.c
./a.out

Tools

Resource Requirements

/usr/bin/time -f "%M KB" command

Usage Examples

Sequential C program

C-program named test.c

#include <stdio.h>
int main(int argc, char **argv) {
    printf("Hello world\n");
    return 0;
}

Makefile

CC     = cc
CFLAGS = -O

#default rules
%.o: %.c
        $(CC) $(CFLAGS) -c $<
%.exe: %.o
        $(CC) -o $@ $< $(LDLIBS)

default:: test_sequential.exe

Batch script

#!/bin/bash
#SBATCH --output=slurm.%j.out    # STDOUT
#SBATCH --error=slurm.%j.err     # STDERR
#SBATCH --partition=any          # partition (queue)
#SBATCH --ntasks=1               # use 1 task
#SBATCH --mem=100                # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)

# start program
./test_sequential.exe

OpenMP C program

C-program named test_openmp.c

#include <stdio.h>
#include <omp.h>
int main(int argc, char **argv) {
#pragma omp parallel
    printf("I am the %d. thread of  %d threads\n", omp_get_thread_num(), omp_get_num_threads());
    return 0;
}

Makefile

CC     = gcc -fopenmp
CFLAGS = -O

#default rules
%.o: %.c
        $(CC) $(CFLAGS) -c $<
%.exe: %.o
        $(CC) -o $@ $< $(LDLIBS)

default:: test_openmp.exe

Batch script

#!/bin/bash
#SBATCH --output=slurm.%j.out    # STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # STDERR
#SBATCH --partition=any          # partition (queue)
#SBATCH --nodes=1                # number of tasks/cores
#SBATCH --ntasks-per-node=32     # number of tasks per node
#SBATCH --mem=1G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)

# start program (with 24 threads, in total 32 threads were requested by the job)
export OMP_NUM_THREADS=24
./test_openmp.exe

MPI C program

C-program named test_mpi.c :

#include <stdio.h>
#include <unistd.h>
#include <mpi.h>
int main(int argc, char **argv) {
  int size, rank;
  char hostname[80];

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  gethostname(hostname, 80);
  printf("Hello world from %d (on node %s) of size %d!\n", rank, hostname, size);
  MPI_Finalize();
  return 0;
}

Makefile

CC     = mpicc
CFLAGS = -O

#default rules
%.o: %.c
        $(CC) $(CFLAGS) -c $<
%.exe: %.o
        $(CC) -o $@ $< $(LDLIBS)

default:: test_mpi.exe

Batch script

#!/bin/bash
#SBATCH --output=slurm.%j.out    # STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # STDERR
#SBATCH --partition=any          # partition (queue)
#SBATCH --nodes=5                # number of tasks/cores
#SBATCH --ntasks-per-node=32     # number of tasks per node
#SBATCH --mem=4G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)

module load gcc openmpi/gnu

# start program (with maximum parallelism as specified in job request, for this example 5*32=160)
mpirun ./test_mpi.exe

Applications

Matlab

Using Matlab interactively

user@wr0: module load matlab
user@wr0: matlab

Using Matlab with the Batch System

    ...
    module load matlab
    matlab -nodisplay -nosplash -nodesktop -r "m-file"

Pitfalls Using Matlab

• If you use Matlab multithreading (with many threads) you should set the environment variable export MALLOC_ARENA_MAX=4. This influences / restricts Matlab's memory allocation in a multithreaded environment.
• Inside your Matlab script clear temporary variables.
• Do not repeatedly change the size of large matrices (memory fragmentation).

OpenFOAM

X11 applications

1. The easiest way to enable this is to login to the WR-cluster with ssh and use the ssh option -Y (or with older ssh versions also -X ) that enables X11 tunneling through your ssh connection. If your login path goes over multiple computers please be sure to use the -Y option for every intermediate host on the path.
   Example:
   

   user@another_host:  ssh -Y user@wr0.wr.inf.h-brs.de
   

   On your local computer (i.e. where the X-server is running) you must allow wr0 to open a window. Execute on your local computer in a shell: xhost +
2. Another possibility it to set the DISPLAY variable on the cluster and to allow other computers (i.e. the WR cluster) to open a window on your local X-Server.
   Example:
   • on cluster (i.e. X-client): export DISPLAY=mycomputer.mydomain:0.0
   • on your local computer (i.e. X-server): xhost + (which would allow any computer to open a window on your X-server)
   Please be aware that newer versions of X-Servers don't support by default IP-Ports but rather Unix ports and therefore this second version doesn't work.

FAQ

• I see no output/error files after a job run?
  Answer 1: Do not use Umlauts or white space in directory names in the whole path name (see File and Directory Names)
  Answer 2: The permissions of your home directory must not allow any other user to be able to write to your home directory. I.e., no write bit set for your group or for other/world.