WR Cluster Usage

Content

• Accounts
• File Systems and I/O
• Software Packages
• Environment Module Usage
• Examples
• Available Modules
• Initial Module Environment Setup
• Using the Batch System
• Usual Steps
• 1) Specify What Should Be Done
• 2) Specify Which Resources You Need
• 3) Submit the Job
• 4) Check Job Status
• 5) Get Results
• Partitions, Limits and Job Priorities
• Environment Variables and Modules
• Special Requests
• Temporary Files
• Information about Job Runs
• GPU Nodes
• Container
• Software Development
• Interactive Development
• Compiler
• Base Software
• Parallel Programming
• Programming Tools
• Resource Requirements
• Usage Examples
• ssh Setup
• Applications
• Graphical X11 Applications
• FAQ on Problems

Accounts

File Systems and I/O

The /tmp directory is on all nodes guaranteed to be located on a fast node-local filesystem. The environment variable $TMPDIR within a batch job contains a name to a job-private fast local directory somewhere in /tmp on a node. This directory is generated on each job start with a job-specific name and removed at job termination. See additional description here. If possible, use this dynamically set environment variable to write to and read from temporary files used only in one job run.

The /work directory can be used for larger amounts of data that needs to be available longer than a batch job run and/or needs to be shared between nodes. The /work directory is shared between all nodes and access to it is slower than to a local /tmp-based file.

Please be aware that data on /tmp filessystems may be deleted without any notice after a certain period of time. And be aware that there is no backup for the /tmp and /work file systems!

mount point	purpose	located	shared on all nodes	daily backup	capacity	access time	default quota
/	operating system	local	no	no	-	-	-
/tmp	node-local temporary user data	local	no	no	small	fast	10 GB
/usr/local	application software	remote server	yes	yes	-	-	-
/home	user data	remote server	yes	yes	medium	medium	100 GB
/work	user data	remote server	yes	no	large	medium	10 TB

We have established quotas on file systems. Users can ask for their own quota with the command quota -s --show-mntpoint. The maximum number of files is per default restricted to 1 Mio. / 2 Mio. (soft / hard limit) files per file system. For the /work filesystem, the numbers are 5 Mio. / 6 Mio.

Beside the standard solutions for most users, we have individual nodes with special local I/O features. If you have special, fast or large I/O demands, contact the system administrator to find the best individual solution.

File and Directory Names

Environment Software Modules

Usage

• module avail shows a list of available modules
• module whatis shows a verbose information on a module
• module load loads a named module
• module list show all currently loaded modules
• module unload removes a named module

Example: Instead of

user@wr0: module load gcc/13.2.0

user@wr0: module load gcc

Examples

user@wr0: module avail

aocc/4.1.0            cuda/default     gnuplot/default         libFHBRS/default    openmpi/4.1.5        pin/default
aocc/default          dinero4/4.7      intel-compiler/2023     likwid/5.2.2        openmpi/default      scalasca/2.6.1
aocl/aocl-aocc-4.1.0  dinero4/default  intel-compiler/default  likwid/default      openmpi/gnu          slurm/23.02.4
aocl/aocl-gcc-4.1.0   ffmpeg/6.0       intel-mpi/2023          metis/5.2.1-32      oracle-java/20       slurm/default
aocl/default          ffmpeg/default   intel-mpi/default       metis/5.2.1-64      oracle-java/default  texlive/2023
cmake/3.27.1          gcc/13.2.0       intel-tools/2023        openjdk/20.0.2      pgi/20.1             texlive/default
cmake/default         gcc/default      intel-tools/default     openjdk/default     pgi/default          valgrind/3.21.0
cuda/12.2             gnuplot/5.4.8    libFHBRS/3.2            openmpi-system/gnu  pin/3.28             valgrind/default

------------------------------------------------- /usr/share/Modules/modulefiles -------------------------------------------------
dot  module-git  module-info  modules  null  use.own

----------------------------------------------------- /usr/share/modulefiles -----------------------------------------------------
mpi/mpich-x86_64  mpi/openmpi-x86_64

Key:
loaded  modulepath

user@wr0: module whatis gcc
gcc                  : GNU compiler suite version 10.1.0

# check current compiler version (system default without loading a module)
user@wr0: gcc --version
nt color=#FF0000>gcc --version
gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4)

# load default version
user@wr0: module load gcc
user@wr0: gcc --version
gcc (GCC) 13.2.0

# unload default version
user@wr0: module unload gcc
user@wr0: gcc --version
gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4)

Available Modules

Initial Module Enviroment Setup

module load gcc openmpi

Using the Batch System

Usual Steps

1) Specify What Should Be Done

Sequential Job

#!/bin/sh
# start sequential program
./test_sequential.exe
# change directory and execute another sequential program
cd subdir
./another_program.exe

OpenMP Job

#!/bin/sh
# set the number of threads
export OMP_NUM_THREADS=16
# start OpenMP program
./test_openmp.exe

MPI Job

#!/bin/sh
# load the OpenMPI environment
module load openmpi/gnu

# start here your MPI program
mpirun ./test_mpi.exe

2) Specify Which Resources You Need

 
#!/bin/bash
#SBATCH --partition=any          # partition / wait queue
#SBATCH --nodes=4                # number of nodes
#SBATCH --ntasks-per-node=32     # number of tasks per node
#SBATCH --mem=4G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation (format D-HH:MM:SS; first parts optional)
#SBATCH --output=slurm.%j.out    # filename for STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # filename for STDERR

# here comes the part with the description of the computational work, for example:
# load the OpenMPI environment
module load openmpi/gnu

# start here your MPI program
mpirun ./test_mpi.exe

• In the first line, a partition with the name any is requested. A partition is a class of hardware nodes with same / similar properties. For most partitions it is valid, that all nodes in that partition have the same or similar hardware properties.
• In the second line, 4 nodes (computers) of that partition are requested.
• In the following line, 32 cores per node are requested. Therefore in total 32 cores/node * 4 nodes = 128 cores.
• The option --mem=4G asks for 4GB of main memory on each of the nodes.
• The option --time=2:00 asks for a maximum of 2 minutes of usage for the requested resources.
• The last options specify two filenames where the job output whould be written to (stdout and stderr, respectively).

 
#!/bin/bash
#SBATCH --partition=any          # partition (queue)
#SBATCH --tasks=80               # number of tasks     <---------- this is different to above
#SBATCH --mem=4G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)
#SBATCH --output=slurm.%j.out    # filename for STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # filename for STDERR

# here comes the part with the description of the computational work, for example:
# load the OpenMPI environment
module load openmpi/gnu

# start here your MPI program
mpirun ./test_mpi.exe

3) Submit the Job

user@wr0: sbatch jobscript.sh

Please be aware, that all loaded modules of your interactive session (where you execute the sbatch command) are as well loaded when starting your submitted batch job. This may lead to different behaviour of a batch job for interactive sessions with differently loaded modules!

4) Check Job Status

1. You can view the batch status of all batch jobs in a web brower ( link). The page gets updated periodically.
2. You can show the status of all of your non-finished jobs in a shell window with the squeue command.

   
   user@wr0: squeue
                JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                   55       any test2.sh user     PD       0:00      2 (Resources)
                   56       any test3.sh user     PD       0:00      2 (Priority)
                   54       any test1.sh user      R       0:08      2 wr[50,51]
   

   In the example the user has 3 jobs submitted that are either running or still waiting. The column ST marks the job state (R=running, PD=waiting).

5) Get Results

user@wr0: ls -l
-rw------- 1 user fb02    316 Mar  9 07:27 slurm.51.out
-rw------- 1 user fb02  11484 Mar  9 07:27 slurm.52.err

Selected Batch System Commands

Partitions, Resource Limits and Job Priorities

Resource Limits

A list of the most important queues is:

queue name	maximum time per job	usable memory	default virt.memory/process (on CPU)	nodes used
any	72 hours	(dependent on node)	1 GB	any node
hpc	72 hours	370 GB (AMD part) or 185 GB (Intel part)	1 GB	wr50-wr106
hpc1	72 hours	370 GB	1 GB	wr50-wr74
hpc3	72 hours	185 GB	1 GB	wr75-wr106
gpu	72 hours	185 GB	1 GB	wr15-wr19
gpu4	72 hours	185 GB (wr14) or 470 GB (wr20-wr25)	1 GB	wr14,wr20-wr25
bigmem	72 hours	750 GB / 1 TB	1 GB	wr43 / wr44

Job Priorities and Job Scheduling

Environment Variables and Modules

variable name	purpose	example
$SLURM_SUBMIT_DIR	working directory where the job was submitted	/home/user/testdir
$SLURM_JOB_ID	Job ID given to the job	65
$SLURM_JOB_NAME	Job name given to the job	testjob
$SLURM_JOB_NUM_NODES	number of nodes assigned to this job	2
$SLURM_JOB_CPUS_PER_NODE	number of cores per node assigned to this job	32(x5) (32 cores, on 5 nodes)
$SLURM_JOB_NODELIST	node names of assigned nodes	wr[50,51]

Special Requests

Hyperthreading

If you do not want to use Hyperthreads (i.e. only real cores), specify in your job request additionally: #SBATCH --ntasks-per-core=1
Example: 1 hpc3-node is requested that has 32 cores / 64 hardware threads. The program starts with 32 OpenMP (software) threads spread over all cores and not using hHyperthreading.

 
#!/bin/bash
#SBATCH --partition=hpc3         # partition
#SBATCH --nodes=1                # number of nodes
#SBATCH --ntasks-per-core=1      # use only real cores
#SBATCH --time=2:00              # total runtime of job allocation

export OMP_NUM_THREADS=32
./test_openmp.exe

Hybrid Programming Models

If you want to use hybrid programming models (e.g. MPI+OpenMP), you can influence the mapping of MPI processes to the requested hardware in several ways, including

• #SBATCH --cpus-per-task=X to reserve X CPUs per (MPI-) task
• #SBATCH --ntasks-per-node=X to spread equally tasks over nodes (X tasks per node in the example)
  Example: 4 nodes with 32 cores are requested. The MPI program starts with 4*32=128 MPI processes spread over the 4 nodes.

   
  #!/bin/bash
  #SBATCH --partition=any          # partition
  #SBATCH --nodes=4                # number of nodes
  #SBATCH --ntasks-per-node=32     # number of cores per node
  #SBATCH --time=2:00              # total runtime of job allocation
  
  module load openmpi
  mpirun ./test_mpi.exe
  

Use Specific Nodes

If you want to use specific nodes (e.g. wr73), this can be specified in the resource specification part of the batch job.

• #SBATCH --nodelist=wr73 to ask for node wr73

use that only in rare cases where this is necessary. Instead of limiting a job to a certain node, it is almost always better to specify a class of nodes, i.e. a partition.

GPU nodes

See the section on GPU Nodes.

Temporary Files

char *basedir = getenv("TMPDIR");
if(basedir != NULL)
  {
    char filename = "test.dat";
    char allname[1024];
    sprintf(allname, "%s/%s", basedir, filename);
    FILE *f = fopen(all, "w");
  }

Information about Job Runs

user@wr0: sstat --format=jobid,maxvmsize,MaxDiskRead 123456.batch
       JobID  MaxVMSize  MaxDiskRead
------------ ---------- ------------
123456.batch  47654040K     39789920

If you need such information for already finishedy jobs, use the command sacct.
Example:

user@wr0: sacct -j 123456.batch --format="jobid,CPUTime,MaxVMSize,MaxDiskRead"
       JobID    CPUTime  MaxVMSize  MaxDiskRead
------------ ---------- ---------- ------------
123456.batch   01:37:04  24173824K      828.59M

GPU Nodes

• Use GPU nodes only if you can utilize them appropriately with a high utilization.
• Use the 4-way GPU nodes only, if you can utilize 4 GPU's in parallel. Otherwise use the nodes with 1 GPU.
• If you can utilize 4 GPUs in your program, use the 4-way GPU nodes.

GPU nodes are in the following queues:

batch queue	#GPUs available (including MIG slices)	#GPU(s) on node	available GPU memory
gpu	5	Nvidia V100	16 GB
gpu4	30	4x Nvidia A100 / 4x Nvidia V100	80 GB / 16 GB

To use one or more GPU's on a GPU-node (one of wr14-wr25), add --gres=gpu to your job request. For this default case, you ask for 1 GPU on the requested node(s). For the nodes wr14 and wr20-25 you may ask for up to 4 GPUs, respectively. To request more than one GPU, add --gres=gpu:4 to your job request, if you ask for example for 4 GPU's.If you don't use the --gres=... option, your job is started on the requested node without access to any GPU, i.e. CPU-only!
Example for a batch job file:

 
#!/bin/bash
#SBATCH --partition=gpu4         # GPU partition
#SBATCH --nodes=1                # number of nodes
#SBATCH --gres=gpu:4             # ask for a node with 4 GPUs
#SBATCH --time=24:00             # total runtime of job allocation

module load cuda
./my_cuda_program.exe

Use in the beginning of your GPU work node wr14 which you can access interactively, i.e. from wr0 do a ssh wr14. Here you start a short but representable program run. Afterwards you can get information about the GPU utilization of this program with the command:

user@wr14: nvidia-smi -q -d ACCOUNTING

==============NVSMI LOG==============

Timestamp                           : Fri Nov 17 16:54:34 2023
Driver Version                      : 440.33.01
CUDA Version                        : 12.3

Attached GPUs                       : 2
GPU 00000000:84:00.0
    Accounting Mode                 : Enabled
    Accounting Mode Buffer Size     : 4000
    Accounted Processes
        Process ID                  : 48147
            GPU Utilization         : 85 %
            Memory Utilization      : 81 %
            Max memory usage        : 171 MiB
            Time                    : 6936 ms
            Is Running              : 0

GPU 00000000:85:00.0
    Accounting Mode                 : Enabled
    Accounting Mode Buffer Size     : 4000
    Accounted Processes             : None

user@wr0: srun -s --jobid your-running-job-id --pty nvidia-smi

user@wr0> srun -s --jobid 123456 --pty nvidia-smi
...
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  | 00000000:01:00.0 Off |                  Off |
| N/A   28C    P0              85W / 500W |    592MiB / 81920MiB |     85%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    386553      C   ./vectorproduct.exe                         534MiB |
+---------------------------------------------------------------------------------------+

Container

Software Development

Interactice Development

• wr0 for all development where you do not need special hardware (e.g. accelerator) or want to use MPI
• wr14 for GPU development, i.e., CUDA, OpenCL, OpenACC. Additionally you can use this system for MPI tests with small data sets and a small number of MPI processes. From wr0, do a ssh -Y wr14 to work interactively on wr14. Don't start productions runs on this test system, for that use instead the batch system.

Compiler

Examples:

• Compile a C file and generate optimized code cc -O2 t.c
• Compile a Fortran file and generate optimized code that is executable on all nodes: module load intel-compiler; ifort -O2 t.f

Base Software

Intel MKL

CC      = icx
CFLAGS  = -qmkl
LDLIBS  = -qmkl

export MKL_NUM_THREADS=1

AMD AOCL

Parallel Programming

OpenMP

Example: Compile and run an OpenMP C file:

module load gcc
gcc -fopenmp -O2 t.c
export OMP_NUM_THREADS=8
./a.out

MPI

Which MPI-compilers are used can be influenced through the module command: with module load openmpi/gnu you can use the GNU compiler environment (gcc, g++, gfortran), and with module load openmpi/intel you can use the Intel compiler environment (icc, icpc, ifort). Be aware that using module load openmpi/intel the MPI compiler names mpicc etc. are mapped to the GNU compilers. To use an Intel compiler you need to specify Intel's own names for that, i.e., mpiicx, mpiicpx, mpiifort. We don't recommend Intel MPI.

All options discussed in the compiler section also apply here, e.g. optimization.

Example: Compile a MPI C file and generate optimised code:

module load openmpi/intel
mpiicx -O2 t.c

CUDA

# defines
CC              = cc
CUDA_CC         = nvcc
LDLIBS          = -lcudart

# default rules based on suffices
#       C
%.o: %.c
        $(CC) -c $(CFLAGS) -o $@ $<

#       CUDA
%.o: %.cu
        $(CUDA_CC) -c $(CUDA_CFLAGS) -o $@ $<

myprogram.exe: myprogram.o kernel.o
        $(CC) -o $@ $^ $(LDLIBS)

OpenACC

Tools

Resource Requirements

/usr/bin/time -f "%M KB" command

Usage Examples

Sequential C program

C-program named test.c

#include <stdio.h>
int main(int argc, char **argv) {
    printf("Hello world\n");
    return 0;
}

Makefile

CC     = cc
CFLAGS = -O

#default rules
%.o: %.c
        $(CC) $(CFLAGS) -c $<
%.exe: %.o
        $(CC) -o $@ $< $(LDLIBS)

default:: test_sequential.exe

Batch script

#!/bin/bash
#SBATCH --output=slurm.%j.out    # STDOUT
#SBATCH --error=slurm.%j.err     # STDERR
#SBATCH --partition=any          # partition (queue)
#SBATCH --ntasks=1               # use 1 task
#SBATCH --mem=100                # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)

# start program
./test_sequential.exe

OpenMP C program

C-program named test_openmp.c

#include <stdio.h>
#include <omp.h>
int main(int argc, char **argv) {
#pragma omp parallel
    printf("I am the %d. thread of  %d threads\n", omp_get_thread_num(), omp_get_num_threads());
    return 0;
}

Makefile

CC     = gcc -fopenmp
CFLAGS = -O

#default rules
%.o: %.c
        $(CC) $(CFLAGS) -c $<
%.exe: %.o
        $(CC) -o $@ $< $(LDLIBS)

default:: test_openmp.exe

Batch script

#!/bin/bash
#SBATCH --output=slurm.%j.out    # STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # STDERR
#SBATCH --partition=any          # partition (queue)
#SBATCH --nodes=1                # number of tasks/cores
#SBATCH --ntasks-per-node=32     # number of tasks per node
#SBATCH --mem=1G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)

# start program (with 24 threads, in total 32 threads were requested by the job)
export OMP_NUM_THREADS=24
./test_openmp.exe

MPI C program

C-program named test_mpi.c :

#include <stdio.h>
#include <unistd.h>
#include <mpi.h>
int main(int argc, char **argv) {
  int size, rank;
  char hostname[80];

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  gethostname(hostname, 80);
  printf("Hello world from %d (on node %s) of size %d!\n", rank, hostname, size);
  MPI_Finalize();
  return 0;
}

Makefile

CC     = mpicc
CFLAGS = -O

#default rules
%.o: %.c
        $(CC) $(CFLAGS) -c $<
%.exe: %.o
        $(CC) -o $@ $< $(LDLIBS)

default:: test_mpi.exe

Batch script

#!/bin/bash
#SBATCH --output=slurm.%j.out    # STDOUT (%N: nodename, %j: job-ID)
#SBATCH --error=slurm.%j.err     # STDERR
#SBATCH --partition=any          # partition (queue)
#SBATCH --nodes=5                # number of tasks/cores
#SBATCH --ntasks-per-node=32     # number of tasks per node
#SBATCH --mem=4G                 # memory per node in MB (different units with suffix K|M|G|T)
#SBATCH --time=2:00              # total runtime of job allocation ((format D-HH:MM:SS; first parts optional)

module load gcc openmpi/gnu

# start program (with maximum parallelism as specified in job request, for this example 5*32=160)
mpirun ./test_mpi.exe

Applications

Matlab

Using Matlab interactively

user@wr0: module load matlab
user@wr0: matlab

Using Matlab with the Batch System

    ...
    module load matlab
    matlab -nodisplay -nosplash -nodesktop -r "m-file"

Pitfalls Using Matlab

• If you use Matlab multithreading (with many threads) you should set the environment variable export MALLOC_ARENA_MAX=4. This influences / restricts Matlab's memory allocation in a multithreaded environment.
• Inside your Matlab script clear temporary variables.
• Do not repeatedly change the size of large matrices (memory fragmentation).

X11 applications

1. The easiest way to enable this is to login to the WR-cluster with ssh and use the ssh option -Y (or with older ssh versions also -X ) that enables X11 tunneling through your ssh connection. If your login path goes over multiple computers please be sure to use the -Y option for every intermediate host on the path.
   Example:
   

   user@another_host:  ssh -Y user@wr0.wr.inf.h-brs.de
   

   On your local computer (i.e. where the X-server is running) you must allow wr0 to open a window. Execute on your local computer in a shell: xhost +
2. Another possibility it to set the DISPLAY variable on the cluster and to allow other computers (i.e. the WR cluster) to open a window on your local X-Server.
   Example:
   • on cluster (i.e. X-client): export DISPLAY=mycomputer.mydomain:0.0
   • on your local computer (i.e. X-server): xhost + (which would allow any computer to open a window on your X-server)
   Please be aware that newer versions of X-Servers don't support by default IP-Ports but rather Unix ports and therefore this second version doesn't work.

FAQ

• I see no output/error files after a job run?
  Answer 1: Do not use Umlauts or white space in directory names in the whole path name (see File and Directory Names)
  Answer 2: The permissions of your home directory must not allow any other user to be able to write to your home directory. I.e., no write bit set for your group or for other/world.