User Tools

Site Tools


Submitting batch jobs

Module environment

In order to set environment variables needed for a specific application, the module environment may be used:

  • module avail lists the available Application-Software, Compilers, Parallel-Environment, and Libraries
  • module list shows currently loaded package of your session
  • module unload <xyz> unload a particular package <xyz> from your session
  • module load <xyz> load a particular package <xyz> into your session
  • module display <xyz> OR module show <xyz> show module details such as the full path of the module file and all (or most) of the environment changes the modulefile will make if loaded
  • module purge unloads all loaded modulefiles
  1. <xyz> format corresponds exactly to the output of module avail. Thus, in order to load or unload a selected module, copy and paste exactly the name listed by module avail.
  2. a list of module load/unload directives may also be included in the top part of a job submission script

When all required/intended modules have been loaded, user packages may be compiled as usual.

SLURM (Simple Linux Utility for Resource Management)

Contrary to the previously on VSC 1 and VSC 2 employed SGE, the scheduler on VSC-3 is SLURM.

Basic SLURM commands:

  • […]$ sinfo gives information on which 'queues'='partitions' are available for job submission. Note: the under SGE termed 'queue' is called a 'partition' under SLURM.
  • […]$ scontrol is used to view SLURM configuration including: job, job step, node, partition, reservation, and overall system configuration. Without a command entered on the execute line, scontrol operates in an interactive mode and prompt for input. With a command entered on the execute line, scontrol executes that command and terminates.
  • […]$ scontrol show job 567890 shows information on the job with number 567890.
  • […]$ scontrol show partition shows information on available partitions.
  • […]$ squeue to see the current list of submitted jobs, their state and resource allocation. Here is a description of the most important job reason codes returned by the squeue command.

Node configuration - hyperthreading

The compute nodes of VSC-3 are configured with the following parameters in SLURM:


This reflects the fact that hyperthreading is activated on all compute nodes and 32 cores may be utilized on each node. In the batch script hyperthreading is selected by adding the line

#SBATCH --ntasks-per-core=2

which allows for 2 tasks per core.

Some codes may experience a performance gain from using all 32 virtual cores, e.g., GROMACS seems to profit. But note that using all virtual cores also leads to more communication and may impact on the performance of large MPI jobs.

NOTE on accounting: the project's core-h are always calculated as job_walltime * nnodes * 16 (16 physical cores per node). SLURM's built in function sreport yields wrong accounting statistics because (depending on the job script) the multiplier is 32 instead of 16. You may instead use the accounting script introduced in this section.

Node allocation policy

On VSC-3 (as on VSC-2) only complete compute Nodes , i.e., integral multiples of 16 cores, can be allocated for user jobs. If you wish to run many single core jobs, there will be a possibility to schedule them in a smart way exploiting all 16 cpus of one node, please see the scheduler script for a series of single core jobs.

Submit a batch job

Partition, quality of service and run time

Depending on the demands of a certain application, the partition (grouping hardware according its type) and quality of service (QOS; defining the run time etc.) can be selected. Additionally, the run time of a job can be limited in the job script to a value lower than the runtime limit of the selected QOS. This allows for a process called backfilling possibly leading to a shorter waiting time in the queue.

The job submission script

It is recommended to write the job script using a text editor on the VSC Linux cluster. Editors in Windows may add additional invisible characters to the job file which render it unreadable and, thus, it is not executed.

Assume a submission script check.slrm

#SBATCH -J chk
#SBATCH --ntasks-per-node=16
#SBATCH --ntasks-per-core=1
#SBATCH --mail-type=BEGIN    # first have to state the type of event to occur 
#SBATCH --mail-user=<>   # and then your email address

# when srun is used, you need to set:

<srun -l -N2 -n32 a.out > 
# or
<mpirun -np 32 a.out>
  • -J job name,
  • -N number of nodes requested (16 cores per node available)
  • -n, –ntasks=<number> specifies the number of tasks to run,
  • –ntasks-per-node number of processes run in parallel on a single node
  • –ntasks-per-core number of tasks a single core should work on
  • srun is an alternative command to mpirun. It provides direct access to SLURM inherent variables and settings.
  • -l adds task-specific labels to the beginning of all output lines.
  • –mail-type sends an email at specific events. The SLURM doku lists the following valid mail-type values: “BEGIN, END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL and REQUEUE), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), and TIME_LIMIT_50 (reached 50 percent of time limit). Multiple type values may be specified in a comma separated list.” (cited from the SLURM doku)
  • –mail-user sends an email to this address

In order to send the job to specific queues, see Queue/Partition setup on VSC-3.

Job submission

[username@l31 ~]$ sbatch check.slrm    # to submit the job             
[username@l31 ~]$ squeue -u `whoami`   # to check the status  of own jobs
[username@l31 ~]$ scancel  JOBID       # for premature removal, where JOBID
                                       # is obtained from the previous command   

A word on srun and mpirun:

Currently (27th March 2015), srun only works when the application uses intel mpi and is compiled with the intel compiler. We will provide compatible versions of MVAPICH2 and OpenMPI in the near future. At the moment, it is recommended to use mpirun in case of MVAPICH2 and OpenMPI.

Hybrid MPI/OMP:

SLURM Script:

#SBATCH --ntasks-per-node=2
#SBATCH -c 8

srun myhybridcode.exe

mpirun pins processes to cores. At least in the case of pure MPI processes (without any threads) the best performance has been observed with our default pinning (pinning to the physical cpus 0, 1, …, 15). If you need to use hybrid MPI/openMP, you may have to disable our default pinning including the following line in the job script:

export I_MPI_PIN_PROCESSOR_LIST=0,8    # configuration for 2 processes / node
export I_MPI_PIN_PROCESSOR_LIST=0,4,8,12    #              4 processes / node
export I_MPI_PIN_PROCESSOR_LIST=0,2,4,6,8,10,12,14    #    8 processes / node

or use the shell script:

if [ $PROC_PER_NODE -gt 1 ]
	if [ $PROC_PER_NODE -eq 2 ]
		export I_MPI_PIN_PROCESSOR_LIST=0,8    # configuration for 2 processes / node
	elif [ $PROC_PER_NODE -eq 4 ]
		export I_MPI_PIN_PROCESSOR_LIST=0,4,8,12    #              4 processes / node
	elif [ $PROC_PER_NODE -eq 8 ]
		export I_MPI_PIN_PROCESSOR_LIST=0,2,4,6,8,10,12,14    #    8 processes / node
		export I_MPI_PIN=disable

See also the Intel Environment Variables.

Job chain

This example is for using a set of 4 nodes to compute a series of jobs in two stages, each of them split into two separate subjobs.

vi check.slrm

#SBATCH -J chk
#SBATCH --ntasks-per-node=16
#SBATCH --ntasks-per-core=1

scontrol show hostnames $SLURM_NODELIST  > ./nodelist

srun -l -N2 -r0 -n32 job1.scrpt &
srun -l -N2 -r2 -n32 job2.scrpt &

srun -l -N2 -r2 -n32 job3.scrpt &
srun -l -N2 -r0 -n32 job4.scrpt &

the file 'nodelist' has been written for information only;
it is important to send the jobs into the background (&) and insert the 'wait' at each synchronization point;
with -r2 one can define an offset in the node list, in particular the -r2 means taking nodes number 2 and 3 from the set of four (where the list starts with node number 0), hence a combination of -N -r -n allows full control over all involved cores and the tasks they are going to be used for;

Scheduler script for many single core jobs

Given a large number of single core jobs, it is the aim to schedule them such that the needed core hours is minimized. The following script sends 64 tasks to a single node. The first 16 tasks are distributed to the 16 cores. After termination of each task, the consecutive task is started.

#SBATCH -J TEST              ## name
#SBATCH -N 1                 ## 1 node
#SBATCH --ntasks=16          ## number of tasks per node
#SBATCH --time=08:00:00      

tasks_to_be_done=64          ## total number of tasks to be scheduled:
                             ## set this to your preferred number
max_tasks=16                 ## number of tasks per node.

current_task=0               ## initialization
running_tasks=0              ## initialization

while (($current_task < $tasks_to_be_done))

   ## ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ##
   ## "test" MUST BE REPLACED by your application ~ * ~ * ##
   ## counts the number of tasks currently running  * ~ * ##
   ## ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ##
   running_tasks=`ps -C test --no-headers | wc -l`

   while (($running_tasks < $max_tasks && ${current_task} < ${tasks_to_be_done}))
      ## ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ ##
      ## run application (here named ''): replace "test" accordingly ##
      ## ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ ##
      ./test_${current_task}.sh -${current_task} & ## run executables test_0, test_1, ...

      running_tasks=`ps -C test --no-headers | wc -l`

The test_–.sh script was here of the form

sleep 10
ps | grep test|grep -v grep | wc -l

Another example for single core jobs with the parallelization in the innermost loop:

#SBATCH -J testjob
#SBATCH --ntasks-per-node=16
#SBATCH --ntasks-per-core=1
##SBATCH --mail-type=BEGIN    # first have to state the type of event to occur 
##SBATCH   # and then your email address
#SBATCH --time=60 # to use backfilling of the slurm queues

export max_num_tasks=16
export executable=dt24
for n in {1027..20000..231}; do
  for k in $((n)) $((n-1)) $((n/2)) $((n/4)) $((n/10)) ; do
    for rdb in $(eval echo 1 {8..16..4}); do
            for blocksize in $(eval echo {64..$n..32}); do
                  ./$executable $n $k $rdb $blocksize $dk                               >dt2.${n}.${k}.${rdb}.${blocksize} 2>&1 &
                  ############################## Here starts the parallelization
                  while (("$(ps -C $executable --no-headers | wc -l)" == $max_num_tasks )); do
                    sleep 10   # replace by appropriate value
                  ############################## Here ends the parallelization
wait # wait for final jobs to finish

(The SLURM inherent command #SBATCH –array starting_value-end_value:stepwidth does not provide this functionality since nodes are exclusively occupied by one job.)

Here are examples how to run multiple MPI jobs on a single node

Generating a host machines file

In general a host file is not needed. However, there may be situations where a host file is necessary, for example for Mathematica Batch Jobs. With the following script a host machines file may be generated

#SBATCH -J par                      # job name
#SBATCH -N 2                        # number of nodes=2
#SBATCH --ntasks-per-node=16        # uses all cpus of one node      
#SBATCH --ntasks-per-core=1
#SBATCH --threads-per-core=1

module load Mathematica/10.0.2 # load desired version

scontrol show hostnames $SLURM_NODELIST > ./nodelist

rm machines_tmp

tasks_per_node=16         # change number accordingly
nodes=2                   # change number accordingly
for ((line=1; line<=nodes; line++))
    for ((tN=1; tN<=tasks_per_node; tN++))
        head -$line nodelist | tail -1 >> machines_tmp

echo "this is the content of the host machines file"
cat machines_tmp

Pay attention to the fact that tasks_per_node and nodes have to be changed accordingly.

Restarting Failed Jobs

Slurm is not longer configured to automatically requeue jobs which were aborted due to node failures. If this is an unwanted behaviour you can prevent ask for automatic requeuing with the following option in your job script:

#SBATCH --requeue

Job Arrays

#SBATCH -J array_job
#SBATCH --tasks-per-node=16
#run all tasks from index 1 to 1000 with stepwidth 3, i.e. 1,4,7, ...
#SBATCH --array=1-1000:3

#usefull variables:

./start_my_task $SLURM_ARRAY_TASK_ID

Job Dependency

Dependent jobs can be used to create a job chain that is executed one job after another.

  1. Submit first job and get its <job id>
  2. Submit dependent job (and get <job_id>):
    #SBATCH -J jobname
    #SBATCH -N 2
    #SBATCH -d afterany:<job_id>
    srun  ./my_program
  3. continue at 2. for further dependent jobs


Software, that uses a license server, has to be specified upon job submission. A list of all available licensed software for your user can be shown by using the command:


Within the job script add the flags as shown with 'slic', e.g. for using both Matlab and Mathematica:

#SBATCH -L matlab@vsc,mathematica@vsc

Prolog Error Codes



doku/slurm.txt · Last modified: 2019/11/13 13:03 by jz