Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
doku:slurm [2017/06/27 12:31] – [SLURM (Simple Linux Utility for Resource Management)] ir | doku:slurm [2023/03/14 12:34] – [Job submission] goldenberg | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== | + | ====== |
- | ===== Module environment ===== | + | Contrary to the previously on VSC-1 and VSC-2 employed SGE, the scheduler on VSC-3, VSC-4, and VSC-5 is [[http:// |
- | In order to set environment variables needed for a specific application, | + | ==== Basic SLURM commands: |
- | * '' | + | * '' |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | == Note: == | + | |
- | + | ||
- | - **< | + | |
- | - a list of '' | + | |
- | + | ||
- | When all required/ | + | |
- | ===== SLURM (Simple Linux Utility for Resource Management) ===== | + | |
- | + | ||
- | Contrary to the previously on VSC 1 and VSC 2 employed SGE, the scheduler on VSC-3 is [[http:// | + | |
- | ==Basic SLURM commands: | + | |
- | * '' | + | |
* '' | * '' | ||
* '' | * '' | ||
* '' | * '' | ||
* '' | * '' | ||
+ | |||
+ | |||
+ | ==== Software Installations and Modules ==== | ||
+ | |||
+ | On VSC-4 and VSC-5, spack is used to install and provide modules, see [[doku: | ||
+ | |||
==== Node configuration - hyperthreading ==== | ==== Node configuration - hyperthreading ==== | ||
- | The compute nodes of VSC-3 are configured with the following parameters in SLURM: | + | The compute nodes of VSC-4 are configured with the following parameters in SLURM: |
< | < | ||
- | CoresPerSocket=8 | + | CoresPerSocket=24 |
Sockets=2 | Sockets=2 | ||
ThreadsPerCore=2 | ThreadsPerCore=2 | ||
</ | </ | ||
- | This reflects the fact that < | + | And the primary nodes of VSC-5 with: |
+ | < | ||
+ | CoresPerSocket=64 | ||
+ | Sockets=2 | ||
+ | ThreadsPerCore=2 | ||
+ | </ | ||
+ | This reflects the fact that < | ||
In the batch script hyperthreading is selected by adding the line | In the batch script hyperthreading is selected by adding the line | ||
< | < | ||
Line 40: | Line 36: | ||
which allows for 2 tasks per core. | which allows for 2 tasks per core. | ||
- | Some codes may experience a performance gain from using all 32 virtual cores, e.g., GROMACS seems to profit. But note that using all virtual cores also leads to more communication and may impact on the performance of large MPI jobs. | + | Some codes may experience a performance gain from using all virtual cores, e.g., GROMACS seems to profit. But note that using all virtual cores also leads to more communication and may impact on the performance of large MPI jobs. |
- | **NOTE on accounting**: | + | **NOTE on accounting**: |
==== Node allocation policy ==== | ==== Node allocation policy ==== | ||
- | On VSC-3 (as on VSC-2) < | + | On VSC-4 there are a set of nodes which accept jobs that do not require entire nodes (anythong from 1 core to less than a full node). These nodes are set up to accomodate different jobs from different users until they are full. They are automatically used for such types of jobs. All other nodes are assigned completely |
+ | On VSC-5 that feature is not yet active, so only complete nodes are assigned | ||
- | ===== The job submission script===== | + | ===== Submit a batch job ===== |
- | It is recommended to write the job script using a [[doku: | + | ==== Partition, quality of service and run time ==== |
+ | |||
+ | Depending on the demands of a certain application, | ||
+ | [[doku: | ||
+ | quality of service (QOS; defining the run time etc.)]] can be selected. | ||
+ | Additionally, | ||
+ | ==== The job submission script==== | ||
+ | |||
+ | It is recommended to write the job script using a [[doku: | ||
Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it is not executed. | Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it is not executed. | ||
Line 59: | Line 64: | ||
#SBATCH -J chk | #SBATCH -J chk | ||
#SBATCH -N 2 | #SBATCH -N 2 | ||
- | #SBATCH --ntasks-per-node=16 | + | #SBATCH --ntasks-per-node=48 |
#SBATCH --ntasks-per-core=1 | #SBATCH --ntasks-per-core=1 | ||
#SBATCH --mail-type=BEGIN | #SBATCH --mail-type=BEGIN | ||
Line 65: | Line 70: | ||
# when srun is used, you need to set: | # when srun is used, you need to set: | ||
- | export I_MPI_PMI_LIBRARY=/ | ||
- | <srun -l -N2 -n32 a.out > | + | <srun -l -N2 -n96 a.out > |
# or | # or | ||
- | <mpirun -np 32 a.out> | + | <mpirun -np 96 a.out> |
</ | </ | ||
* **-J** | * **-J** | ||
Line 81: | Line 85: | ||
* **--mail-user** sends an email to this address | * **--mail-user** sends an email to this address | ||
- | In order to send the job to specific queues, see [[doku:vsc3_queue|Queue/Partition setup on VSC-3]]. | + | In order to send the job to specific queues, see [[doku:vsc4_queue|Queue |
====Job submission==== | ====Job submission==== | ||
< | < | ||
- | [username@l31 ~]$ sbatch check.slrm | + | [username@l42 ~]$ sbatch check.slrm |
- | [username@l31 ~]$ squeue -u `whoami` | + | [username@l42 ~]$ squeue -u `whoami` |
- | [username@l31 ~]$ scancel | + | [username@l42 ~]$ scancel |
# is obtained from the previous command | # is obtained from the previous command | ||
</ | </ | ||
Line 99: | Line 103: | ||
At the moment, it is recommended to use **mpirun** in case of MVAPICH2 and OpenMPI. | At the moment, it is recommended to use **mpirun** in case of MVAPICH2 and OpenMPI. | ||
- | ==== Further examples for job submission scripts ==== | ||
- | + | ==== Hybrid MPI/ | |
- | === Hybrid MPI/OMP: === | + | |
SLURM Script: | SLURM Script: | ||
Line 145: | Line 147: | ||
- | === Job chain === | + | ==== Job chain ==== |
This example is for using a set of 4 nodes to compute a series of jobs in two stages, each of them split into two separate subjobs. \\ | This example is for using a set of 4 nodes to compute a series of jobs in two stages, each of them split into two separate subjobs. \\ | ||
Line 158: | Line 160: | ||
#SBATCH --ntasks-per-core=1 | #SBATCH --ntasks-per-core=1 | ||
- | export I_MPI_PMI_LIBRARY=/ | ||
scontrol show hostnames $SLURM_NODELIST | scontrol show hostnames $SLURM_NODELIST | ||
Line 264: | Line 265: | ||
</ | </ | ||
(The SLURM inherent command //#SBATCH --array starting_value-end_value: | (The SLURM inherent command //#SBATCH --array starting_value-end_value: | ||
+ | |||
+ | [[doku: | ||
===== Generating a host machines file ===== | ===== Generating a host machines file ===== | ||
Line 300: | Line 303: | ||
==== Restarting Failed Jobs ==== | ==== Restarting Failed Jobs ==== | ||
- | Slurm is configured to automatically requeue jobs which were aborted due to node failures. If this is an unwanted behaviour you can prevent automatic requeuing with the following option in your job script: | + | Slurm is __not longer__ |
- | < | + | < |
==== Job Arrays ==== | ==== Job Arrays ==== | ||
Line 350: | Line 352: | ||
</ | </ | ||
+ | ===== Prolog Error Codes ===== | ||
+ | |||
+ | < | ||
+ | ERROR_MEMORY=200 | ||
+ | ERROR_INFINIBAND_HW=201 | ||
+ | ERROR_INFINIBAND_SW=202 | ||
+ | ERROR_IPOIB=203 | ||
+ | ERROR_BEEGFS_SERVICE=204 | ||
+ | ERROR_BEEGFS_USER=205 | ||
+ | ERROR_BEEGFS_SCRATCH=206 | ||
+ | ERROR_NFS=207 | ||
+ | |||
+ | ERROR_USER_GROUP=220 | ||
+ | ERROR_USER_HOME=221 | ||
+ | |||
+ | ERROR_GPFS_START=228 | ||
+ | ERROR_GPFS_MOUNT=229 | ||
+ | ERROR_GPFS_UNMOUNT=230 | ||
+ | |||
+ | </ |