Differences

This shows you the differences between two versions of the page.

--- doku:slurm [2023/03/14 12:15] – [Node configuration - hyperthreading] goldenberg
+++ doku:slurm [2024/07/11 09:05] (current) – grokyta
@@ Line 15: / Line 15: @@
 On VSC-4 and VSC-5, spack is used to install and provide modules, see [[doku:spack|SPACK - a package manager for HPC systems]]. The methods described in [[doku:modules]] can still be used for backwards compatibility, but we suggest using spack.
+In order to set environment variables needed for a specific application, the **module** environment may be used:
+  * ''module avail''     lists the **available** Application-Software, Compilers, Parallel-Environment, and Libraries
+  * ''module list''      shows currently loaded package of your session
+  * ''module unload <xyz>'' unload a particular package <xyz> from your session
+  * ''module load <xyz>'' load a particular package <xyz> into your session
+  * ''module display <xyz>'' OR ''module show <xyz>'' show module details such as the full  path  of  the module file and all (or most) of the environment changes the modulefile will make if loaded
+  * ''module purge'' unloads all loaded modulefiles
+== Note: ==
+  - **<xyz>** format corresponds exactly to the output of ''module avail''. Thus, in order to load or unload a selected module, copy and paste exactly the name listed by ''module avail''.\\
+  - a list of ''module load/unload'' directives may also be included in the top part of a job submission script\\
+When all required/intended modules have been loaded, user packages may be compiled as usual.
 ==== Node configuration - hyperthreading ====
@@ Line 29: / Line 42: @@
 ThreadsPerCore=2
 </code>
-This reflects the fact that <html> <font color=#cc3300> hyperthreading </font> </html> is activated on all compute nodes and <html> <font color=#cc3300> 96 cores on VSC4 and 256 cores on VSC5 </font> </html> may be utilized on each node.
+This reflects the fact that <color #cc3300> hyperthreading </color> is activated on all compute nodes and <color #cc3300> 96 cores on VSC4 and 256 cores on VSC5 </color> may be utilized on each node.
 In the batch script hyperthreading is selected by adding the line
-<code>
+<code>#SBATCH --ntasks-per-core=2
-#SBATCH --ntasks-per-core=2
 </code>
 which allows for 2 tasks per core.
@@ Line 41: / Line 53: @@
 ==== Node allocation policy ====
-On VSC-3 (as on VSC-2)  <html> <font color=#cc3300> only complete compute Nodes </font> </html>, i.e., integral multiples of 16 cores, can be allocated for user jobs. If you wish to run many single core jobs, there will be a possibility to schedule them in a smart way exploiting all 16 cpus of one node, please see the [[doku:slurm&#scheduler_script_for_many_single_core_jobs|scheduler script for a series of single core jobs]].
+On VSC-4 & VSC-5 there is a set of nodes that accept jobs that do not require entire exclusive nodes (anything from 1 core to less than a full node). These nodes are set up to accommodate different jobs from different users until they are full. They are automatically used for such types of jobs. All other nodes are assigned completely (and exclusively) to a job whenever the '-N' argument is used.
@@ Line 49: / Line 62: @@
 Depending on the demands of a certain application, the
-[[doku:vsc3_queue|partition (grouping hardware according its type) and
+[[doku:vsc5_queue|partition (grouping hardware according its type) and
 quality of service (QOS; defining the run time etc.)]] can be selected.
-Additionally, the run time of a job can be limited in the job script to a value lower than the runtime limit of the selected QOS. This allows for a process called [[doku:vsc3_queue#backfilling|backfilling]] possibly leading to a <html><font color=#cc3300>shorter waiting time</font></html> in the queue.
+Additionally, the run time of a job can be limited in the job script to a value lower than the runtime limit of the selected QOS. This allows for a process called backfilling possibly leading to a <color #cc3300>shorter waiting time</color> in the queue.
 ==== The job submission script====
-It is recommended to write the job script using a [[doku:win2vsc&#the_job_filetext_editors_on_the_cluster|text editor]] on the VSC //Linux// cluster.
+It is recommended to write the job script using a [[doku:win2vsc&#the_job_filetext_editors_on_the_cluster|text editor]] on the VSC //Linux// cluster or on any Linux/Mac system.
-Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it is not executed.
+Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it cannot be not executed.
 Assume a submission script ''check.slrm''
@@ Line 63: / Line 76: @@
 #SBATCH -J chk
 #SBATCH -N 2
-#SBATCH --ntasks-per-node=16
+#SBATCH --ntasks-per-node=48
 #SBATCH --ntasks-per-core=1
 #SBATCH --mail-type=BEGIN    # first have to state the type of event to occur
@@ Line 70: / Line 83: @@
 # when srun is used, you need to set:
-<srun -l -N2 -n32 a.out >
+<srun -l -N2 -n96 a.out >
 # or
-<mpirun -np 32 a.out>
+<mpirun -np 96 a.out>
 </code>
   * **-J**     job name,\\
-  * **-N**     number of nodes requested (16 cores per node available)\\
+  * **-N**     number of nodes requested\\
   * **-n, --ntasks=<number>** specifies the number of tasks to run,
   * **--ntasks-per-node**     number of processes run in parallel on a single node \\
@@ Line 84: / Line 97: @@
   * **--mail-user** sends an email to this address
-In order to send the job to specific queues, see [[doku:vsc3_queue|Queue/Partition setup on VSC-3]].
+In order to send the job to specific queues, see [[doku:vsc4_queue|Queue | Partition setup on VSC-4]] or [[doku:vsc5_queue|Queue | Partition setup on VSC-5]].
 ====Job submission====
 <code>
-[username@l31 ~]$ sbatch check.slrm    # to submit the job
+[username@l42 ~]$ sbatch check.slrm    # to submit the job
-[username@l31 ~]$ squeue -u `whoami`   # to check the status  of own jobs
+[username@l42 ~]$ squeue -u `whoami`   # to check the status  of own jobs
-[username@l31 ~]$ scancel  JOBID       # for premature removal, where JOBID
+[username@l42 ~]$ scancel  JOBID       # for premature removal, where JOBID
                                        # is obtained from the previous command
 </code>
-====A word on srun and mpirun:====
-Currently (27th March 2015), **srun** only works when the application uses **intel mpi** and is compiled with the **intel compiler**. We will provide compatible versions of MVAPICH2 and OpenMPI in the near future.
-At the moment, it is recommended to use **mpirun** in case of MVAPICH2 and OpenMPI.
 ==== Hybrid MPI/OMP: ====
@@ Line 144: / Line 148: @@
 </code>
 See also the [[https://software.intel.com/sites/products/documentation/hpc/ics/impi/41/lin/Reference_Manual/Environment_Variables_Process_Pinning.htm|Intel Environment Variables]].
@@ Line 156: / Line 161: @@
 #SBATCH -J chk
 #SBATCH -N 4
-#SBATCH --ntasks-per-node=16
+#SBATCH --ntasks-per-node=48
 #SBATCH --ntasks-per-core=1
@@ Line 162: / Line 167: @@
 scontrol show hostnames $SLURM_NODELIST  > ./nodelist
-srun -l -N2 -r0 -n32 job1.scrpt &
+srun -l -N2 -r0 -n96 job1.scrpt &
-srun -l -N2 -r2 -n32 job2.scrpt &
+srun -l -N2 -r2 -n96 job2.scrpt &
 wait
-srun -l -N2 -r2 -n32 job3.scrpt &
+srun -l -N2 -r2 -n96 job3.scrpt &
-srun -l -N2 -r0 -n32 job4.scrpt &
+srun -l -N2 -r0 -n96 job4.scrpt &
 wait
@@ Line 275: / Line 280: @@
 #SBATCH -J par                      # job name
 #SBATCH -N 2                        # number of nodes=2
-#SBATCH --ntasks-per-node=16        # uses all cpus of one node
+#SBATCH --ntasks-per-node=48        # uses all cpus of one node
 #SBATCH --ntasks-per-core=1
 #SBATCH --threads-per-core=1
@@ Line 285: / Line 290: @@
 rm machines_tmp
-tasks_per_node=16         # change number accordingly
+tasks_per_node=48         # change number accordingly
 nodes=2                   # change number accordingly
 for ((line=1; line<=nodes; line++))
@@ Line 337: / Line 342: @@
   - continue at 2. for further dependent jobs
-===== Licenses =====
-Software, that uses a license server, has to be specified upon job submission. A list of all available licensed software for your user can be shown by using the command:
-<code>
-slic
-</code>
-Within the job script add the flags as shown with 'slic', e.g. for using both Matlab and Mathematica:
-<code>
-#SBATCH -L matlab@vsc,mathematica@vsc
-</code>
 ===== Prolog Error Codes =====