Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
Last revisionBoth sides next revision
doku:slurm [2023/03/14 12:34] – [The job submission script] goldenbergdoku:slurm [2024/07/11 08:52] – [Partition, quality of service and run time] grokyta
Line 29: Line 29:
 ThreadsPerCore=2 ThreadsPerCore=2
 </code> </code>
-This reflects the fact that <html> <font color=#cc3300> hyperthreading </font> </html> is activated on all compute nodes and <html> <font color=#cc3300> 96 cores on VSC4 and 256 cores on VSC5 </font> </html> may be utilized on each node. +This reflects the fact that <color #cc3300> hyperthreading </color> is activated on all compute nodes and <color #cc3300> 96 cores on VSC4 and 256 cores on VSC5 </color> may be utilized on each node. 
 In the batch script hyperthreading is selected by adding the line In the batch script hyperthreading is selected by adding the line
-<code>  +<code>#SBATCH --ntasks-per-core=2
-#SBATCH --ntasks-per-core=2+
 </code> </code>
 which allows for 2 tasks per core. which allows for 2 tasks per core.
Line 41: Line 40:
  
 ==== Node allocation policy ==== ==== Node allocation policy ====
-On VSC-4 there are a set of nodes which accept jobs that do not require entire nodes (anythong from 1 core to less than a full node). These nodes are set up to accomodate different jobs from different users until they are full. They are automatically used for such types of jobs. All other nodes are assigned completely to a job.  +On VSC-4 & VSC-5 there is a set of nodes that accept jobs that do not require entire exclusive nodes (anything from 1 core to less than a full node). These nodes are set up to accommodate different jobs from different users until they are full. They are automatically used for such types of jobs. All other nodes are assigned completely (and exclusively) to a job whenever the '-N' argument is used. 
-On VSC-5 that feature is not yet active, so only complete nodes are assigned to jobs.+
  
  
Line 50: Line 49:
  
 Depending on the demands of a certain application, the   Depending on the demands of a certain application, the  
-[[doku:vsc3_queue|partition (grouping hardware according its type) and +[[doku:vsc5_queue|partition (grouping hardware according its type) and 
 quality of service (QOS; defining the run time etc.)]] can be selected. quality of service (QOS; defining the run time etc.)]] can be selected.
-Additionally, the run time of a job can be limited in the job script to a value lower than the runtime limit of the selected QOS. This allows for a process called [[doku:vsc3_queue#backfilling|backfilling]] possibly leading to a <html><font color=#cc3300>shorter waiting time</font></html> in the queue.+Additionally, the run time of a job can be limited in the job script to a value lower than the runtime limit of the selected QOS. This allows for a process called backfilling possibly leading to a <color #cc3300>shorter waiting time</color> in the queue.
 ==== The job submission script==== ==== The job submission script====
  
 It is recommended to write the job script using a [[doku:win2vsc&#the_job_filetext_editors_on_the_cluster|text editor]] on the VSC //Linux// cluster or on any Linux/Mac system.  It is recommended to write the job script using a [[doku:win2vsc&#the_job_filetext_editors_on_the_cluster|text editor]] on the VSC //Linux// cluster or on any Linux/Mac system. 
-Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it is not executed. +Editors in //Windows// may add additional invisible characters to the job file which render it unreadable and, thus, it cannot be not executed. 
  
 Assume a submission script ''check.slrm'' Assume a submission script ''check.slrm''
Line 76: Line 75:
 </code> </code>
   * **-J**     job name,\\    * **-J**     job name,\\ 
-  * **-N**     number of nodes requested (16 cores per node available)\\ +  * **-N**     number of nodes requested\\ 
   * **-n, --ntasks=<number>** specifies the number of tasks to run,         * **-n, --ntasks=<number>** specifies the number of tasks to run,      
   * **--ntasks-per-node**     number of processes run in parallel on a single node \\           * **--ntasks-per-node**     number of processes run in parallel on a single node \\        
Line 89: Line 88:
    
 <code> <code>
-[username@l31 ~]$ sbatch check.slrm    # to submit the job              +[username@l42 ~]$ sbatch check.slrm    # to submit the job              
-[username@l31 ~]$ squeue -u `whoami`   # to check the status  of own jobs +[username@l42 ~]$ squeue -u `whoami`   # to check the status  of own jobs 
-[username@l31 ~]$ scancel  JOBID       # for premature removal, where JOBID+[username@l42 ~]$ scancel  JOBID       # for premature removal, where JOBID
                                        # is obtained from the previous command                                           # is obtained from the previous command   
 </code> </code>
Line 99: Line 98:
  
  
-====A word on srun and mpirun:==== 
-Currently (27th March 2015), **srun** only works when the application uses **intel mpi** and is compiled with the **intel compiler**. We will provide compatible versions of MVAPICH2 and OpenMPI in the near future. 
-At the moment, it is recommended to use **mpirun** in case of MVAPICH2 and OpenMPI. 
  
- 
-==== Hybrid MPI/OMP: ==== 
- 
-SLURM Script: 
-<code> 
-#SBATCH -N 3  
-#SBATCH --ntasks-per-node=2 
-#SBATCH -c 8 
- 
-export OMP_NUM_THREADS=8 
-srun myhybridcode.exe 
-</code> 
- 
-**mpirun** pins processes to cores.  
-At least in the case of pure MPI processes (without any threads) the best performance has been observed with our default pinning (pinning to the physical cpus 0, 1, ..., 15). 
-If you need to use hybrid MPI/openMP, you may have to disable our default pinning including the following line in the job script: 
-<code> 
-unset I_MPI_PIN_PROCESSOR_LIST 
-export I_MPI_PIN_PROCESSOR_LIST=0,8    # configuration for 2 processes / node 
-export I_MPI_PIN_PROCESSOR_LIST=0,4,8,12    #              4 processes / node 
-export I_MPI_PIN_PROCESSOR_LIST=0,2,4,6,8,10,12,14    #    8 processes / node 
-</code> 
-or use the shell script: 
-<code> 
-if [ $PROC_PER_NODE -gt 1 ] 
-then 
- unset I_MPI_PIN_PROCESSOR_LIST 
- if [ $PROC_PER_NODE -eq 2 ] 
- then 
- export I_MPI_PIN_PROCESSOR_LIST=0,8    # configuration for 2 processes / node 
- elif [ $PROC_PER_NODE -eq 4 ] 
- then 
- export I_MPI_PIN_PROCESSOR_LIST=0,4,8,12    #              4 processes / node 
- elif [ $PROC_PER_NODE -eq 8 ] 
- then 
- export I_MPI_PIN_PROCESSOR_LIST=0,2,4,6,8,10,12,14    #    8 processes / node 
- else 
- export I_MPI_PIN=disable 
- fi 
-fi 
-</code> 
-See also the [[https://software.intel.com/sites/products/documentation/hpc/ics/impi/41/lin/Reference_Manual/Environment_Variables_Process_Pinning.htm|Intel Environment Variables]]. 
  
  
Line 157: Line 111:
 #SBATCH -J chk #SBATCH -J chk
 #SBATCH -N 4 #SBATCH -N 4
-#SBATCH --ntasks-per-node=16+#SBATCH --ntasks-per-node=48
 #SBATCH --ntasks-per-core=1 #SBATCH --ntasks-per-core=1
  
Line 163: Line 117:
 scontrol show hostnames $SLURM_NODELIST  > ./nodelist scontrol show hostnames $SLURM_NODELIST  > ./nodelist
  
-srun -l -N2 -r0 -n32 job1.scrpt & +srun -l -N2 -r0 -n96 job1.scrpt & 
-srun -l -N2 -r2 -n32 job2.scrpt &+srun -l -N2 -r2 -n96 job2.scrpt &
 wait wait
  
-srun -l -N2 -r2 -n32 job3.scrpt & +srun -l -N2 -r2 -n96 job3.scrpt & 
-srun -l -N2 -r0 -n32 job4.scrpt &+srun -l -N2 -r0 -n96 job4.scrpt &
 wait wait
  
Line 276: Line 230:
 #SBATCH -J par                      # job name #SBATCH -J par                      # job name
 #SBATCH -N 2                        # number of nodes=2 #SBATCH -N 2                        # number of nodes=2
-#SBATCH --ntasks-per-node=16        # uses all cpus of one node      +#SBATCH --ntasks-per-node=48        # uses all cpus of one node      
 #SBATCH --ntasks-per-core=1 #SBATCH --ntasks-per-core=1
 #SBATCH --threads-per-core=1 #SBATCH --threads-per-core=1
Line 286: Line 240:
 rm machines_tmp rm machines_tmp
  
-tasks_per_node=16         # change number accordingly+tasks_per_node=48         # change number accordingly
 nodes=2                   # change number accordingly nodes=2                   # change number accordingly
 for ((line=1; line<=nodes; line++)) for ((line=1; line<=nodes; line++))
Line 338: Line 292:
   - continue at 2. for further dependent jobs   - continue at 2. for further dependent jobs
  
-===== Licenses ===== 
  
-Software, that uses a license server, has to be specified upon job submission. A list of all available licensed software for your user can be shown by using the command: 
- 
-<code> 
-slic 
-</code> 
- 
-Within the job script add the flags as shown with 'slic', e.g. for using both Matlab and Mathematica: 
- 
-<code> 
-#SBATCH -L matlab@vsc,mathematica@vsc 
-</code> 
  
 ===== Prolog Error Codes ===== ===== Prolog Error Codes =====
  • doku/slurm.txt
  • Last modified: 2024/07/11 09:05
  • by grokyta