Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
doku:vsc5_queue [2023/04/07 16:03] – msiegeldoku:vsc5_queue [2024/04/25 13:17] (current) – [Partitions] goldenberg
Line 1: Line 1:
-====== Queue | Partition setup on VSC-5 ====== +====== Queue | Partition | QOS setup on VSC-5 ====== 
-On VSC-5, the type of hardware and the quality of service (QOS) where the jobs run on may be selected. Nodes of the same type of hardware are grouped to partitions, the QOS defines the maximum run time of a job and the number and type of allocate-able nodes.  +On VSC-5, Nodes of the same type of hardware are grouped to partitions. The quality of service (QOS), former calle //Queue// defines the maximum run time of a job and the number and type of allocate-able nodes.
- +
-===== Partitions ===== +
-There are three basic types of hardware that differ in architecture: +
-  * Intel CPU nodes: there is only one variant with Cascadelake CPUs and 368GB RAM. +
-  * AMD CPU nodes: they all have Zen3 CPU nodes, but come in three memeory versions - 512GB, 1TB and 2TB RAM. +
-  * GPU nodes: there are two versions, one with Zen2 CPUs, 256GB RAM and 2x nVidia A40 GPUs, and one with Zen3 CPUs, 512GB RAM and 2x nVidia A100 GPUs. +
- +
-On VSC-5, the hardware is grouped into so-called <html><font color=#cc3300>&#x27A0; partitions</font></html>, type ''sinfo -o %P'' to see the available partitions: +
- +
-^ Partition ^ Nodes ^ Architecture ^ CPU ^ GPU ^ RAM ^ Use ^ +
-| zen3_0512* || AMD | AMD 7713 (64 Core/CPU) | No | 512 GB | The default partition | +
-| zen3_1024 || AMD | AMD 7713 (64 Core/CPU) | No | 1 TB | High Memory partition | +
-| zen3_2048 || AMD | AMD 7713 (64 Core/CPU) | No | 2 TB | Higher Memory partition | +
-| cascadelake_0384 | | Intel | 2x Intel Cascadelake | No | 384 GB | Directly use programs compiled for VSC-4 | +
-| zen2_0256_a40x2 | 45 | AMD | 2x AMD 7252 (8 Core/CPU) | 2x NIVIDA A40 | 256 GB | Best for single precision GPU code | +
-| zen3_0512_a100x2 | 60 | AMD | 2x AMD 7713 (64 Core/CPU) | 2x NIVIDA A100 | 512 GB | Best for double precision GPU code | +
- +
-Internally used special partitions, that can not be selected manually: +
- +
-^ Partition ^ Description ^ +
-| login5 | login nodes, not an actual slurm partition | +
-| jupyter | reserved for the jupyterhub | +
- +
- +
-===== Quality of service (QOS) ===== +
- +
-Access to node partitions is granted by the so-called <html><font color=#cc3300>&#x27A0; quality of service (QOS)</font></html>. The QOSs constrain the number of allocate-able nodes and limit job wall time. The naming scheme of the QOSs is: +
-<project_type>_<memoryConfig>+
  
 For submitting jobs to [[doku:slurm]], three parameters are important: For submitting jobs to [[doku:slurm]], three parameters are important:
Line 41: Line 13:
  
   * Core hours will be charged to the specified account.   * Core hours will be charged to the specified account.
-  * Note that account, partition, and qos have to fit together+  * Account, partition, and qos have to fit together
   * If the account is not given, the default account will be used.   * If the account is not given, the default account will be used.
-  * If partition and qos are not given, default values are ''zen3_0512'' for both.+  * If partition and QOS are not given, default values are ''zen3_0512'' for both. 
 + 
 + 
 +===== Partitions ===== 
 + 
 +Nodes of the same type of hardware are grouped to partitions, there are three basic types: 
 +  * Intel CPU nodes: There is only one variant with Cascadelake CPUs and 368GB RAM. 
 +  * AMD CPU nodes: They all have Zen3 CPU nodes, but come in three memory versions - 512GB, 1TB and 2TB RAM. 
 +  * GPU nodes: there are two versions, one with Zen2 CPUs, 256GB RAM and 2x nVidia A40 GPUs, and one with Zen3 CPUs, 512GB RAM and 2x nVidia A100 GPUs. 
 + 
 +These are the partitions on VSC-5: 
 + 
 +^ Partition ^ Nodes ^ Architecture ^ CPU ^ Cores per CPU (physical/with HT) ^ GPU ^ RAM ^ Use ^ 
 +| zen3_0512* | 564 | AMD | 2x AMD 7713 | 64/128 | No | 512 GB | The default partition | 
 +| zen3_1024 | 120 | AMD | 2x AMD 7713 | 64/128 | No | 1 TB | High Memory partition | 
 +| zen3_2048 | 20 | AMD | 2x AMD 7713 | 64/128 | No | 2 TB | Higher Memory partition | 
 +| cascadelake_0384 | 48 | Intel | 2x Intel Cascadelake | 48/96 | No | 384 GB | Directly use programs compiled for VSC-4 | 
 +| zen2_0256_a40x2 | 45 | AMD | 2x AMD 7252 | 8/16 | 2x NIVIDA A40 | 256 GB | Best for single precision GPU code | 
 +| zen3_0512_a100x2 | 60 | AMD | 2x AMD 7713 | 64/128 | 2x NIVIDA A100 | 512 GB | Best for double precision GPU code | 
 + 
 +Type ''sinfo -o %P'' on any node to see all the available partitions. 
 + 
 +For the sake of completeness there are internally used //special// partitions, that can not be selected manually: 
 + 
 +^ Partition ^ Description ^ 
 +| login5 | login nodes, not an actual slurm partition | 
 +| rackws5 | GUI login nodes, not an actual slurm partition | 
 +| _jupyter | variations of zen3, a40 and a100 nodes reserved for the jupyterhub | 
 + 
 + 
 +===== Quality of service (QOS) ===== 
 + 
 +The QOS defines the maximum run time of a job and the number and type of allocate-able nodes.
  
 The QOSs that are assigned to a specific user can be viewed with: The QOSs that are assigned to a specific user can be viewed with:
 +
 <code> <code>
 sacctmgr show user `id -u` withassoc format=user,defaultaccount,account,qos%40s,defaultqos%20s sacctmgr show user `id -u` withassoc format=user,defaultaccount,account,qos%40s,defaultqos%20s
 </code> </code>
-The default QOS and all QOSs usable are also shown right after login.+ 
 +All QOS usable are also shown right after login.
  
  
 ==== QOS, Partitions and Run time limits ==== ==== QOS, Partitions and Run time limits ====
  
-The following QoS are available for normal (=non private) projects:+The following QoS are available for all normal (=non private) projects:
  
 ^ QOS name ^ Gives access to Partition ^ Hard run time limits  ^ Description ^ ^ QOS name ^ Gives access to Partition ^ Hard run time limits  ^ Description ^
-|zen3_0512 | zen3_0512 | 72h (3 days) | default | +|zen3_0512 | zen3_0512 | 72h (3 days) | Default | 
-|zen3_1024 | zen3_1024 | 72h (3 days) | +|zen3_1024 | zen3_1024 | 72h (3 days) | High Memory | 
-|zen3_2048 | zen3_2048 | 72h (3 days) |+|zen3_2048 | zen3_2048 | 72h (3 days) | Higher Memory |
 |cascadelake_0384 | cascadelake_0384 | 72h (3 days) | |cascadelake_0384 | cascadelake_0384 | 72h (3 days) |
-|zen2_0256_a40x2 | zen2_0256_a40x2 | 72h (3 days) | +|zen2_0256_a40x2 | zen2_0256_a40x2 | 72h (3 days) | GPU Nodes | 
-|zen3_0512_a100x2 | zen3_0512_a100x2 | 72h (3 days) | +|zen3_0512_a100x2 | zen3_0512_a100x2 | 72h (3 days) | GPU Nodes | 
-|zen3_0512_devel | 5 nodes on zen3_0512 | 10min |+|zen3_0512_devel | 5 nodes on zen3_0512 | 10min | Fast Feedback |
  
  
 ==== Idle QOS ==== ==== Idle QOS ====
  
-Generally, it can be distinguished in QOS defined on the generally available compute nodes and on private nodes. Furthermore, there is a distinction whether a project still has available computing time or if the computing time has already been consumed. In the latter case, jobs of this project are running with low job priority and reduced maximum run time limit in the <html><font color=#cc3300>&#x27A0; idle queue</font></html>. There is no idle on cascadelake or GPUs.+If a project runs out of compute time, jobs of this project are now running with low job priority and reduced maximum run time limit in the //idle// QOS.
  
-^ QOS name ^ Gives access to Partition ^ Hard run time limits ^  +There is no //idle// QOS on ''cascadelake'' or ''zen2_0256_a40x2''/''zen3_0512_a100x2'' GPU Nodes.
-| idle_0512 | zen3_0512 | 24h (1 day) | projects out of compute time |  +
-| idle_1024 | zen3_1024 | 24h (1 day) | projects out of compute time |  +
-| idle_2048 | zen3_2048 | 24h (1 day) | projects out of compute time | +
  
 +^ QOS name ^ Gives access to Partition ^ Hard run time limits ^  Description ^
 +| idle_0512 | zen3_0512 | 24h (1 day) | Projects out of compute time | 
 +| idle_1024 | zen3_1024 | 24h (1 day) | Projects out of compute time | 
 +| idle_2048 | zen3_2048 | 24h (1 day) | Projects out of compute time | 
  
-==== zen3_0512_devel ==== 
  
-The <html><font color=#cc3300>&#x27A0; devel queue</font></html> gives fast feed-back to the user if her or his job is running. It is possible to connect to the node where the actual job is running and to directly [[doku:monitoring|monitor]] the job, e.g., for the purpose of checking if the threads/processes are doing what is expected. This might be recommended before sending the job to one of the 'computing' queues.+==== Devel QOS ==== 
 + 
 +The //devel// QOS gives fast feedback to the user when their job is running. Connect to the node where the actual job is running to directly [[doku:monitoring|monitor]] to check if the threads/processes are doing what you expect. We recommend this before sending the job to one of the ''compute'' queues.
  
 ^ QOS name ^ Gives access to Partition ^ Hard run time limits  ^ ^ QOS name ^ Gives access to Partition ^ Hard run time limits  ^
 |zen3_0512_devel | 5 nodes on zen3_0512 | 10min | |zen3_0512_devel | 5 nodes on zen3_0512 | 10min |
  
-==== Private queues ====+==== Private Projects ====
  
-Private projects come with different QoS; still partition, qos, and account have to fit together.+Private projects come with different QOS; nevertheless partition, QOS, and account have to fit together.
  
 ^ QOS name ^ Gives access to Partition ^ Hard run time limits  ^  ^ QOS name ^ Gives access to Partition ^ Hard run time limits  ^ 
-| private queues p....._0...  | various | up to 240h (10 days) |+| p....._0...  | various | up to 240h (10 days) | private queues |
  
 For submitting jobs to [[doku:slurm]], three parameters are important: For submitting jobs to [[doku:slurm]], three parameters are important:
  
 <code bash> <code bash>
-#SBATCH --account=p7xxxx  +#SBATCH --account=pxxxxx  
-#SBATCH --partition=zen3_0512 +#SBATCH --partition=zen3_xxxx 
-#SBATCH --qos=p7xxx_xxxx+#SBATCH --qos=pxxxx_xxxx
 </code>  </code> 
  
  
-==== Run time limits ====+==== Run time ====
  
 The QOS's run time limits can also be requested via the command The QOS's run time limits can also be requested via the command
Line 105: Line 113:
 <code>sacctmgr show qos  format=name%20s,priority,grpnodes,maxwall,description%40s</code> <code>sacctmgr show qos  format=name%20s,priority,grpnodes,maxwall,description%40s</code>
  
-SLURM allows for setting a run time limit //below// the default QOS's run time limit. After the specified time is elapsed, the job is killed:+If you know how long your job usually runs, you can set the run time limit in SLURM: 
 + 
 +<code> 
 +#SBATCH --time=<time> 
 +</code>
  
-<code>#SBATCH --time=<time> </code>+ Of course this has to be //below// the default QOS's run time limit. Your job might start earlier, which is nice; But after the specified time is elapsed, the job is killed!
  
-Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".+Acceptable time formats include: 
 +  * "minutes" 
 +  * "minutes:seconds" 
 +  * "hours:minutes:seconds" 
 +  * "days-hours" 
 +  * "days-hours:minutes" 
 +  * "days-hours:minutes:seconds".
  
  • doku/vsc5_queue.1680883406.txt.gz
  • Last modified: 2023/04/07 16:03
  • by msiegel