Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
doku:vsc5quickstart [2022/06/24 09:53] – [Submit a Job] jz | doku:vsc5quickstart [2023/05/17 14:54] – msiegel | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Quick start guide for VSC-5 ====== | ====== Quick start guide for VSC-5 ====== | ||
- | |||
- | **Status: 2022/04** | ||
- | |||
- | This page is under construction. | ||
===== Connecting ===== | ===== Connecting ===== | ||
Line 23: | Line 19: | ||
- | ===== Loading Modules & Spack Environments ===== | + | ==== New SPACK without environments |
- | Different CPUs come with different compilers, so we use | + | Having worked |
- | the new spack feature '' | + | |
- | package. | + | |
- | On login the default '' | + | There are now three separate spack installation trees corresponding to the CPU/GPU architectures |
- | automatically, | + | |
- | with '' | + | |
- | On VSC5 no default modules are loaded. Please do that by yourself | + | * skylake - Intel CPUs; works on Intel Skylake and Cascadelake CPUs |
- | using '' | + | * zen - AMD CPUs; works on Zen 2 and 3 CPUs |
+ | * cuda-zen - AMD CPUs + NVIDIA GPUs; works on all nodes equipped with graphics cards | ||
- | Find the official SPACK documentation at https://spack.readthedocs.io/ | + | By default |
- | + | ||
- | + | ||
- | ==== List Spack Environments ==== | + | |
- | + | ||
- | Type '' | + | |
- | which one is active. | + | |
< | < | ||
- | $ spack env list | + | zen [user@l51 ~]$ |
- | ==> 2 environments | + | |
- | cascadelake | + | |
</ | </ | ||
- | The current '' | + | Read more about SPACK at: |
+ | * [[doku:spack-transition | Transition to new SPACK without Environments]] | ||
+ | * [[doku:spack]] | ||
+ | * [[https:// | ||
- | < | ||
- | (zen3) [myname@l55 ~]# | ||
- | </ | ||
- | Mind that if your prompt is changed later, like when loading a '' | + | ==== Load a module |
- | environment'' | + | |
- | not be shown correctly in your prompt. | + | |
- | + | ||
- | When a spack environment is activated, the command '' | + | |
- | environment. | + | |
- | + | ||
- | The command '' | + | |
- | spack environment. | + | |
- | + | ||
- | + | ||
- | ==== Change Spack Environment ==== | + | |
- | + | ||
- | If you want to look for a certain package that belongs to another | + | |
- | architecture, | + | |
- | + | ||
- | < | + | |
- | $ spacktivate < | + | |
- | $ spacktivate cascadelake | + | |
- | </ | + | |
- | + | ||
- | Only then '' | + | |
- | + | ||
- | + | ||
- | ==== Save Spack Environment ==== | + | |
- | + | ||
- | The following creates a load script for your current spack environment | + | |
- | with all loaded modules: | + | |
- | + | ||
- | < | + | |
- | $ spack env loads -r | + | |
- | </ | + | |
- | + | ||
- | This creates a file called '' | + | |
- | directory. Sourcing that file in bash will make the environment | + | |
- | available to the user. The '' | + | |
- | '' | + | |
- | environment, | + | |
- | + | ||
- | + | ||
- | ==== Load a Module | + | |
Please always use spack, see [[doku: | Please always use spack, see [[doku: | ||
HPC systems]]. | HPC systems]]. | ||
+ | |||
+ | Of course '' | ||
- | ===== Compile | + | ===== Compile |
A program needs to be compiled on the hardware it will later run | A program needs to be compiled on the hardware it will later run | ||
Line 120: | Line 67: | ||
128 physical cores (core-id 0-127) and 256 virtual cores available. | 128 physical cores (core-id 0-127) and 256 virtual cores available. | ||
- | The A100 GPU nodes have 512GB RAM and the NVIDIA A100 cards have 40GB RAM each. | + | The A100 GPU nodes have 512GB RAM and the two NVIDIA A100 cards have 40GB RAM each. |
- | At the moment 40 GPU nodes are installed. | + | 60 A100 nodes are installed. |
+ | The A40 GPU nodes have 256GB RAM and the two NVIDIA A40 cards have 46GB each. | ||
+ | 45 A40 nodes are installed. | ||
< | < | ||
$ nvidia-smi | $ nvidia-smi | ||
Line 155: | Line 104: | ||
===== SLURM ===== | ===== SLURM ===== | ||
- | + | For the partition/ | |
- | The following partitions are currently available: | + | type '' |
- | < | + | |
- | $ sinfo -o %P | + | |
- | PARTITION | + | |
- | gpu_a100_dual* -> Currently the default partition. AMD CPU nodes with 2x AMD Epyc (Milan) and 2x NIVIDA A100 and 512GB RAM | + | |
- | cascadelake_0384 -> Intel CPU nodes with 2x Intel Cascadelake and 384GB RAM | + | |
- | zen3_0512 -> AMD CPU nodes with 2x AMD Epyc (Milan) and 512GB RAM | + | |
- | zen3_1024 -> AMD CPU nodes with 2x AMD Epyc (Milan) and 1TB RAM | + | |
- | zen3_2048 -> AMD CPU nodes with 2x AMD Epyc (Milan) and 2TB RAM | + | |
- | </ | + | |
- | + | ||
- | ==== QoS ==== | + | |
- | + | ||
- | During the friendly user test phase the QoS '' | + | |
==== Submit a Job ==== | ==== Submit a Job ==== | ||
Line 179: | Line 115: | ||
#SBATCH -J < | #SBATCH -J < | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
- | #SBATCH --gres=gpu: | ||
./ | ./ | ||
</ | </ | ||
- | This will submit a job in the default partition (gpu_a100_dual) using the default QoS (gpu_a100_dual). | + | This will submit a job in the default partition (zen3_0512) using the default QoS (zen3_0512). |
To submit a job to the cascadelake nodes: | To submit a job to the cascadelake nodes: | ||
Line 191: | Line 126: | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
#SBATCH --partition=cascadelake_0384 | #SBATCH --partition=cascadelake_0384 | ||
- | #SBATCH --qos goodluck | + | #SBATCH --qos cascadelake_0384 |
./ | ./ | ||
</ | </ | ||
Line 202: | Line 137: | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
#SBATCH --partition=zen3_0512 | #SBATCH --partition=zen3_0512 | ||
- | #SBATCH --qos goodluck | + | #SBATCH --qos zen3_0512 |
./ | ./ | ||
</ | </ | ||
Line 211: | Line 146: | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
#SBATCH --partition=zen3_1024 | #SBATCH --partition=zen3_1024 | ||
- | #SBATCH --qos goodluck | + | #SBATCH --qos zen3_1024 |
./ | ./ | ||
</ | </ | ||
Line 220: | Line 155: | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
#SBATCH --partition=zen3_2048 | #SBATCH --partition=zen3_2048 | ||
- | #SBATCH --qos goodluck | + | #SBATCH --qos zen3_2048 |
./ | ./ | ||
</ | </ | ||
Line 230: | Line 165: | ||
#SBATCH -J < | #SBATCH -J < | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
- | #SBATCH --partition=gpu_a100_dual | + | #SBATCH --partition=zen3_0512_a100x2 |
- | #SBATCH --qos goodluck | + | #SBATCH --qos zen3_0512_a100x2 |
#SBATCH --gres=gpu: | #SBATCH --gres=gpu: | ||
./ | ./ | ||
Line 240: | Line 175: | ||
#!/bin/sh | #!/bin/sh | ||
#SBATCH -J < | #SBATCH -J < | ||
- | #SBATCH --partition=gpu_a100_dual | + | #SBATCH --partition=zen3_0512_a100x2 |
- | #SBATCH --qos goodluck | + | #SBATCH --qos zen3_0512_a100x2 |
#SBATCH --gres=gpu: | #SBATCH --gres=gpu: | ||
./ | ./ | ||
Line 257: | Line 192: | ||
===== Intel MPI ===== | ===== Intel MPI ===== | ||
- | When using Intel-MPI and mpirun please set the following environment variable in your job script to allow for correct process pinning: | + | When **using Intel-MPI |
< | < |