Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision | ||
doku:vsc5quickstart [2022/04/27 08:19] – jz | doku:vsc5quickstart [2023/05/17 15:00] – msiegel | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Quick start guide for VSC-5 ====== | ====== Quick start guide for VSC-5 ====== | ||
- | |||
- | **Status: 2022/04** | ||
- | |||
- | This page is under construction. | ||
===== Connecting ===== | ===== Connecting ===== | ||
Line 14: | Line 10: | ||
< | < | ||
ssh < | ssh < | ||
- | </code. | + | </code> |
===== Data storage ===== | ===== Data storage ===== | ||
Line 23: | Line 19: | ||
- | ===== Loading Modules & Spack Environments | + | ===== Software installations |
- | Different CPUs come with different compilers, so we use | + | ==== New SPACK without environments ==== |
- | the new spack feature '' | + | |
- | package. | + | |
- | On login the default '' | + | Having worked with spack environments for some time, we have encountered several severe issues which have convinced us that we need to find a more practical way of maintaining software packages at VSC. |
- | automatically, so only modules | + | |
- | with '' | + | |
- | On VSC5 no default modules | + | There are now three separate |
- | using '' | + | |
- | Find the official SPACK documentation at https:// | + | * skylake - Intel CPUs; works on Intel Skylake and Cascadelake CPUs |
+ | * zen - AMD CPUs; works on Zen 2 and 3 CPUs | ||
+ | * cuda-zen - AMD CPUs + NVIDIA GPUs; works on all nodes equipped with graphics cards | ||
- | + | By default the spack installation tree suitable for the current compute/ | |
- | ==== List Spack Environments ==== | + | |
- | + | ||
- | Type '' | + | |
- | which one is active. | + | |
< | < | ||
- | $ spack env list | + | zen [user@l51 ~]$ |
- | ==> 2 environments | + | |
- | cascadelake | + | |
</ | </ | ||
- | The current '' | + | Read more about SPACK at: |
- | + | * [[doku:spack-transition | Transition to new SPACK without Environments]] | |
- | < | + | * [[doku:spack]] |
- | (zen3) | + | * [[https://spack.readthedocs.io/ |
- | </ | + | |
- | + | ||
- | Mind that if your prompt is changed later, like when loading a '' | + | |
- | environment'' | + | |
- | not be shown correctly in your prompt. | + | |
- | + | ||
- | When a spack environment is activated, the command '' | + | |
- | environment. | + | |
- | + | ||
- | The command '' | + | |
- | spack environment. | + | |
- | + | ||
- | + | ||
- | ==== Change Spack Environment ==== | + | |
- | + | ||
- | If you want to look for a certain package that belongs to another | + | |
- | architecture, | + | |
- | + | ||
- | < | + | |
- | $ spacktivate < | + | |
- | $ spacktivate cascadelake | + | |
- | </code> | + | |
- | + | ||
- | Only then '' | + | |
- | + | ||
- | + | ||
- | ==== Save Spack Environment ==== | + | |
- | + | ||
- | The following creates a load script for your current spack environment | + | |
- | with all loaded modules: | + | |
- | + | ||
- | < | + | |
- | $ spack env loads -r | + | |
- | </ | + | |
- | This creates a file called '' | ||
- | directory. Sourcing that file in bash will make the environment | ||
- | available to the user. The '' | ||
- | '' | ||
- | environment, | ||
+ | ==== Load a module ==== | ||
- | ==== Load a Module ==== | + | Most software is installed via SPACK, so you can use spack commands like '' |
- | Please always use spack, see [[doku: | + | Some software is still installed by hand, find available modules with '' |
- | HPC systems]]. | + | |
- | ===== Compile | + | ===== Compile |
A program needs to be compiled on the hardware it will later run | A program needs to be compiled on the hardware it will later run | ||
Line 120: | Line 68: | ||
128 physical cores (core-id 0-127) and 256 virtual cores available. | 128 physical cores (core-id 0-127) and 256 virtual cores available. | ||
- | The A100 GPU nodes have 512GB RAM and the NVIDIA A100 cards have 40GB RAM each. | + | The A100 GPU nodes have 512GB RAM and the two NVIDIA A100 cards have 40GB RAM each. |
- | At the moment 40 GPU nodes are installed. | + | 60 A100 nodes are installed. |
+ | The A40 GPU nodes have 256GB RAM and the two NVIDIA A40 cards have 46GB each. | ||
+ | 45 A40 nodes are installed. | ||
< | < | ||
$ nvidia-smi | $ nvidia-smi | ||
Line 155: | Line 105: | ||
===== SLURM ===== | ===== SLURM ===== | ||
- | + | For the partition/queue setup see [[doku: | |
- | The following partitions are currently available: | + | type '' |
- | < | + | |
- | $ sinfo -o %P | + | |
- | PARTITION | + | |
- | gpu_a100_dual* -> Currently | + | |
- | cascadelake_0384 -> Intel CPU nodes with 2x Intel Cascadelake and 384GB RAM | + | |
- | </code> | + | |
- | + | ||
- | + | ||
- | < | + | |
- | $ sinfo | + | |
- | PARTITION | + | |
- | gpu_a100_dual | + | |
- | cascadelake_0384* | + | |
- | </ | + | |
- | + | ||
- | ==== QoS ==== | + | |
- | At the moment two QoS are defined: | + | |
==== Submit a Job ==== | ==== Submit a Job ==== | ||
Line 183: | Line 116: | ||
#SBATCH -J < | #SBATCH -J < | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
- | #SBATCH --gres=gpu: | ||
./ | ./ | ||
</ | </ | ||
- | This will submit a job in the default partition (gpu_a100_dual) using the default QoS (gpu_a100_dual). | + | This will submit a job in the default partition (zen3_0512) using the default QoS (zen3_0512). |
To submit a job to the cascadelake nodes: | To submit a job to the cascadelake nodes: | ||
Line 198: | Line 130: | ||
./ | ./ | ||
</ | </ | ||
+ | |||
+ | Job Scripts for the AMD CPU nodes: | ||
+ | |||
+ | <file sh zen3_0512.sh> | ||
+ | #!/bin/sh | ||
+ | #SBATCH -J < | ||
+ | #SBATCH -N 1 | ||
+ | #SBATCH --partition=zen3_0512 | ||
+ | #SBATCH --qos zen3_0512 | ||
+ | ./ | ||
+ | </ | ||
+ | |||
+ | <file sh zen3_1024.sh> | ||
+ | #!/bin/sh | ||
+ | #SBATCH -J < | ||
+ | #SBATCH -N 1 | ||
+ | #SBATCH --partition=zen3_1024 | ||
+ | #SBATCH --qos zen3_1024 | ||
+ | ./ | ||
+ | </ | ||
+ | |||
+ | <file sh zen3_2048.sh> | ||
+ | #!/bin/sh | ||
+ | #SBATCH -J < | ||
+ | #SBATCH -N 1 | ||
+ | #SBATCH --partition=zen3_2048 | ||
+ | #SBATCH --qos zen3_2048 | ||
+ | ./ | ||
+ | </ | ||
+ | |||
Example job script to use both GPUs on a GPU nodes: | Example job script to use both GPUs on a GPU nodes: | ||
Line 204: | Line 166: | ||
#SBATCH -J < | #SBATCH -J < | ||
#SBATCH -N 1 | #SBATCH -N 1 | ||
- | #SBATCH --partition=gpu_a100_dual | + | #SBATCH --partition=zen3_0512_a100x2 |
- | #SBATCH --qos gpu_100_dual | + | #SBATCH --qos zen3_0512_a100x2 |
#SBATCH --gres=gpu: | #SBATCH --gres=gpu: | ||
./ | ./ | ||
Line 214: | Line 176: | ||
#!/bin/sh | #!/bin/sh | ||
#SBATCH -J < | #SBATCH -J < | ||
- | #SBATCH --partition=gpu_a100_dual | + | #SBATCH --partition=zen3_0512_a100x2 |
- | #SBATCH --qos gpu_100_dual | + | #SBATCH --qos zen3_0512_a100x2 |
#SBATCH --gres=gpu: | #SBATCH --gres=gpu: | ||
./ | ./ | ||
Line 228: | Line 190: | ||
Official Slurm documentation: | Official Slurm documentation: | ||
+ | |||
+ | ===== Intel MPI ===== | ||
+ | |||
+ | When **using Intel-MPI on the AMD nodes and mpirun** please set the following environment variable in your job script to allow for correct process pinning: | ||
+ | |||
+ | < | ||
+ | export I_MPI_PIN_RESPECT_CPUSET=0 | ||
+ | </ | ||