Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision | Next revision Both sides next revision | ||
doku:gromacs [2022/06/23 13:24] msiegel |
doku:gromacs [2022/07/15 11:35] msiegel added recommendations |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== GROMACS ====== | ====== GROMACS ====== | ||
+ | |||
+ | Our recommendation: | ||
+ | |||
+ | - Use the most recent version of GROMACS that we provide or build your own. | ||
+ | - Use the newest Hardware: the partitions '' | ||
+ | - Read our article on multi GPU setup and do some performance analysis. | ||
+ | - Run on multiple nodes with MPI; each with 1 GPU | ||
+ | - Additionally use multiple GPUs per node | ||
+ | |||
===== GPU Partition ===== | ===== GPU Partition ===== | ||
First you have to decide on which hardware GROMACS should run, we call | First you have to decide on which hardware GROMACS should run, we call | ||
- | this a '' | + | this a '' |
- | '' | + | SLURM]]. On any login node, type '' |
- | setup has different hardware, for example the partition | + | available partitions. The partition has to be set in the batch script, |
- | '' | + | see the example below. Be aware that each partition |
- | with 2 hyperthreads each core, listed at [[doku: | + | hardware, for example the partition '' |
- | The partition has to be set in the batch script, see the example | + | 1 GPU and a single socket à 4 cores, with 2 hyperthreads each core, |
- | below. | + | listed at [[doku: |
- | ('' | + | makes sense to let GROMACS run on 8 threads ('' |
- | that, as this would lead to oversubscribing. GROMACS decides mostly on | + | makes little sense to force more threads than that, as this would lead |
- | its own how it wants to work, so don't be surprised if it ignores | + | to oversubscribing. GROMACS decides mostly on its own how it wants to |
- | settings like environment variables. | + | work, so don't be surprised if it ignores settings like environment |
+ | variables. | ||
+ | |||
+ | ===== Batch Script ===== | ||
- | === Batch Script === | + | Write a '' |
- | In order to be scheduled efficiently with [[doku: | + | |
- | script'' | + | |
* some SLURM parameters: the ''# | * some SLURM parameters: the ''# | ||
- | * exporting environment variables: '' | + | * exporting environment variables: |
* cleaning the environment: | * cleaning the environment: | ||
* loading modules: '' | * loading modules: '' | ||
* last but not least starting the program in question: '' | * last but not least starting the program in question: '' | ||
- | < | + | < |
#!/bin/bash | #!/bin/bash | ||
#SBATCH --job-name=myname | #SBATCH --job-name=myname | ||
Line 42: | Line 52: | ||
</ | </ | ||
- | Type '' | + | Type '' |
- | get the job id, and your job will be scheduled, and executed | + | [[doku: |
- | automatically. | + | executed |
- | === CPU / GPU Load === | + | ===== CPU / GPU Load ===== |
- | There is a whole page dedicated to [[doku: | + | There is a whole page dedicated to [[doku: |
- | GROMACS the relevant sections are section [[doku: | + | GPU, for GROMACS the relevant sections are section |
+ | [[doku: | ||
- | === Performance === | ||
- | As an example we ran '' | + | ===== Performance ===== |
- | options, where '' | + | |
- | actually care about the result. Without any options GROMACS already | + | There is a whole article about the [[doku: |
- | runs fine (a). Setting the number of tasks (b,c) is not needed; if set | + | |
- | wrong can even slow the calculation down significantly (over | + | As a short example we ran '' |
- | provisioning)! Enforcing | + | different |
- | we assume that the tasks are pinned automatically already. The only | + | we don't actually care about the result. Without any options GROMACS |
- | improvement we had was using the '' | + | already runs fine (a). Setting the number of tasks (b) is not needed; |
- | load on the GPU. This might not work however if we use more than one | + | if set wrong can even slow the calculation down significantly ( c ) due |
- | GPU. | + | to over provisioning! |
+ | example it does not show any effects | ||
+ | tasks are pinned automatically already. The only further | ||
+ | we could get was using the '' | ||
+ | load on the GPU (e). | ||
^ # ^ cmd ^ ns / day ^ cpu load / % ^ gpu load / % ^ notes ^ | ^ # ^ cmd ^ ns / day ^ cpu load / % ^ gpu load / % ^ notes ^ | ||
Line 72: | Line 86: | ||
| e | -update gpu | 170 | 100 | 90 | | | | e | -update gpu | 170 | 100 | 90 | | | ||
+ | |||
+ | ==== GROMACS2020 ==== | ||
+ | |||
+ | The following environment variables need to be set with Gromacs2020 | ||
+ | when using multiple GPUs: It is not necessary to set these variables | ||
+ | for Gromacs2021 onwards; they are already included and setting them | ||
+ | explicitly might actually decrease performance again. | ||
+ | |||
+ | <code bash> | ||
+ | export GMX_GPU_PME_PP_COMMS=true | ||
+ | export GMX_GPU_DD_COMMS=true | ||
+ | export GMX_GPU_FORCE_UPDATE_DEFAULT_GPU=true | ||
+ | </ |