Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
| doku:gromacs [2014/11/04 15:35] – sh | doku:gromacs [2022/07/15 11:35] – added recommendations msiegel | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ===== GROMACS | + | ====== GROMACS ====== |
| - | ==== Installation VSC-1 (MPI-parallel) | + | |
| - | 1. Make sure we use a recent version of the Intel MPI toolchain. | + | Our recommendation: |
| - | < | + | |
| - | mpi-selector | + | |
| - | mpi-selector --list | + | |
| - | ... in case | + | |
| - | mpi-selector | + | |
| - | exit | + | |
| - | relogin | + | |
| - | </ | + | |
| - | 2. Follow [[http:// | + | |
| - | and prepare for installation, | + | - Use the newest Hardware: the partitions '' |
| - | < | + | - Read our article on multi GPU setup and do some performance analysis. |
| - | cd /opt/sw/ | + | - Run on multiple nodes with MPI; each with 1 GPU |
| - | wget ftp:// | + | - Additionally use multiple GPUs per node |
| - | gunzip ./gromacs-5.0.1.tar.gz | + | |
| - | tar xvf ./gromacs-5.0.1.tar | + | |
| - | gzip gromacs-5.0.1.tar | + | |
| - | mv ./ | + | |
| - | chown -R root ./ | + | |
| - | chgrp -R root ./ | + | |
| - | rm -rf ./ | + | |
| - | mkdir ./gromacs-5.0.1_mpi_build | + | |
| - | cd ./ | + | |
| - | </ | + | |
| - | 3. Download the latest version of '' | ||
| - | double precision versions of it. For future CUDA compatibility | ||
| - | '' | ||
| - | < | ||
| - | cd / | ||
| - | cp from/ | ||
| - | gunzip ./ | ||
| - | tar xvf ./ | ||
| - | gzip fftw-3.3.4.tar | ||
| - | cd ./ | ||
| - | ./configure --help | ||
| - | export CC=gcc | ||
| - | export F77=gfortran | ||
| - | ./configure --prefix=/ | ||
| - | make clean | ||
| - | make | ||
| - | make install | ||
| - | </ | ||
| - | 4. Do the CMAKE build, | + | ===== GPU Partition |
| - | < | + | |
| - | cd / | + | |
| - | export CMAKE_PREFIX_PATH=/ | + | |
| - | export FFTW_LOCATION=/ | + | |
| - | export CC=mpiicc | + | |
| - | export CXX=mpiicpc | + | |
| - | / | + | |
| - | make | + | |
| - | make install | + | |
| - | </ | + | |
| + | First you have to decide on which hardware GROMACS should run, we call | ||
| + | this a '' | ||
| + | SLURM]]. On any login node, type '' | ||
| + | available partitions. The partition has to be set in the batch script, | ||
| + | see the example below. Be aware that each partition has different | ||
| + | hardware, for example the partition '' | ||
| + | 1 GPU and a single socket à 4 cores, with 2 hyperthreads each core, | ||
| + | listed at [[doku: | ||
| + | makes sense to let GROMACS run on 8 threads ('' | ||
| + | makes little sense to force more threads than that, as this would lead | ||
| + | to oversubscribing. GROMACS decides mostly on its own how it wants to | ||
| + | work, so don't be surprised if it ignores settings like environment | ||
| + | variables. | ||
| + | ===== Batch Script ===== | ||
| + | Write a '' | ||
| + | |||
| + | * some SLURM parameters: the ''# | ||
| + | * exporting environment variables: e.g. '' | ||
| + | * cleaning the environment: | ||
| + | * loading modules: '' | ||
| + | * last but not least starting the program in question: '' | ||
| + | <code bash mybatchscript.sh> | ||
| + | #!/bin/bash | ||
| + | #SBATCH --job-name=myname | ||
| + | #SBATCH --partition=gpu_gtx1080single | ||
| + | #SBATCH --gres=gpu: | ||
| + | #SBATCH --nodes=1 | ||
| + | unset OMP_NUM_THREADS | ||
| + | export CUDA_VISIBLE_DEVICES=0 | ||
| + | module purge | ||
| + | module load gcc/7.3 nvidia/1.0 cuda/ | ||
| + | gmx_mpi mdrun -s topol.tpr | ||
| + | </ | ||
| - | ==== GROMACS with gpu support ==== | + | Type '' |
| + | [[doku: | ||
| + | executed automatically. | ||
| - | up to version 4.5.x openmm is needed for gpu support. For versions from 4.6.x on gpu support is included in gromaces source already. | ||
| - | On VSC-1 Gromacs was built with this cmake command (specifying mpiiccc and mpiicpc did not work here, -DGMX_MPI=ON is enough for compiling with MPI support): | + | ===== CPU / GPU Load ===== |
| - | < | + | There is a whole page dedicated to [[doku: |
| + | GPU, for GROMACS the relevant sections are section | ||
| + | [[doku: | ||
| - | CC=icc CXX=icpc / | ||
| - | </ | ||
| + | ===== Performance ===== | ||
| - | There might be a problem coming from CUDA, where the intel compiler version is checked. In this case uncomment this part in the / | + | There is a whole article about the [[doku: |
| - | < | + | As a short example we ran '' |
| - | //#if defined(__ICC) | + | different options, where '' |
| + | we don't actually care about the result. Without any options GROMACS | ||
| + | already runs fine (a). Setting the number of tasks (b) is not needed; | ||
| + | if set wrong can even slow the calculation down significantly ( c ) due | ||
| + | to over provisioning! We would advice to enforce pinning, in our | ||
| + | example it does not show any effects though (d), we assume that the | ||
| + | tasks are pinned automatically already. The only further improvement | ||
| + | we could get was using the '' | ||
| + | load on the GPU (e). | ||
| - | //#if !(__INTEL_COMPILER == 9999 && __INTEL_COMPILER_BUILD_DATE == 20110811) | + | ^ # ^ cmd ^ ns / day ^ cpu load / % ^ gpu load / % ^ notes ^ |
| + | | a | -- | 160 | 100 | 80 | | | ||
| + | | b | -ntomp 8 | 160 | 100 | 80 | | | ||
| + | | c | -ntomp 16 | 140 | 40 | 70 | gromacs warning: over provisioning | | ||
| + | | d | -pin on | 160 | 100 | 80 | | | ||
| + | | e | -update gpu | 170 | ||
| - | //#error -- unsupported ICC configuration! Only ICC 12.1 on Linux x86_64 is supported! | ||
| - | //#endif /* !(__INTEL_COMPILER | + | ==== GROMACS2020 ==== |
| - | //#endif /* __ICC */ | + | The following environment variables need to be set with Gromacs2020 |
| - | </ | + | when using multiple GPUs: It is not necessary to set these variables |
| + | for Gromacs2021 onwards; they are already included and setting them | ||
| + | explicitly might actually decrease performance again. | ||
| + | <code bash> | ||
| + | export GMX_GPU_PME_PP_COMMS=true | ||
| + | export GMX_GPU_DD_COMMS=true | ||
| + | export GMX_GPU_FORCE_UPDATE_DEFAULT_GPU=true | ||
| + | </ | ||