Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
| doku:gromacs [2014/11/04 15:34] – sh | doku:gromacs [2022/06/23 13:24] – msiegel | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ===== GROMACS | + | ====== GROMACS ====== |
| - | ==== Installation VSC-1 (MPI-parallel) | + | |
| - | 1. Make sure we use a recent version of the Intel MPI toolchain. | + | ===== GPU Partition ===== |
| - | < | + | |
| - | mpi-selector --query | + | |
| - | mpi-selector --list | + | |
| - | ... in case | + | |
| - | mpi-selector --set intel_mpi_intel64-4.1.1.036 | + | |
| - | exit | + | |
| - | relogin | + | |
| - | </ | + | |
| - | 2. Follow [[http:// | + | First you have to decide on which hardware GROMACS should run, we call |
| - | and prepare for installation, | + | this a '' |
| - | < | + | '' |
| - | cd /opt/sw/ | + | setup has different hardware, for example |
| - | wget ftp://ftp.gromacs.org/ | + | '' |
| - | gunzip ./ | + | with 2 hyperthreads each core, listed at [[doku:vsc3gpuqos | GPU Partitions on VSC3]]. |
| - | tar xvf ./ | + | The partition has to be set in the batch script, see the example |
| - | gzip gromacs-5.0.1.tar | + | below. Thus here it makes sense to let GROMACS run on 8 threads |
| - | mv ./ | + | ('' |
| - | chown -R root ./ | + | that, as this would lead to oversubscribing. GROMACS decides mostly on |
| - | chgrp -R root ./ | + | its own how it wants to work, so don't be surprised if it ignores |
| - | rm -rf ./ | + | settings like environment variables. |
| - | mkdir ./ | + | |
| - | cd ./ | + | |
| - | </ | + | |
| - | 3. Download the latest version | + | === Batch Script === |
| - | double precision versions of it. For future CUDA compatibility | + | In order to be scheduled efficiently with [[doku: |
| - | '' | + | script'' |
| - | < | + | |
| - | cd / | + | * some SLURM parameters: the '' |
| - | cp from/ | + | |
| - | gunzip | + | * cleaning the environment: |
| - | tar xvf ./fftw-3.3.4.tar | + | |
| - | gzip fftw-3.3.4.tar | + | * last but not least starting the program in question: '' |
| - | cd ./ | + | |
| - | ./configure --help | + | |
| - | export CC=gcc | + | |
| - | export F77=gfortran | + | |
| - | ./configure --prefix=/ | + | |
| - | make clean | + | |
| - | make | + | |
| - | make install | + | |
| - | </ | + | |
| - | 4. Do the CMAKE build, | + | < |
| - | < | + | #!/bin/bash |
| - | cd /opt/sw/ | + | # |
| - | export CMAKE_PREFIX_PATH=/ | + | # |
| - | export FFTW_LOCATION=/ | + | # |
| - | export CC=mpiicc | + | # |
| - | export CXX=mpiicpc | + | |
| - | / | + | |
| - | make | + | |
| - | make install | + | |
| - | </ | + | |
| + | unset OMP_NUM_THREADS | ||
| + | export CUDA_VISIBLE_DEVICES=0 | ||
| + | module purge | ||
| + | module load gcc/7.3 nvidia/1.0 cuda/ | ||
| - | + | gmx_mpi mdrun -s topol.tpr | |
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
| - | ==== GROMACS with gpu support ==== | + | |
| - | + | ||
| - | up to version 4.5.x openmm is needed for gpu support. For versions from 4.6.x on gpu support is included in gromaces source already. | + | |
| - | + | ||
| - | On VSC-1 Gromacs was built with this cmake command (specifying mpiiccc and mpiicpc did not work here, -DGMX_MPI=ON is enough for compiling with MPI support): | + | |
| - | + | ||
| - | < | + | |
| - | + | ||
| - | CC=icc CXX=icpc / | + | |
| </ | </ | ||
| + | Type '' | ||
| + | get the job id, and your job will be scheduled, and executed | ||
| + | automatically. | ||
| - | There might be a problem coming from CUDA, where the intel compiler version is checked. In this case uncomment this part in the / | ||
| - | < | + | === CPU / GPU Load === |
| - | //#if defined(__ICC) | + | |
| - | //#if !(__INTEL_COMPILER == 9999 && __INTEL_COMPILER_BUILD_DATE == 20110811) || !defined(__GNUC__) || !defined(__LP64__) | + | There is a whole page dedicated to [[doku: |
| + | GROMACS the relevant sections are section [[doku: | ||
| - | //#error -- unsupported ICC configuration! Only ICC 12.1 on Linux x86_64 is supported! | + | === Performance === |
| - | //#endif /* !(__INTEL_COMPILER == 9999 && __INTEL_COMPILER_BUILD_DATE == 20110811) || !__GNUC__ || !__LP64__ */ | + | As an example we ran '' |
| + | options, where '' | ||
| + | actually care about the result. Without any options GROMACS already | ||
| + | runs fine (a). Setting the number of tasks (b,c) is not needed; if set | ||
| + | wrong can even slow the calculation down significantly (over | ||
| + | provisioning)! Enforcing pinning also does not show any effects (d), | ||
| + | we assume that the tasks are pinned automatically already. The only | ||
| + | improvement we had was using the '' | ||
| + | load on the GPU. This might not work however if we use more than one | ||
| + | GPU. | ||
| - | //#endif /* __ICC */ | + | ^ # ^ cmd ^ ns / day ^ cpu load / % ^ gpu load / % ^ notes ^ |
| - | </code> | + | | a | -- | 160 | 100 | 80 | | |
| + | | b | -ntomp 8 | 160 | 100 | 80 | | | ||
| + | | c | -ntomp 16 | 140 | 40 | 70 | gromacs warning: over provisioning | | ||
| + | | d | -pin on | 160 | 100 | 80 | | | ||
| + | | e | -update gpu | 170 | 100 | 90 | | | ||