===== LAMMPS ===== Installation is "straightforward": 1. Cluster specific settings/files: * VSC-1: * mpi-selector: qlogicmpi_intel-0.1.0 * Makefile Oct 2012 version: {{:doku:lammps:makefile_20oct12.vsc1}} * Makefile Sep 2013 version: {{:doku:lammps:makefile_9sep13.vsc1}} * mpi-selector: intel_mpi_intel64-4.1.1.036 * Makefile Feb 2014 version: {{:doku:lammps:makefile_1Feb14.vsc1}} * Makefile Jun 2014 version:{{:doku:lammps:makefile_28jun14.vsc1|}} * VSC-2: * mpi-selector: intel_mpi_intel64-4.0.3.008 * Makefile Oct 2012 version: {{:doku:lammps:makefile_20oct12.vsc2}} * Makefile Feb 2013 version: {{:doku:lammps:makefile_3feb13.vsc2}} * Makefile Jul 2013 version: {{:doku:lammps:Makefile_8Jul13.vsc2}} 2. Fix the KIM_API first. It is an additional package that will be included in the standard build process later on. Watch out for the special patch to make it working with INTEL compilers. On VSC-1 su cd /opt/src/openkim wget http://s3.openkim.org/kim-api/kim-api-v1.6.3.tgz gunzip kim-api-v1.6.3.tgz tar xvf kim-api-v1.6.3.tar cd kim-api-v1.6.3 cp Makefile.KIM_Config.example Makefile.KIM_Config vi Makefile.KIM_Config KIM_DIR = /opt/src/openkim/kim-api-v1.6.3 KIM_COMPILERSUITE = INTEL KIM_SYSTEMLINKER = linux KIM_SYSTEMARCH = 64bit KIM_LINK = dynamic-load prefix = /opt/sw/openkim-api/1.6.3 vi build_system/compiler_defaults/Makefile.INTEL ( check general settings ) ... replace the Makefile in 'src/utils/Makefile' with the one from https://groups.google.com/forum/#!topic/openkim/XOTVFVYVw5o make clean make ( ignore all warnings regarding unresolved libifport.so... ) make install make install-set-default-to-v1 3. Download source code from [[http://lammps.sandia.gov/download.html#tar|lammps homepage]] and confirm usage of intel_mpi_intel64-4.1.1.036: cd /opt/src/lammps/sh_vii/ cp /from/wherever/it/is/lammps-28Jun14.tar.gz ./ gunzip lammps-28Jun14.tar.gz tar xvf lammps-28Jun14.tar gzip lammps-28Jun14.tar cd lammps-28Jun14 4. Apply all the available patches known at the time of installation, ie wget http://lammps.sandia.gov/patches/patch.11Jul14 patch -bp1 < patch.11Jul14 wget http://lammps.sandia.gov/patches/patch.22Jul14 patch -bp1 < patch.22Jul14 wget http://lammps.sandia.gov/patches/patch.29Jul14 patch -bp1 < patch.29Jul14 wget http://lammps.sandia.gov/patches/patch.1Aug14 patch -bp1 < patch.1Aug14 wget http://lammps.sandia.gov/patches/patch.6Aug14 patch -bp1 < patch.6Aug14 wget http://lammps.sandia.gov/patches/patch.8Aug14 patch -bp1 < patch.8Aug14 wget http://lammps.sandia.gov/patches/patch.15Aug14 patch -bp1 < patch.15Aug14 wget http://lammps.sandia.gov/patches/patch.27Aug14 patch -bp1 < patch.27Aug14 wget http://lammps.sandia.gov/patches/patch.28Aug14 patch -bp1 < patch.28Aug14 wget http://lammps.sandia.gov/patches/patch.29Aug14 patch -bp1 < patch.29Aug14 wget http://lammps.sandia.gov/patches/patch.5Sep14 patch -bp1 < patch.5Sep14 wget http://lammps.sandia.gov/patches/patch.8Sep14 patch -bp1 < patch.8Sep14 wget http://lammps.sandia.gov/patches/patch.11Sep14 patch -bp1 < patch.11Sep14 wget http://lammps.sandia.gov/patches/patch.18Sep14 patch -bp1 < patch.18Sep14 wget http://lammps.sandia.gov/patches/patch.25Sep14 patch -bp1 < patch.25Sep14 wget http://lammps.sandia.gov/patches/patch.30Sep14 patch -bp1 < patch.30Sep14 wget http://lammps.sandia.gov/patches/patch.7Oct14 patch -bp1 < patch.7Oct14 wget http://lammps.sandia.gov/patches/patch.8Oct14 patch -bp1 < patch.8Oct14 wget http://lammps.sandia.gov/patches/patch.9Oct14 patch -bp1 < patch.9Oct14 wget http://lammps.sandia.gov/patches/patch.21Oct14 patch -bp1 < patch.21Oct14 5. Activate most recent CUDA toolchain export PATH=/opt/sw/cuda-6.5/bin:$PATH export LD_LIBRARY_PATH=/opt/sw/cuda-6.5/lib64:$LD_LIBRARY_PATH 6. Compile individual optional packages in ./lib regardless of whether they will be used for the actual build of the LAMMPS executable: cd ./lib/atc vi Makefile.mpic++ ( set CC = mpiicpc ) make -f Makefile.mpic++ cd ../awpmd vi Makefile.mpicc ( set CC = mpiicpc ) make -f Makefile.mpicc cd ../colvars vi Makefile.g++ ( set CXX = icpc ) make -f Makefile.g++ cd ../cuda make -n make cd ../gpu vi Makefile.fermi ( set CUDA_HOME = /opt/sw/cuda-6.5 EXTRAMAKE = Makefile.lammps.standard CUDA_ARCH = -arch=sm_20 CUDA_PRECISION = -D_SINGLE_DOUBLE CUDA_INCLUDE = -I$(CUDA_HOME)/include CUDA_LIB = -L$(CUDA_HOME)/lib64 -Xlinker -rpath -Xlinker $(CUDA_HOME)/lib64 CUDA_OPTS = -DUNIX -O3 -Xptxas -v --use_fast_math CUDR_CPP = mpicxx -DMPI_GERYON -DUCL_NO_EXIT -DMPICH_IGNORE_CXX_SEEK -I$(CUDA_HOME)/include CUDR_OPTS = -O3 -ffast-math -funroll-loops -DMPI_GERYON ) make -f Makefile.fermi vi Makefile.lammps ( set gpu_SYSPATH = -L/opt/sw/cuda-6.5/lib64 ) cd ../kokkos vi Makefile.lammps ( set CUDA_PATH ?= /opt/sw/cuda-6.5 ) cd ../linalg make -f Makefile.gfortran cd ../meam make -f Makefile.ifort vi Makefile.lammps ( delete flag '-lompstub' from meam_SYSLIB ) cd ../poems make -f Makefile.icc cd ../qmmm make -f Makefile.gfortran cd ../reax make -f Makefile.ifort 7. Copy VSC-X specific Makefile to correct directory (see list at point 1. above): cp makefile_.vscX lammps-20Oct12/src/MAKE/Makefile.vscX ... for version 28Jun14 confirm the following vi ./Makefile.vsc1 ( set CCFLAGS = -O3 -fno-alias -ip -unroll0 -restrict LIB = -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -liomp5 -lstdc++ ) 8. Change into lammps source directory and build lammps; start with the "standard" build: cd .../wherever/it/is/lammps-28Jun14/src export PATH=$PATH:/opt/sw/openkim-api/1.6.3/bin export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:/opt/intel/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH make yes-standard make yes-user-misc make no-voronoi make no-user-cuda make no-gpu make no-kokkos make help make clean-vsc1 make -j 8 vsc1 make vsc1 9. Upon successful compilation a binary called lmp_vsc1 should show up which should be copied to a save place. cp ./lmp_vsc1 /opt/sw/lammps/lmp_vsc1 NOTE: To get information about included packages make package-status ===== LAMMPS on GPUs ====== ==== Installation ==== === Example: lammps-24Jun14 on VSC-1 === - Aiming at the KOKKOS package (see the 2014-09-04 message of A Kohlmeyer, http://sourceforge.net/p/lammps/mailman/message/32798716/) - Follow points 1-7 of the "standard build" described in the previous section; - Select different packages for the actual LAMMPS build still based on the same VSC-1-specific Makefile provided above; cd .../wherever/it/is/lammps-28Jun14/src export PATH=$PATH:/opt/sw/openkim-api/1.6.3/bin export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:/opt/intel/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH make yes-standard make yes-user-misc make no-voronoi make no-user-cuda make yes-gpu make yes-kokkos make help make clean-vsc1 make vsc1 cp ./lmp_vsc1 /opt/sw/lammps/lmp_fermi[kepler]_vsc1 NOTE: for the fermi nodes use ''CUDA_ARCH = -arch=sm_20'', and for the kepler nodes use ''CUDA_ARCH = -arch=sm_35''; so switching from building for 'fermi' to building for 'kepler' requires re-compilation of package ''gpu'', ie cd .../wherever/it/is/lammps-28Jun14/lib/gpu cp Makefile.fermi Makefile.kepler vi Makefile.kepler ( set CUDA_ARCH = -arch=sm_35 ) make -f Makefile.kepler clean make -f Makefile.kepler cd ../../src ... continue with the above block to build lmp_vsc1 ===== Testing LAMMPS ====== ==== MPI-Parallel ==== === For example /opt/sw/lammps/examples/peptide/ on VSC-1 === 1. Change into some temporary directory and prepare the submit script for SGE. vi ./sge_lammps_peptide.scrpt #$ -N lammps_peptide #$ -S /bin/bash #$ -cwd #$ -pe mpich 8 #$ -V export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH export I_MPI_FABRICS=shm:dapl mpirun -machinefile $TMPDIR/machines -np $NSLOTS /opt/sw/lammps/lmp_vsc1 -in ./in.peptide 2. Submit it and compare the results - log.lammps - to the reference log files in /opt/sw/lammps/examples/peptide. qsub ./sge_lammps_peptide.scrpt ==== GPU fermi ==== === For example /opt/sw/lammps/examples/kokkos/ on VSC-1 === 1. Change into some temporary directory and prepare the submit script for SGE (fermi queue). Two consecutive short test runs will employ 1 GPU and 2 GPUs per node. vi ./sge_lammps_fermi_kokkos.scrpt #$ -N lammps_kokkos #$ -S /bin/bash #$ -cwd #$ -pe smp 12 #$ -V #$ -q fermi #$ -P fermi export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH export I_MPI_FABRICS=shm:dapl # single GPU /opt/sw/lammps/lmp_fermi_vsc1 -k on t 6 -sf kk -in ./in.kokkos mv ./log.lammps ./log.lammps_fermi.1gpu.kokkos # 2 x GPU mpirun -np 2 /opt/sw/lammps/lmp_fermi_vsc1 -k on t 6 -sf kk -in ./in.kokkos mv ./log.lammps ./log.lammps_fermi.2gpu.kokkos 2. Change into the fermi group and submit it to the appropriate queue. sg fermi qsub.py ./sge_lammps_fermi_kokkos.scrpt 3. Compare the results - ./log.lammps_fermi.[1,2]gpu.kokkos - to the reference log files in /opt/sw/lammps/examples/kokkos. ==== GPU kepler ==== === For example /opt/sw/lammps/examples/kokkos/ on VSC-1 === 1. Change into some temporary directory and prepare the submit script for SGE (kepler queue). The test is analogous to the one for fermi, hence two consecutive short runs employing 1 GPU or 2 GPUs per node. vi ./sge_lammps_kepler_kokkos.scrpt #$ -N lammps_kokkos #$ -S /bin/bash #$ -cwd #$ -pe smp 12 #$ -V #$ -q kepler #$ -P fermi export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH export I_MPI_FABRICS=shm:dapl # single GPU /opt/sw/lammps/lmp_kepler_vsc1 -k on t 6 -sf kk -in ./in.kokkos mv ./log.lammps ./log.lammps_kepler.1gpu.kokkos # 2 x GPU mpirun -np 2 /opt/sw/lammps/lmp_kepler_vsc1 -k on t 6 -sf kk -in ./in.kokkos mv ./log.lammps ./log.lammps_kepler.2gpu.kokkos 2. Change into the fermi group and submit it to the appropriate --- kepler --- queue. sg fermi qsub.py ./sge_lammps_kepler_kokkos.scrpt 3. Compare the results - ./log.lammps_kepler.[1,2]gpu.kokkos - to the reference log files in /opt/sw/lammps/examples/kokkos.