Differences

This shows you the differences between two versions of the page.

--- doku:lammps [2014/10/29 10:03] – sh
+++ doku:lammps [2014/11/04 09:36] – sh
@@ Line 183: / Line 183: @@
   - Aiming at the KOKKOS package (see the 2014-09-04 message of A Kohlmeyer, http://sourceforge.net/p/lammps/mailman/message/32798716/)
   - Follow points 1-7 of the "standard build" described in the previous section;
-  - Select different packages for the actual LAMMPS build still based on the same VSC-1-specific Makefile provided above, ie
+  - Select different packages for the actual LAMMPS build still based on the same VSC-1-specific Makefile provided above;
 <code>
 cd .../wherever/it/is/lammps-28Jun14/src
@@ Line 197: / Line 197: @@
 make clean-vsc1
 make vsc1
+cp ./lmp_vsc1   /opt/sw/lammps/lmp_fermi_vsc1
 </code>
-   - NOTE: for the fermi nodes use ''CUDA_ARCH = -arch=sm_20'', and for the kepler nodes use ''CUDA_ARCH = -arch=sm_35''
+NOTE: for the fermi nodes use ''CUDA_ARCH = -arch=sm_20'', and for the kepler nodes use ''CUDA_ARCH = -arch=sm_35''; so switching from building for 'fermi' to building for 'kepler' requires re-compilation of package ''gpu'', ie
-   - Compile the gpu code<code>make -f Makefile.fermi</code>
+<code>
-   - Edit the file {{:doku:lammps:gpu:makefile.lammps|Makefile.lammps}}:<code>gpu_SYSINC =
+      cd .../wherever/it/is/lammps-28Jun14/lib/gpu
-gpu_SYSLIB = -lcudart -lcuda
+      cp Makefile.fermi Makefile.kepler
-gpu_SYSPATH = -L/opt/sw/cuda-6.5/lib64</code>
+      vi Makefile.kepler   ( set CUDA_ARCH = -arch=sm_35 )
-   - Go to the lammps source directory:<code>cd ../../src</code>
+      make -f Makefile.kepler clean
-   - Place {{:doku:lammps:makefile_28jun14.vsc1|this makefile}} in the MAKE folder, rename to ''Makefile.vsc1'' and run <code>make yes-standard
+      make -f Makefile.kepler
-make no-voronoi
+</code>
-make yes-user-cuda
-make yes-gpu
+NOTE_2: To get information about included packages
-make yes-user-misc
+<code>
-make vsc1</code>
+make package-status
+</code>
-To get information about packages that will be used:<code># make package-status
-Installed YES: package ASPHERE
-Installed YES: package BODY
-Installed YES: package CLASS2
-Installed YES: package COLLOID
-Installed YES: package DIPOLE
-Installed YES: package FLD
-Installed YES: package GPU
-  src/pair_lj_sdk_coul_long_gpu.cpp does not exist
-  src/pair_lj_sdk_gpu.cpp does not exist
-  src/pair_lj_sdk_coul_long_gpu.h does not exist
-  src/pair_lj_sdk_gpu.h does not exist
-Installed YES: package GRANULAR
-Installed YES: package KIM
-Installed  NO: package KOKKOS
-Installed YES: package KSPACE
-Installed YES: package MANYBODY
-Installed YES: package MC
-Installed YES: package MEAM
-Installed YES: package MISC
-Installed YES: package MOLECULE
-Installed YES: package MPIIO
-Installed YES: package OPT
-Installed YES: package PERI
-Installed YES: package POEMS
-Installed YES: package REAX
-Installed YES: package REPLICA
-Installed YES: package RIGID
-Installed YES: package SHOCK
-Installed YES: package SRD
-Installed  NO: package VORONOI
-Installed YES: package XTC
-Installed YES: package USER-ATC
+===== Testing LAMMPS ======
-Installed  NO: package USER-AWPMD
-Installed  NO: package USER-CG-CMM
+==== MPI-Parallel ====
-Installed  NO: package USER-COLVARS
-Installed YES: package USER-CUDA
+=== For example /opt/sw/lammps/examples/peptide/ on VSC-1 ===
-  src/pair_lj_sdk_coul_long_cuda.cpp does not exist
-  src/pair_lj_sdk_cuda.cpp does not exist
+. Change into some temporary directory and prepare the submit script for SGE.
-  src/pair_lj_sdk_coul_long_cuda.h does not exist
-  src/pair_lj_sdk_cuda.h does not exist
+vi ./sge_lammps_peptide.scrpt
-Installed  NO: package USER-EFF
+<code>
-Installed  NO: package USER-FEP
+#$ -N lammps_peptide
-Installed  NO: package USER-LB
+#$ -S /bin/bash
-Installed YES: package USER-MISC
+#$ -cwd
-Installed  NO: package USER-MOLFILE
+#$ -pe mpich 8
-Installed  NO: package USER-OMP
+#$ -V
-Installed  NO: package USER-PHONON
-Installed  NO: package USER-QMMM
+export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:$LD_LIBRARY_PATH
-Installed  NO: package USER-REAXC
+export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:$LD_LIBRARY_PATH
-Installed  NO: package USER-SPH</code>
+export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH
+export I_MPI_FABRICS=shm:dapl
+mpirun -machinefile $TMPDIR/machines -np $NSLOTS /opt/sw/lammps/lmp_vsc1  -in  ./in.peptide
+</code>
+. Submit it and compare the results - log.lammps - to the reference log files in /opt/sw/lammps/examples/peptide.
+<code>
+qsub ./sge_lammps_peptide.scrpt
+</code>
+==== GPU fermi ====
+=== For example /opt/sw/lammps/examples/kokkos/ on VSC-1 ===
+. Change into some temporary directory and prepare the submit script for SGE (fermi queue). Two consecutive short test runs will employ 1 GPU and 2 GPUs per node.
+vi ./sge_lammps_fermi_kokkos.scrpt
+<code>
+#$ -N lammps_kokkos
+#$ -S /bin/bash
+#$ -cwd
+#$ -pe smp 12
+#$ -V
+#$ -q fermi
+#$ -P fermi
+export LD_LIBRARY_PATH=/opt/sw/openkim-api/1.6.3/lib/libkim-api-v1.so:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:$LD_LIBRARY_PATH
+export I_MPI_FABRICS=shm:dapl
+# single GPU
+/opt/sw/lammps/lmp_fermi_vsc1 -k on t 6  -sf kk -in ./in.kokkos
+mv ./log.lammps ./log.lammps_fermi.1gpu.kokkos
+# 2 x GPU
+mpirun -np 2 /opt/sw/lammps/lmp_fermi_vsc1 -k on t 6 -sf kk -in ./in.kokkos
+mv ./log.lammps ./log.lammps_fermi.2gpu.kokkos
+</code>
+. Change into the fermi group and submit it to the appropriate queue.
+<code>
+sg fermi
+qsub.py ./sge_lammps_fermi_kokkos.scrpt
+</code>
+. Compare the results - ./log.lammps_fermi.[1,2]gpu.kokkos - to the reference log files in /opt/sw/lammps/examples/kokkos.