Following Revisions of GPAW and ASE were used, specify the revisions in install_gpaw_vsc_sl65.sh:

• gpaw (?): 11253 ; ase (?): 3547
• gpaw 0.10.0: 11364 ; ase 3.8.1: 3440
• gpaw-setups-0.9.9672
• numpy-1.6.2

• MPI Version: impi-4.1.x
• FFTW from INTEL MKL
• files: site.cfg.numpy.sl65_icc_mkl.txt, install_gpaw_vsc_sl65.sh, customize_sl65_icc_mkl.py, config.py

Notes:

• The library libxc had to be installed.
• In customize_sl65_icc_mkl.py extra_link_args += ['-lxc'] had to be added.
• In config.py mpicompiler = 'mpiicc' needed to be set.

Following Versions of GPAW and ASE were used:

• gpaw: svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk gpaw -r 10428
• ase: svn checkout https://svn.fysik.dtu.dk/projects/ase/trunk ase -r 2801
• gpaw-setups-0.8.7929
• numpy-1.6.2

• MPI Version: impi-4.1.0.024
• FFTW from INTEL MKL
• files: site.cfg.numpy.sl64_icc_mkl.txt, install_gpaw_vsc_sl64.sh, customize_sl64_icc_mkl.py, config.py

• With Scientific Linux 6.4 the header file '/usr/include/python2.6/modsupport.h' makes some Problems, for compiling gpaw some RedHat specific lines at the bottom of the file had to be commented out

Following Versions of GPAW and ASE were used:

• gpaw: svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk gpaw -r 9616
• ase: svn checkout https://svn.fysik.dtu.dk/projects/ase/trunk ase -r 2801
• gpaw-setups-0.8.7929
• numpy-1.6.2

• MPI Version: mvapich2_intel_qlc-1.6 ; qlogic MPI did not work
• FFTW from INTEL MKL
• files: site.cfg.numpy.vsc1.txt, install_gpaw_vsc_x.sh, customize_vsc1_icc.py, config.py

Following Versions of GPAW and ASE were used:

• gpaw: svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk gpaw -r 9616
• ase: svn checkout https://svn.fysik.dtu.dk/projects/ase/trunk ase -r 2801
• gpaw-setups-0.8.7929
• numpy-1.6.2

• MPI Version: intel_mpi_intel64-4.0.3.008
• FFTW from INTEL MKL
• files: site.cfg.numpy.vsc2.txt, install_gpaw_vsc_x.sh, customize_vsc2_icc.py, config.py

As other versions, but with special config file:

• config.qmmm.py
• use other config files for specific cluster / OS

• For installing download the following files from one of the settings sections above to a directory of your choice
• After downloading edit file 'install_gpaw_*.sh'.
• Execute install_gpaw_*.sh

   bash install_gpaw_*.sh all
   

• Testing of GPAW:

gpaw-python <DIR_TO_GPAW_INST_BIN>/gpaw-test

For certain problems an assert statement in

~/gpaw_inst_latest/lib64/python2.6/site-packages/gpaw/wavefunctions/lcao.py

has to be commented out (approx line 239):

#assert abs(c_n.imag).max() < 1e-14

#!/bin/sh
#$ -N Cl5_4x4x1
#$ -pe mpich 256
#$ -V

mpirun -machinefile $TMPDIR/machines -np $NSLOTS gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64

If each of your processes require more than 2GB (and less than 4GB) of memory, you can use the parallel environment mpich8. This will allocate only 8 processes on each node while still starting 256 processes but distributed over 32 nodes. It is necessary to use the variable $NSLOTS_REDUCED instead of $NSLOTS in that case.

#!/bin/sh
#$ -N Cl5_4x4x1
#$ -pe mpich8 256
#$ -V

mpirun -machinefile $TMPDIR/machines -np $NSLOTS_REDUCED gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64

If even more memory per process is required the environments mpich4, mpich2, and mpich1 are also available, as discussed in Hybrid OpenMP/MPI jobs. Alternatively, you can simply start your GPAW job with more processes which will reduce the amount of memory per process. GPAW usually scales well with the number of processes.

Significant speed up is seen in our test case when –sl_default is set. First two parameters for the BLACS grid should be similar in size to get an optimal memory distribution on the nodes.