===== GPAW =====

=== Settings Scientific linux 6.5 (VSC 1+2) ===
Following Revisions of GPAW and ASE were used, specify the revisions in ''install_gpaw_vsc_sl65.sh'':

  * gpaw (?): 11253 ; ase (?): 3547
  * gpaw 0.10.0: 11364 ; ase 3.8.1: 3440 
  * gpaw-setups-0.9.9672
  * numpy-1.6.2

  * MPI Version: impi-4.1.x
  * FFTW from INTEL MKL
  * files: {{:doku:gpaw:sl65:site.cfg.numpy.sl65_icc_mkl.txt}}, {{:doku:gpaw:sl65:install_gpaw_vsc_sl65.sh}},  {{:doku:gpaw:sl65:customize_sl65_icc_mkl.py}}, {{:doku:gpaw:sl65:config.py}} 

Notes:
   * The library ''libxc'' had to be installed.
   * In customize_sl65_icc_mkl.py ''extra_link_args += ['-lxc']'' had to be added.
   * In config.py ''mpicompiler = 'mpiicc' '' needed to be set.

=== Settings Scientific linux 6.4 (VSC 1+2) ===
Following Versions of GPAW and ASE were used:
  * gpaw: svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk gpaw -r 10428
  * ase: svn checkout https://svn.fysik.dtu.dk/projects/ase/trunk ase -r 2801
  * gpaw-setups-0.8.7929
  * numpy-1.6.2

  * MPI Version: impi-4.1.0.024 
  * FFTW from INTEL MKL
  * files: {{:doku:gpaw:site.cfg.numpy.sl64_icc_mkl.txt}}, {{:doku:gpaw:install_gpaw_vsc_sl64.sh}},  {{:doku:gpaw:customize_sl64_icc_mkl.py}}, {{:doku:gpaw:config.py}}
 
  * With Scientific Linux 6.4 the header file '/usr/include/python2.6/modsupport.h' makes some Problems, for compiling gpaw some RedHat specific lines at the bottom of the file had to be commented out

=== Settings VSC-1 centos 5.7 (old) ===
Following Versions of GPAW and ASE were used:
  * gpaw: svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk gpaw -r 9616
  * ase: svn checkout https://svn.fysik.dtu.dk/projects/ase/trunk ase -r 2801
  * gpaw-setups-0.8.7929
  * numpy-1.6.2

  * MPI Version: mvapich2_intel_qlc-1.6 ; qlogic MPI did not work
  * FFTW from INTEL MKL
  * files: {{:doku:gpaw:site.cfg.numpy.vsc1.txt}}, {{:doku:gpaw:install_gpaw_vsc_x.sh}}, {{:doku:gpaw:customize_vsc1_icc.py}}, {{:doku:gpaw:config.py}}
 



=== Settings VSC-2 scientific linux 6.1 (old) ===
Following Versions of GPAW and ASE were used:
  * gpaw: svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk gpaw -r 9616
  * ase: svn checkout https://svn.fysik.dtu.dk/projects/ase/trunk ase -r 2801
  * gpaw-setups-0.8.7929
  * numpy-1.6.2

  * MPI Version: intel_mpi_intel64-4.0.3.008
  * FFTW from INTEL MKL
  * files: {{:doku:gpaw:site.cfg.numpy.vsc2.txt}}, {{:doku:gpaw:install_gpaw_vsc_x.sh}}, {{:doku:gpaw:customize_vsc2_icc.py}}, {{:doku:gpaw:config.py}}

=== Settings gpaw qmmm version ===
As other versions, but with special config file:
  * {{:doku:gpaw:config.qmmm.py}}
  * use other config files for specific cluster / OS

=== Installation procedure ===
  * For installing download the following files from one of the settings sections above to a directory of your choice
  * After downloading edit file 'install_gpaw_*.sh'.
  * Execute install_gpaw_*.sh
   <code>
   bash install_gpaw_*.sh all
   </code>
  * Testing of GPAW:
<code>
gpaw-python <DIR_TO_GPAW_INST_BIN>/gpaw-test
</code>


For certain problems an assert statement in 
<code>
~/gpaw_inst_latest/lib64/python2.6/site-packages/gpaw/wavefunctions/lcao.py
</code>

has to be commented out (approx line 239):
<code>
#assert abs(c_n.imag).max() < 1e-14
</code>

=== running gpaw jobs ===
== Job submission using all cores on the compute nodes (VSC-1 and VSC-2) ==
<code>
#!/bin/sh
#$ -N Cl5_4x4x1
#$ -pe mpich 256
#$ -V

mpirun -machinefile $TMPDIR/machines -np $NSLOTS gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64
</code>

== Job submission using half of the cores on the compute nodes on VSC-2. ==
If each of your processes require more than 2GB (and less than 4GB) of memory, you can use the parallel environment ''mpich8''. This will allocate only 8 processes on each node while still starting 256 processes but distributed over 32 nodes. It is necessary to use the variable ''$NSLOTS_REDUCED'' instead of ''$NSLOTS'' in that case.
<code>
#!/bin/sh
#$ -N Cl5_4x4x1
#$ -pe mpich8 256
#$ -V

mpirun -machinefile $TMPDIR/machines -np $NSLOTS_REDUCED gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64
</code>
If even more memory per process is required the environments ''mpich4'', ''mpich2'', and ''mpich1'' are also available, as discussed in [[doku:ompmpi|Hybrid OpenMP/MPI jobs]].
Alternatively, you can simply start your GPAW job with more processes which will reduce the amount of memory per process. GPAW usually scales well with the number of processes.

Significant speed up is seen **in our test case** when --sl_default is set. First two parameters for the BLACS grid should be similar in size to get an optimal memory distribution on the nodes.