Differences

This shows you the differences between two versions of the page.

--- doku:gpaw [2014/04/09 13:27] – markus
+++ doku:gpaw [2014/08/27 11:26] – jz
@@ Line 2: / Line 2: @@
 === Settings Scientific linux 6.5 (VSC 1+2) ===
-Following Revisions of GPAW and ASE were used, specify the revisions in <code>install_gpaw_vsc_sl65.sh</code>:
+Following Revisions of GPAW and ASE were used, specify the revisions in ''install_gpaw_vsc_sl65.sh'':
   * gpaw (?): 11253 ; ase (?): 3547
@@ Line 85: / Line 85: @@
 === running gpaw jobs ===
-Job submission on VSC-1 (on VSC-2 also mpich8, mpich4 can be used instead)
+== Job submission using all cores on the compute nodes (VSC-1 and VSC-2) ==
 <code>
 #!/bin/sh
@@ Line 91: / Line 91: @@
 #$ -pe mpich 256
 #$ -V
-export OMP_NUM_THREADS=1
-NSLOTS_PER_NODE_AVAILABLE=8
-NSLOTS_PER_NODE_USED=4
-NSLOTS_REDUCED=`echo "$NSLOTS / $NSLOTS_PER_NODE_AVAILABLE * $NSLOTS_PER_NODE_USED" | bc  `
-echo "starting run with $NSLOTS_REDUCED processes; $NSLOTS_PER_NODE_USED per node"
+mpirun -machinefile $TMPDIR/machines -np $NSLOTS gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64
-for i in `seq 1 $NSLOTS_PER_NODE_USED`
+</code>
-do
-	uniq $TMPDIR/machines >> $TMPDIR/tmp
-done
-sort $TMPDIR/tmp  > $TMPDIR/myhosts
+== Job submission using half of the cores on the compute nodes on VSC-2. ==
-cat $TMPDIR/myhosts
+If each of your processes require more than 2GB (and less than 4GB) of memory, you can use the parallel environment ''mpich8''. This will allocate only 8 processes on each node while still starting 256 processes but distributed over 32 nodes. It is necessary to use the variable ''$NSLOTS_REDUCED'' instead of ''$NSLOTS'' in that case.
+<code>
-mpirun -machinefile $TMPDIR/myhosts -np $NSLOTS_REDUCED gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64
+#!/bin/sh
+#$ -N Cl5_4x4x1
+#$ -pe mpich8 256
+#$ -V
+mpirun -machinefile $TMPDIR/machines -np $NSLOTS_REDUCED gpaw-python static.py --domain=None --band=1 --sl_default=4,4,64
 </code>
+If even more memory per process is required the environments ''mpich4'', ''mpich2'', and ''mpich1'' are also available, as discussed in [[doku:ompmpi|Hybrid OpenMP/MPI jobs]].
+Alternatively, you can simply start your GPAW job with more processes which will also reduce the amount of memory per processes needed. GPAW usually scales well with the number of processes.
-Significant speed up is seen when --sl_default is set. First two parameters for the BLACS grid should be similar in size to get an optimal memory distribution on the nodes.
+Significant speed up is seen **in our test case** when --sl_default is set. First two parameters for the BLACS grid should be similar in size to get an optimal memory distribution on the nodes.