Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
doku:ompmpi [2014/08/12 13:45] – [2. Automatic modification through parallel environment variable] ir | doku:ompmpi [2021/05/13 17:47] (current) – removed goldenberg | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Hybrid OpenMP/MPI jobs ====== | ||
- | |||
- | Starting fewer MPI processes per node than slots available can be established in two different ways, | ||
- | - either by manually modifying the machine file | ||
- | - or by automatic modification through a parallel environment variable. | ||
- | |||
- | |||
- | ==== 1. Manually modifying the machine file ==== | ||
- | |||
- | In cases where not all CPUs of one node are required, the machines file can be modified to guarantee the right behaviour of mpirun. The $TMPDIR/ | ||
- | |||
- | < | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | </ | ||
- | |||
- | |||
- | For running a job on less than eight cores the $TMPDIR/ | ||
- | |||
- | < | ||
- | #$ -N test | ||
- | #$ -pe mpich 16 | ||
- | |||
- | NSLOTS_PER_NODE_AVAILABLE=8 | ||
- | NSLOTS_PER_NODE_USED=4 | ||
- | NSLOTS_REDUCED=`echo " | ||
- | |||
- | echo " | ||
- | for i in `seq 1 $NSLOTS_PER_NODE_USED` | ||
- | do | ||
- | uniq $TMPDIR/ | ||
- | done | ||
- | sort $TMPDIR/ | ||
- | cat $TMPDIR/ | ||
- | |||
- | |||
- | mpirun -machinefile $TMPDIR/ | ||
- | |||
- | </ | ||
- | |||
- | This job script yields the following machines file: | ||
- | |||
- | < | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r10n01 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | r12n10 | ||
- | </ | ||
- | |||
- | ==== 2. Automatic modification through parallel environment ==== | ||
- | |||
- | Jobs using more than 2 GB per process can be executed in one of the parallel environments | ||
- | * mpich : 2 GB per process | ||
- | * mpich8: 4 GB per process | ||
- | * mpich4: 8 GB per process | ||
- | * mpich2: 16 GB per process | ||
- | * mpich1: 32 GB per process | ||
- | |||
- | Please keep in mind that on VSC-2 jobs are node-exclusive and therefore your contingent of CPU-hours will be computed by full nodes and are therefore significantly more expensive. | ||
- | For example, < | ||
- | |||
- | |||
- | == what to include in your jobscript == | ||
- | < | ||
- | mpirun -machinefile $TMPDIR/ | ||
- | The parallel environment option (pe) includes the type of the requested environment (mpich2) and the number of processes (4). The number of processes per node (ppn) is 2. | ||
- | The machine file and the variable NSLOTS_REDUCED are automatically modified such that they reflect the number of slots requested (4), whereas the variable NSLOTS is set to the number of physical cores allocated in the queueing system. This latter number NSLOTS corresponds to the cost calculation in the previous paragraph: | ||
- | |||
- | === Hybrid MPI/OpenMP jobs === | ||
- | < | ||
- | export OMP_NUM_THREADS=8 | ||
- | mpirun -machinefile $TMPDIR/ | ||
- | In this example, two nodes are reserved in the grid engine. On both nodes 2 processes each spanning 8 threads are started. |