Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision | ||
doku:memory [2014/07/30 11:28] – [ad 2. parallel environment with fewer processes per node] ir | doku:memory [2017/09/01 12:56] – removed ir | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Memory intensive Jobs ====== | ||
- | Jobs requiring more than 2 GB per process have several alternatives. They might use | ||
- | - a node with more memory, | ||
- | - a parallel environment with fewer processes | ||
- | - increased virtual memory, or | ||
- | - swap space (still experimental). | ||
- | |||
- | ==== ad 1. node with more memory ==== | ||
- | |||
- | The free space on the nodes of a job can be listed, e.g. by | ||
- | qstat -F mem_free|grep -B 2 < | ||
- | | ||
- | == 256 GB per node == | ||
- | |||
- | there are 2 nodes with 64 cores and 256 GB of memory each which are accessible via the '' | ||
- | < | ||
- | #$ -q highmem.q | ||
- | </ | ||
- | | ||
- | == 64 GB and 128 GB per node == | ||
- | |||
- | several nodes on the VSC-2 have 64 GB or more instead of the 32 GB of the standard nodes. | ||
- | To use one of these nodes, just add ''' | ||
- | |||
- | See also [[doku: | ||
- | |||
- | ==== ad 2. parallel environment with fewer processes | ||
- | |||
- | Jobs using more than 2 GB per process can be executed in one of the parallel environments | ||
- | * mpich : 2 GB per process | ||
- | * mpich8: 4 GB per process | ||
- | * mpich4: 8 GB per process | ||
- | * mpich2: 16 GB per process | ||
- | * mpich1: 32 GB per process | ||
- | |||
- | Please keep in mind that on VSC-2 jobs are node-exclusive and therefore your contingent of CPU-hours will be computed by full nodes and are therefore significantly more expensive. | ||
- | For example, < | ||
- | |||
- | |||
- | == what to include in your jobscript == | ||
- | < | ||
- | mpirun -machinefile $TMPDIR/ | ||
- | The parallel environment option (pe) includes the type of the requested environment (mpich2) and the number of processes (4). The number of processes per node (ppn) is 2. | ||
- | The machine file and the variable NSLOTS_REDUCED are automatically modified such that they reflect the number of slots requested (4), whereas the variable NSLOTS is set to the number of physical cores allocated in the queueing system. This latter number NSLOTS corresponds to the cost calculation in the previous paragraph: | ||
- | |||
- | === Hybrid MPI/OpenMP jobs === | ||
- | < | ||
- | export OMP_NUM_THREADS=8 | ||
- | mpirun -machinefile $TMPDIR/ | ||
- | In this example, each of the 4 processes allocates 8 cores and starts 8 threads. | ||
- | ==== ad 3. increased virtual memory ==== | ||
- | |||
- | Some programs allocate more memory than they use. This was especially true in old FORTRAN 77 programs, which had to decide at compile time how much memory will be used. These programs are allowed to allocate 50% more memory than available by '''# | ||
- | |||
- | ==== ad 4. swap space (still experimental) ==== | ||
- | |||
- | A novel feature of the VSC-2 is remote swap space (implemented using 'SCSI RDMA Protocol', |