====== MPI ======

==== MPI Version ====

On VSC-2 several versions of MPI are available.
Choose one using 'mpi-selector' or 'mpi-selector-menu':
<code>
#list available MPI versions:
$ mpi-selector --list
impi_intel-4.1.0.024
impi_intel-4.1.1.036
intel_mpi_intel64-4.0.3.008
mvapich2_1.8_intel_limic
mvapich2_gcc-1.9a2
mvapich2_intel
openmpi-1.5.4_gcc
openmpi-1.5.4_intel
openmpi_gcc-1.6.4

#see the currently used MPI version:
$ mpi-selector --query
default:impi_intel-4.1.0.024
level:user

#set the MPI version:
$ mpi-selector --set impi_intel-4.1.0.024
</code>
Modifications will be active after logging in again.

==== General recommendations ====
To make sure that the MPI communication happens via the infiniband fabric, please use the following settings in your job-script and/or in your ''.bashrc'' file:
<code>
export I_MPI_DAT_LIBRARY=/usr/lib64/libdat2.so.2
export OMP_NUM_THREADS=1
export I_MPI_FABRICS=shm:dapl
export I_MPI_FALLBACK=0
export I_MPI_CPUINFO=proc
export I_MPI_PIN_PROCESSOR_LIST=1,14,9,6,5,10,13,2,3,12,11,4,7,8,15,0
export I_MPI_JOB_FAST_STARTUP=0
</code>

==== Jobs with more than 1024 cores ====
[[doku:large|Jobs with more than 1024 cores]]