Recommendations

To make sure that the MPI communication happens via the infiniband fabric, please use the following settings in your job-script and/or in your .bashrc file:

export I_MPI_DAT_LIBRARY=/usr/lib64/libdat2.so.2
export OMP_NUM_THREADS=1
export I_MPI_FABRICS=shm:dapl
export I_MPI_FALLBACK=0
export I_MPI_CPUINFO=proc
export I_MPI_PIN_PROCESSOR_LIST=1,14,9,6,5,10,13,2,3,12,11,4,7,8,15,0
export I_MPI_JOB_FAST_STARTUP=0

• VASP
• ANTARES
• WIEN2k
• MPI-Helium
• WRFV3
• Gaussian09
• Sequential codes

• fft libraries
• large jobs with more than 1024 cores
• memory intensive jobs requiring more than 2 GB per core
• ScaLAPACK compile options
• NWChem
• Linking to BLAS Libraries
• user defined prolog and epilog scripts