To make sure that the MPI communication happens via the infiniband fabric, please use the following settings in your job-script and/or in your .bashrc file:

export I_MPI_DAT_LIBRARY=/usr/lib64/libdat2.so.2
export I_MPI_JOB_FAST_STARTUP=0