check currently available openfoam modules/versions:

module avail openfoam*

load openfoam:

spack load -r openfoam-org@7 %intel@19.0.5.281     #load openfoam-org version 7 and additional packages
module load intel/19.0.5                           #loading compiler with spack not working, needs to be loaded separately

Example submission script:

#!/bin/sh
#SBATCH -J sim
#SBATCH -N 1
#SBATCH -A p71428
#SBATCH --qos p71428_0096 
#SBATCH --tasks-per-node=48 

module purge
spack load -r openfoam-org@7 %intel@19.0.5.281
module load intel/19.0.5

EXE=`which foamExec`

$EXE decomposePar -allRegions
mpirun -np 48 $EXE chtMultiRegionFoam -parallel
$EXE reconstructPar -newTimes -allRegions
$EXE rm -r processor*

load openfoam source file to have environment variables and be able to compile custom code:

source $WM_PROJECT_DIR/etc/bashrc

ODE-Solvers within openFoam-org versions should be used with the gcc-compiler.

a simple jobscript for using the /work/tmp directory on VSC1 for openFOAM:

job_tmp.sh

in July 2013 VSC-1 has been updated to Scientific Linux 6.4 (identical with VSC-2). Also the latest Infiniband Software only includes Intel MPI. For installing OpenFoam on VSC-1 please use the installation Scripts from VSC-2

Set the MPI Version to 'qlogicmpi_intel-0.1.0', logout and login again.

Download the source packages for OpenFoam and Thirdparties. Place them at the same directory as the scripts which are provided below.

installopenfoam_1.7.1_vsc1.sh

Patched File: turbulentheatfluxtemperaturefvpatchscalarfield.c

Place at same directory as the script is executed.

installopenfoam_2.1.1_vsc1.sh

Set the MPI Version to 'intel_mpi_intel64-4.0.3.008' (1.7.1, 2.1.1) or 'impi_intel-4.1.0.024' (2.2.0), logout and login again.

Download the source packages for OpenFoam and Thirdparties. Place them at the same directory as the scripts which are provided below.

installopenfoam_1.7.1_vsc2.sh

Patched File: turbulentheatfluxtemperaturefvpatchscalarfield.c

Place at same directory as the script is executed.

installopenfoam_2.1.1_vsc2.sh

installopenfoam_2.2.0_vsc2.sh

installopenfoam_2.2.2_vsc2.sh

installopenfoam_2.3.x_vsc2.sh

version using the latest git repository of openfoam:

intel mpi 4.1.0.024 installopenfoam_2.3.x_vsc2_git_intel_mpi.4.1.0.024.sh

intel mpi 4.1.3.045 installopenfoam_2.3.x_vsc2_git_intel_mpi.4.1.3.045.sh

install_foam-ext-3.0_git.sh

script may needed to be run twice, because git clone some how terminates the script …

install_foam-ext-3.1_git.sh

compilation succeeded except some mesh conversion tools

Performance tests were done using different setups, depending on the used cluster:

VSC-1

• Intel C-compiler 11.1
• Qlogic MPI 0.1.0

VSC-2

• Intel C-compiler 12.1.3
• Intel MPI 4.0.3.008 / impi_intel-4.1.0.024

Compared to the gcc + openmpi Version a speed up of about 1/3 was found in case of 32 cores on VSC-1

Note: the runs denoted “VSC1 qlogic scaled” are the same as “VSC1 qlogic” but the runtimes were multiplied by a factor of 1.33 which is the nominal factor in clockspeed between VSC-1 and VSC-2

• mpich16: 16 cores per node were used for calculation
• mpich8: 8 cores per node were used for calculation

• mpich16: 16 cores per node were used for calculation

was used; ask admins