check currently available openfoam modules/versions:

module avail openfoam*

load openfoam:

spack load -r openfoam-org@7 %intel@19.0.5.281     #load openfoam-org version 7 and additional packages
module load intel/19.0.5                           #loading compiler with spack not working, needs to be loaded separately

Example submission script:

#!/bin/sh
#SBATCH -J sim
#SBATCH -N 1
#SBATCH -A p71428
#SBATCH --qos p71428_0096 
#SBATCH --tasks-per-node=48 

module purge
spack load -r openfoam-org@7 %intel@19.0.5.281
module load intel/19.0.5

EXE=`which foamExec`

$EXE decomposePar -allRegions
mpirun -np 48 $EXE chtMultiRegionFoam -parallel
$EXE reconstructPar -newTimes -allRegions
$EXE rm -r processor*

load openfoam source file to have environment variables and be able to compile custom code:

source $WM_PROJECT_DIR/etc/bashrc

ODE-Solvers within openFoam-org versions should be used with the gcc-compiler.

Performance tests were done using different setups, depending on the used cluster:

VSC-1

  • Intel C-compiler 11.1
  • Qlogic MPI 0.1.0

VSC-2

  • Intel C-compiler 12.1.3
  • Intel MPI 4.0.3.008 / impi_intel-4.1.0.024

Compared to the gcc + openmpi Version a speed up of about 1/3 was found in case of 32 cores on VSC-1

Note: the runs denoted “VSC1 qlogic scaled” are the same as “VSC1 qlogic” but the runtimes were multiplied by a factor of 1.33 which is the nominal factor in clockspeed between VSC-1 and VSC-2


Cavity Benchmark for version 2.2.0

scaling_openfoam_2.2.0.jpg

cores VSC-2 intel.4.1.0.024 mpich16 VSC-2 intel.4.1.0.024 mpich8
16 4471 4949
32 4753 2202
64 1832 809
128 508 279
256 234 182

Cavity Benchmark for version 2.1.1

scaling_openfoam_2.1.1.jpg

real times for benchmark in seconds:
  • mpich16: 16 cores per node were used for calculation
  • mpich8: 8 cores per node were used for calculation
cores VSC-1 qlogic VSC-2 intel.4.0.3.008 mpich16 VSC-2 intel.4.0.3.008 mpich8 VSC-1 gcc + openmpi
16 2843 4426 3083
32 1282 2176 1504 1818
64 545 937 663
128 229 360 270
256 134 207 173

Cavity benchmark for version 1.7.1

real times for benchmark in seconds:
  • mpich16: 16 cores per node were used for calculation
cores VSC-1 qlogic VSC-2 intel.4.0.3.008 mpich16
16 2826 4473
32 1260 2086
64 517 887
128 224 333
256 - 225
  • doku/openfoam.txt
  • Last modified: 2023/05/17 14:18
  • by msiegel