===== OpenFOAM =====
==== OpenFOAM on VSC-4 ====
check currently available openfoam modules/versions:
module avail openfoam*
load openfoam:
spack load -r openfoam-org@7 %intel@19.0.5.281 #load openfoam-org version 7 and additional packages
module load intel/19.0.5 #loading compiler with spack not working, needs to be loaded separately
Example submission script:
#!/bin/sh
#SBATCH -J sim
#SBATCH -N 1
#SBATCH -A p71428
#SBATCH --qos p71428_0096
#SBATCH --tasks-per-node=48
module purge
spack load -r openfoam-org@7 %intel@19.0.5.281
module load intel/19.0.5
EXE=`which foamExec`
$EXE decomposePar -allRegions
mpirun -np 48 $EXE chtMultiRegionFoam -parallel
$EXE reconstructPar -newTimes -allRegions
$EXE rm -r processor*
load openfoam source file to have environment variables and be able to compile custom code:
source $WM_PROJECT_DIR/etc/bashrc
ODE-Solvers within openFoam-org versions should be used with the gcc-compiler.
===== Scaling of OpenFOAM =====
Performance tests were done using different setups, depending on the used cluster:
**VSC-1**
* Intel C-compiler 11.1
* Qlogic MPI 0.1.0
**VSC-2**
* Intel C-compiler 12.1.3
* Intel MPI 4.0.3.008 / impi_intel-4.1.0.024
Compared to the gcc + openmpi Version a speed up of about 1/3 was found in case of 32 cores on VSC-1
Note: the runs denoted "VSC1 qlogic scaled" are the same as "VSC1 qlogic" but the runtimes were multiplied
by a factor of 1.33 which is the nominal factor in clockspeed between VSC-1 and VSC-2
----
=== Cavity Benchmark for version 2.2.0 ===
{{:doku:openfoam:scaling_openfoam_2.2.0.jpg}}
^ cores ^ VSC-2 intel.4.1.0.024 mpich16 ^ VSC-2 intel.4.1.0.024 mpich8 ^
| 16 | 4471 | 4949 |
| 32 | 4753 | 2202 |
| 64 | 1832 | 809 |
| 128| 508 | 279 |
| 256| 234 | 182 |
=== Cavity Benchmark for version 2.1.1 ===
{{:doku:openfoam:scaling_openfoam_2.1.1.jpg}}
== real times for benchmark in seconds: ==
* mpich16: 16 cores per node were used for calculation
* mpich8: 8 cores per node were used for calculation
^ cores ^ VSC-1 qlogic ^ VSC-2 intel.4.0.3.008 mpich16 ^ VSC-2 intel.4.0.3.008 mpich8 ^ VSC-1 gcc + openmpi ^
| 16 | 2843 | 4426 | 3083 | |
| 32 | 1282 | 2176 | 1504 | 1818 |
| 64 | 545 | 937 | 663 | |
| 128| 229 | 360 | 270 | |
| 256| 134 | 207 | 173 | |
----
=== Cavity benchmark for version 1.7.1 ===
{{:doku:openfoam:scaling_openfoam_1.7.1.jpg|}}
== real times for benchmark in seconds: ==
* mpich16: 16 cores per node were used for calculation
^ cores ^ VSC-1 qlogic ^ VSC-2 intel.4.0.3.008 mpich16 ^
| 16 | 2826 | 4473 |
| 32 | 1260 | 2086 |
| 64 | 517 | 887 |
| 128| 224 | 333 |
| 256| - | 225 |