===== OpenFOAM =====

==== OpenFOAM on VSC-4 ====

check currently available openfoam modules/versions:

<code>
module avail openfoam*
</code>

load openfoam:

<code>
spack load -r openfoam-org@7 %intel@19.0.5.281     #load openfoam-org version 7 and additional packages
module load intel/19.0.5                           #loading compiler with spack not working, needs to be loaded separately
</code>

Example submission script:
<code>
#!/bin/sh
#SBATCH -J sim
#SBATCH -N 1
#SBATCH -A p71428
#SBATCH --qos p71428_0096 
#SBATCH --tasks-per-node=48 

module purge
spack load -r openfoam-org@7 %intel@19.0.5.281
module load intel/19.0.5

EXE=`which foamExec`

$EXE decomposePar -allRegions
mpirun -np 48 $EXE chtMultiRegionFoam -parallel
$EXE reconstructPar -newTimes -allRegions
$EXE rm -r processor*
</code>


load openfoam source file to have environment variables and be able to compile custom code:
<code>
source $WM_PROJECT_DIR/etc/bashrc
</code>

ODE-Solvers within openFoam-org versions should be used with the gcc-compiler.


===== Scaling of OpenFOAM =====

Performance tests were done using different setups, depending on the used cluster:

**VSC-1**
  * Intel C-compiler 11.1
  * Qlogic MPI 0.1.0

**VSC-2**
  * Intel C-compiler 12.1.3
  * Intel MPI 4.0.3.008 / impi_intel-4.1.0.024


Compared to the gcc + openmpi Version a speed up of about 1/3 was found in case of 32 cores on VSC-1

Note: the runs denoted "VSC1 qlogic scaled" are the same as "VSC1 qlogic" but the runtimes were multiplied 
by a factor of 1.33 which is the nominal factor in clockspeed between VSC-1 and VSC-2

----
=== Cavity Benchmark for version 2.2.0 ===
 {{:doku:openfoam:scaling_openfoam_2.2.0.jpg}}

^ cores   ^ VSC-2 intel.4.1.0.024 mpich16 ^ VSC-2 intel.4.1.0.024 mpich8 ^ 
| 	16 |	4471 |	4949 |
|	32 | 	4753 |	2202 |
|	64 |	 1832 |	 809 |
| 	128|	 508 |	 279 |
|	256|	 234 |	 182 |




=== Cavity Benchmark for version 2.1.1 ===

{{:doku:openfoam:scaling_openfoam_2.1.1.jpg}}

== real times for benchmark in seconds: ==
  * mpich16: 16 cores per node were used for calculation
  * mpich8: 8 cores per node were used for calculation

^ cores    ^ VSC-1 qlogic ^ VSC-2 intel.4.0.3.008 mpich16 ^ VSC-2 intel.4.0.3.008 mpich8 ^ VSC-1 gcc + openmpi ^
| 	16 |	2843 |	4426 |	3083 |   |
|	32 | 	1282 |	2176 |	1504 |   1818 |
|	64 |	 545 |	 937 |	 663 |   |
| 	128|	 229 |	 360 |	 270 |   |
|	256|	 134 |	 207 |	 173 |   |

----
=== Cavity benchmark for version 1.7.1 ===

{{:doku:openfoam:scaling_openfoam_1.7.1.jpg|}}


== real times for benchmark in seconds: ==
  * mpich16: 16 cores per node were used for calculation

^ cores    ^ VSC-1 qlogic ^ VSC-2 intel.4.0.3.008 mpich16 ^ 
| 	16 |	2826 |  4473 |
|	32 | 	1260 |  2086 |
|	64 |	 517 |   887 |
| 	128|	 224 |   333 |
|	256|	   - |   225 |