Compiling and running

Quick-start

To compile STREAmS, you need at least a Fortran and a C compiler alongside a working installation of the Message Passing Interface (MPI) library.

In order to compile, first go to code folder and search for a configuration file in the make-templates folder. The most basic CPU configuration is makefile.inc.linux64_cpu_gnu. Copy the template file to makefile.inc.

cd code
cp make-templates/makefile.inc.linux64_cpu_gnu makefile.inc

Once the configuration file is ready, it is possible to compile:

make

The executable streams_2.exe should be now in the code folder.

Prepare an input file for your equation. For singleideal equation, the input file is singleideal.ini. Some examples of input files are provided in the folder examples. For instance, copy the subsonic_channel case to your run folder.

mkdir RUN && cd RUN
cp examples/subsonic_channel/singleideal.ini  .

To run the simulation you can use MPI launcher:

mpirun -np 4 ./streams_2.exe

Compilation

STREAmS supports many computational backends and compilers to work in recent High Performance Computing contexts.

In general, the compile configuration file makefile.inc contains 7 parameters:

EQUATION: solved equation: currently only singleideal (single-species Navier-Stokes) is supported
BACKEND: computing backend:
- cpu traditional CPU
- ompc: OpenMP for CPUs
- gpu: CUDA Fortran for NVIDIA GPUs
- amd: HIP for AMD GPUs
- omp: OpenMP-offload potentially for different GPUs, so far tested on Intel PVC GPUs
COMPILE: compiler:
- nv-cuda (NVIDIA CUDA)
- nv (NVIDIA CPU)
- intel (Intel)
- gnu (GNU)
- ibmxl (IBM XL)
- cray-cuda (Cray CUDA)
- hip (AMD HIP)
MODE: compilation mode:
- opt (optimized, default)
- debug (debug options enabled)
PREC: floating point precision:
- double (double precision, default)
- single (single precision, support is experimental)
CUDA_VER: CUDA version (meaningful if CUDA is enabled): e.g., 11.0. Empty string value “” disables CUDA version specification.
MPI_CMD: MPI wrapper command: this is an optional parameter to override the MPI wrapper command given by the chosen compiler
MPI_BACKEND: additional compiler option valid for certain compilers, e.g. -fc=ifx for Intel compiler
CATALYST: specify the in situ visualization library: supported values are v2 to enable Catalystv2 compilation or anthing else to disable in situ.

Several examples of makefile.inc are available in the make-templates folder. STREAmS has been compiled and tested on many HPC clusters. The naming of makefile.inc`s is: `makefile.inc.<machine_name>_<backend>_<compiler>.

In typical HPC machines, a suitable environment has to be loaded within the cluster before compilation so that the correct compiler and libraries are available. It is possible to show some information on the environment associated to a given makefile.inc by typing:

make load

After preparing makefile.inc and setting the environment, it is possible to compile the code:

make

The executable streams_2.exe should be now in the code folder.

To enable Catalyst compilation additional lines are required in makefile.inc. An example is given in makefile.inc.lumi_hip_gnu_insitu. Please refer to the in situ compilation section for details.

Running

First prepare an input file for your equation. For singleideal equation, the input file is singleideal.ini. Some examples of input files are provided in the folder examples. To understand how to write an input file please go to the input section in the documentation.

For curved boundary layer and airfoil cases, an external grid must be provided as grid2d.xyz file. See the section input to understand how to write the grid.

To run the simulation you can use MPI launcher:

mpirun -np 4 ./streams_2.exe

You may need additional settings depending on the backend and compilers. For ompc MPI+OpenMP backend you need to set OMP_NUM_THREADS and optionally affinity variables.

Depending on the configuration of your machine, you may need to use a queue system to launch the simulation. For instance, on Leonardo-Booster at CINECA, a SLURM submission script to run a simulation using 4 GPUs may be:

#!/bin/bash
#SBATCH --partition=boost_usr_prod
#SBATCH --time=24:00:00
#SBATCH --cpus-per-task 8
#SBATCH --exclusive
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-node=4
#SBATCH --ntasks-per-socket 2
#SBATCH --nodes 1

module purge
module load nvhpc/24.3
module load openmpi/4.1.6--nvhpc--24.3

mpirun -np 4 ./streams_2.exe

In situ visualization

STREAmS supports in situ visualization through Catalyst-2 (https://gitlab.kitware.com/paraview/catalyst).

To use it, you first need to compile and install Catalyst2, e.g.:

cmake ../catalyst -DCMAKE_INSTALL_PREFIX=<PATH_CATALYST_INSTALLATION> -DCATALYST_WRAP_FORTRAN=ON

Then you need to compile ParaView with Catalyst 2 support. We first recommend using OSMESA (avoiding EGL) which can be easily installed using Spack (spack install osmesa).

Some significant options to compile ParaView are:

-DVTK_OPENGL_HAS_EGL=OFF
-DPARAVIEW_ENABLE_CATALYST=ON
-DOSMESA_LIBRARY=<PATH_OSMESA_INSTALLATION>/lib/libOSMesa.so
-DOSMESA_INCLUDE_DIR=<PATH_OSMESA_INSTALLATION>/include

The Catalyst installation path must be specified in an environment variable before compiling ParaView:

export catalyst_DIR=<PATH_CATALYST_INSTALLATION>

We assume ParaView is now installed in <PATH_PARAVIEW_INSTALLATION>.

To compile STREAmS, in makefile.inc two more variables are needed:

CATALYST_INCLUDE := -I${PATH_CATALYST_INSTALLATION}/include/catalyst-2.0 -I${PATH_CATALYST_INSTALLATION}/include/catalyst-2.0/conduit
CATALYST_LDFLAGS := -Wl,-rpath,${PATH_CATALYST_INSTALLATION}/lib64 ${PATH_CATALYST_INSTALLATION}/lib64/libcatalyst.so.3 ${PATH_CATALYST_INSTALLATION}/lib64/libcatalyst_fortran.so -Wl,-rpath,${PATH_CATALYST_INSTALLATION}/lib64/catalyst/

Depending on the installation, libstc++ shared library may be added to CATALYST_LDFLAGS as well.

To run STREAmS, two environment variables must be set before running, namely:

export CATALYST_IMPLEMENTATION_PATHS="<PATH_PARAVIEW_INSTALLATION>/lib64/catalyst"
export CATALYST_IMPLEMENTATION_NAME="paraview"

Please note that, sometimes, compiling ParaView using the compiler used to compile STREAmS may be challenging. In these cases, catalyst can be compiled twice: once with the compiler used for ParaView and once with the compiler used for STREAmS. The first one has to be specified in the catalyst_DIR when compiling ParaView, the second one when setting PATH_CATALYST_INSTALLATION used when compiling STREAmS.

An example – to be completed – of makefile.inc for in situ compilation is given in makefile.inc.lumi_hip_gnu_insitu.

To run STREAmS using in situ input file has to be properly configured. Please refer to the input section about in situ for details.