Building ICON#
The process of building ICON consists of two parts: configuring the options and compiler flags, and building the source code with those options and flags.
Configuration#
The configuration step is done by calling the configure
script with arguments specifying the location of libraries and tools required for building, as well as options enabling or disabling particular model features. For example:
./configure CC=mpicc FC=mpif90 LIBS='-lnetcdff -lnetcdf -llapack -lblas' --disable-ocean --disable-coupling
The configure
script of ICON is implemented using Autoconf and its interface should be familiar to those who have experience with Autotools-based building systems.
The following sections provide information on some features and implementation details of the configuration process.
ICON dependencies#
Fig. 1 shows a partial dependency graph of the model. A dependency can be either mandatory (i.e. the package is required regardless of the specified configure options) or optional (i.e. the package is required only if particular features of the model are enabled), and some of the dependencies are provided together with the source code of ICON as git submodules and referred to as bundled packages later in the text.
NOTE: The term bundled package does not apply to all packages listed in .gitmodules
: some of them, e.g. JSBACH and ICON-ART, have circular dependencies with the ICON source code and therefore are treated as part of it.
Fig. 1. ICON dependency graph
The list of packages required for successful configuration and building depends on the selected options. To make the configuration process more transparent, the configure
script does not accept paths to the installation directories of the packages, which would be used to extend the corresponding compiler flags. Instead, paths to the header and library files of the packages must be provided by the user as compiler and linker flags, i.e. in the FCFLAGS
, CPPFLAGS
, and LDFLAGS
arguments. Moreover, the script does not try to guess the list of libraries to be used, therefore all the -l
linker flags need to be specified in the LIBS
argument in the correct order. The recommended (topologically sorted) order for the LIBS
argument is presented in Table 1, and the recommended order for the FCFLAGS
, CPPFLAGS
, LDFLAGS
is the reversed one.
Package |
Required flags1 |
|
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
MTIME (Fortran interface) |
|
|
MTIME (C interface) |
|
|
|
|
|
|
|
|
PPM (C interface) |
|
|
ECCODES (Fortran interface) |
|
|
ECCODES (C interface) |
|
|
YAXT (Fortran interface) |
|
|
YAXT (C interface) |
|
|
SCT (Fortran interface) |
|
|
|
|
|
LAPACK (or analogue) |
mandatory |
|
BLAS (or analogue) |
mandatory |
|
|
|
|
|
|
|
mandatory |
|
|
|
|
|
HDF5 (low- and high-level Fortran interfaces) |
|
|
HDF5 (low-level C interface) |
|
|
|
|
|
static linking |
|
|
MPI (Fortran interface) |
|
|
MPI (C interface) |
|
|
|
|
|
|
|
|
|
|
The dependency conditions and required flags are specified assuming that the shared versions of the libraries containing
RPATH
entries pointing to their dependencies are used (see section Dynamic libraries).ZLIB is used via the
ISO_C_BINDING
interface and does not require additional preprocessor flags.The provided standard C++ library must be compatible with the code generated by
CXX
and/or the host compiler ofCUDACXX
/HIPCXX
.
Bundled packages#
As it was mentioned in the previous section, some of the packages are bundled together with the ICON source code. However, users can download and install those packages before configuring ICON and use them instead. This is controlled by the --with-external-<package>
arguments of the configure
script of ICON. The arguments accept either yes
or no
. If the usage of an external version of a package is requested (i.e. --with-external-<package>=yes
), the compiler and linker flags, i.e. FCFLAGS
, CPPFLAGS
, LDFLAGS
and LIBS
, are supposed to be extended accordingly (see Table 1). The configure
script of ICON fails if the flags are not set correctly or enable a version of the package that is known to be incompatible with ICON.
By default, the bundled versions of the packages are used. In this case, the configure
script of ICON runs the configure and CMake scripts of the packages and extends the compiler and linker flags automatically. The arguments that are passed to the configure and CMake scripts of the bundled packages are composed based on the arguments provided to the configure
script of ICON as follows:
Autotools-based packages:
by default, the arguments are passed unchanged, which means that if you need to give an additional argument to the configure script of a bundled package, you can specify it when calling the
configure
script of ICON, even though, the argument is not listed in its help message;arguments that potentially break the configuration and building of ICON are filtered out (see the expansion of the
ACX_SUBDIR_REMOVE_ARGS
macro inconfigure.ac
), for example, the libraries of the bundled packages must be linked statically, therefore the argument--enable-shared
is never passed to the configure scripts that support it;the list of arguments is extended with ones that enforce consistent building (see expansion to the
ACX_SUBDIR_APPEND_ARGS
macro inconfigure.ac
), for example, the configure scripts of the packages receive additional arguments--disable-shared
and--enable-static
;compiler flags are modified as described in section Compiler flags.
CMake-based packages:
the packages are configured out-of-source using the
CMAKE
command (see section Compilers and tools) with theUnix Makefiles
specified as the generator;the list of arguments is package-specific (see expansion to the
ACX_SUBDIR_APPEND_ARGS
macro inconfigure.ac
) but the shared libraries and tests are generally disabled (-DBUILD_SHARED_LIBS=OFF -DBUILD_TESTING=OFF
);compiler flags that are passed to the
configure
script of ICON are modified as described in section Compiler flags and passed as the respective-DCMAKE_<LANG>_FLAGS
arguments;compiler commands (see section Compilers and tools) are passed via the respective environment variables.
Compilers and tools#
Compilers and tools to be used for building are read by the configure
script of ICON from the following environment variables (can be passed as the command-line arguments):
FC
— Fortran compiler command;CC
— C compiler command;CXX
— C++ compiler command (used by some of the bundled packages);CUDACXX
— CUDA C++ compiler command (used only when the GPU support with CUDA is enabled);HIPCXX
— HIP C++ compiler command (used only when the GPU support with HIP is enabled);AR
— archiver command (used to create static libraries);RANLIB
— archive indexer command (used to create static libraries);PYTHON
— Python interpreter command;CMAKE
— CMake command (used to configure some of the bundled packages);FPP
— Fortran preprocessor command (used when explicit Fortran preprocessing is enabled, see section Preprocessing for more details), must treat the first positional command-line argument as the path to the input source file and print the result to the standard output stream;SB2PP
— Serialbox2 preprocessor command (used when the Serialbox2 serialization is enabled, see section Preprocessing for more details);MPI_LAUNCH
— interactive (synchronous) MPI launcher command (used by the bundled packages for configure-time checks).
If the variables are set, the configure
script checks whether their values meet the requirements, otherwise, the script tries to guess suitable values for them. Thus, if you want to make sure that a particular command for a particular operation is used, you need to specify the corresponding variable explicitly. For example, the usage of NAG compiler is enforced with the following additional command-line argument of the configure
script:
./configure FC=nagfor <other arguments>
Compiler flags#
The configure script supports several groups of compiler flags. Each group is associated with one of the following environment variables (can be passed as the command-line arguments):
FCFLAGS
— Fortran compiler flags to be used when configuring, compiling and linking ICON, as well as passed to the configure and CMake (via the-DCMAKE_Fortran_FLAGS
argument) scripts of the bundled packages (in contrast to standard Autoconf-based scripts, theconfigure
script of ICON does not setFCFLAGS
to-g -O2
by default);ICON_FCFLAGS
— Fortran compiler flags to be appended toFCFLAGS
when configuring, compiling and linking ICON;ICON_<NAME>_FCFLAGS
— Fortran compiler flags to be appended toFCFLAGS
instead ofICON_FCFLAGS
when compiling files of the Fortran compile group<NAME>
(defaults toICON_FCFLAGS
, which can be overridden by setting the variable to an empty value:ICON_OCEAN_FCFLAGS=
);ICON_BUNDLED_FCFLAGS
— Fortran compiler flags to be appended toFCFLAGS
when configuring the bundled packages (defaults toICON_FCFLAGS
, which can be overridden by setting the variable to an empty value:ICON_BUNDLED_FCFLAGS=
);ICON_RTE_RRTMGP_FCFLAGS
,ICON_ECRAD_FCFLAGS
, etc. — Fortran compiler flags to be appendedFCFLAGS
when configuring the respective bundled packages (defaults toICON_BUNDLED_FCFLAGS
, which can be overridden by setting the variables to empty values:ICON_RTE_RRTMGP_FCFLAGS=
,ICON_ECRAD_FCFLAGS=
, etc.);CFLAGS
— C compiler flags to be used when configuring and compiling ICON, as well as passed to the configure and CMake (via the-DCMAKE_C_FLAGS
argument) scripts of the bundled packages (in contrast to standard Autoconf-based scripts, theconfigure
script of ICON does not setFCFLAGS
to-g -O2
by default);CPPFLAGS
— C preprocessor flags to be used when configuring and compiling ICON, as well as passed to the configure and CMake (via the-DCMAKE_C_FLAGS
and-DCMAKE_CXX_FLAGS
arguments) scripts of the bundled packages;ICON_CFLAGS
— C compiler flags to be appended toCFLAGS
when configuring and compiling ICON;ICON_BUNDLED_CFLAGS
— C compiler flags to be appended toCFLAGS
when configuring the bundled packages (defaults toICON_CFLAGS
, which can be overridden by setting the variable to an empty value:ICON_BUNDLED_CFLAGS=
);ICON_CDI_CFLAGS
,ICON_MTIME_CFLAGS
, etc. — C compiler flags to be appended toCFLAGS
when configuring the respective bundled packages (defaults toICON_BUNDLED_CFLAGS
, which can be overridden by setting the variables to empty values:ICON_CDI_CFLAGS=
,ICON_MTIME_CFLAGS=
, etc.);CXXFLAGS
— C++ compiler flags to be passed to the configure and CMake (via the-DCMAKE_CXX_FLAGS
argument) scripts of the bundled packages (in contrast to standard Autoconf-based scripts, theconfigure
script of ICON does not setCXXFLAGS
to-g -O2
by default);ICON_BUNDLED_CXXFLAGS
— C++ compiler flags to be appended toCXXFLAGS
when configuring the bundled packages;CUDAFLAGS
— CUDA C++ compiler flags to be used when configuring and compiling ICON;HIPFLAGS
— HIP C++ compiler flags to be used when configuring and compiling ICON;LDFLAGS
— common Fortran, C and C++ compiler flags to be used when configuring and linking ICON, as well as passed to the configure and CMake (via the-DCMAKE_EXE_LINKER_FLAGS
,-DCMAKE_MODULE_LINKER_FLAGS
and-DCMAKE_SHARED_LINKER_FLAGS
arguments) scripts and of the bundled packages;ICON_LDFLAGS
— Fortran compiler flags to be appended toLDFLAGS
when configuring and linking ICON;LIBS
— a list of libraries (see Table 1 for the recommended order) to be passed to the linker by the Fortran compiler when linking ICON and to the configure and CMake (via the-DCMAKE_Fortran_STANDARD_LIBRARIES
,-DCMAKE_C_STANDARD_LIBRARIES
and-DCMAKE_CXX_STANDARD_LIBRARIES
arguments) scripts of the bundled packages.
The general recommendation to follow when composing the flags is:
Flags specifying search paths for header and module files, i.e. the
-I<path>
flags, should be specified asFCFLAGS
andCPPFLAGS
, depending on whether they need to be passed to Fortran or C compiler, respectively.By default, other flags, e.g. the optimization ones, that are meant for Fortran and C compilers should be appended to
FCFLAGS
andCFLAGS
, respectively.Fortran and C compiler flags that need to be used when configuring, compiling and linking ICON but at the same time can break the configuration (a flag is too restrictive for the configure checks to pass, e.g.
-fimplicit-none
for Gfortran) or the functionality of the bundled packages (e.g. the optimization level required for ICON is too high and leads to errors in the functionality of the bundled packages) can be put toICON_FCFLAGS
,ICON_CFLAGS
orICON_LDFLAGS
.Special optimization flags for a selected set of Fortran source files of ICON can be put to
ICON_<NAME>_FCFLAGS
, where<NAME>
is the name of a Fortran compile group.Fortran and C compiler flags that need to be used when compiling and linking the bundled packages but at the same time conflict with the flags required for ICON (e.g. you want to compile ICON with
-O3
flag but the bundled packages need to be compiled with-O2
) can be specified asICON_BUNDLED_FCFLAGS
andICON_BUNDLED_CFLAGS
, respectively.If a set of Fortran (or C) compiler flags needs to be passed only to some particular bundled package, it can be specified in the respective variable, e.g. in
ICON_CDI_FCFLAGS
(orICON_CDI_CFLAGS
).
Fortran compile groups#
Certain Fortran source files of ICON, including its components that do not have their own build system (e.g. JSBACH) might require special compiler flags. Either due to a decision made by the developers (e.g. the ocean component) or due to compiler bugs. Such cases can be covered with Fortran compile groups. For example, a separate set of flags for the ocean component can be specified at the configure time as follows:
./configure \
--enable-fcgroup-OCEAN=src/hamocc:src/ocean:src/sea_ice \
ICON_OCEAN_FCFLAGS=<special ocean component flags> \
<other arguments>
Or alternatively:
./configure \
--enable-fcgroup-OCEAN \
ICON_OCEAN_PATH=src/hamocc:src/ocean:src/sea_ice \
ICON_OCEAN_FCFLAGS=<special ocean component flags> \
<other arguments>
Dynamic libraries#
For each -L<path>
flag found in the LDFLAGS
and LIBS
variables, the configure
script of ICON generates an additional linker flag that puts the <path>
on the list of runtime library search paths of the ICON executable. This allows for the automatic location of the required libraries by the dynamic linker at the runtime. The flags are appended to LDFLAGS
at the configure time and their actual form depends on the Fortran compiler in use. By default, the flags are composed using the template -Wl,-rpath -Wl,<path>
with currently the only exception for NAG compiler, which accepts the flags in the form -Wl,-Wl,,-rpath -Wl,-Wl,,<path>
. If the -rpath
flags generated by the configure
script break the building or you perform a completely static linking, you can disable the feature with the --disable-rpaths
argument.
Configuration and building environments#
It is important that both the configuration and the building stages are performed in the same environment, i.e. the environment variables that might influence the way the compilers and the linker work are set to the same values when running the configure
script and when running the make
command for building. For example, NAG compiler will not work if the environment variable NAG_KUSARI_FILE
is not set properly. Keeping track of all the steps required to re-initialize the environment for the building stage, e.g. when the configuration stage has already been done but in another terminal session, might be challenging, especially in HPC environments offering multiple compilers and libraries.
To ensure consistency between configuration and building environments, you can set the BUILD_ENV
argument of the configure
script with a series of shell commands that initialize the necessary environment variables. If the BUILD_ENV
variable is not empty, the configure
script will run the commands it contains before running any checks. Additionally, the commands will be saved to the Makefile
, so they will be executed each time the make
command is launched for building. The shell script that is provided as value of the BUILD_ENV
argument must be a one-liner ending with a semicolon (;) symbol, for example:
./configure BUILD_ENV='. /etc/profile.d/modules.sh; module purge; module load intel;' <other arguments>
Also, a proper implementation of the BUILD_ENV
script allows for switching between multiple build directories (see section Out-of-source configuration (building)) without having to re-initialize the environment accordingly.
Configure wrappers#
Real-case configuration commands might be rather long and complex. For example, below, you can find an example of the configuration command for Levante@DKRZ:
./configure \
AR=xiar \
BUILD_ENV=". ./config/dkrz/module_switcher; \
switch_for_module \
intel-oneapi-compilers/2022.0.1-gcc-11.2.0 \
openmpi/4.1.2-intel-2021.5.0;" \
CC=mpicc \
CFLAGS="-g -gdwarf-4 -qno-opt-dynamic-align -m64 -march=core-avx2 \
-mtune=core-avx2 -fma -ip -pc64 -std=gnu99" \
CPPFLAGS="-I/sw/spack-levante/hdf5-1.12.1-tvymb5/include \
-I/sw/spack-levante/netcdf-c-4.8.1-2k3cmu/include \
-I/sw/spack-levante/eccodes-2.21.0-3ehkbb/include \
-I/usr/include/libxml2" \
FC=mpif90 \
FCFLAGS="-I/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include \
-m64 -march=core-avx2 -mtune=core-avx2 -g -gdwarf-4 -pc64 \
-fp-model source" \
ICON_BUNDLED_CFLAGS='-O2 -ftz' \
ICON_CDI_CFLAGS='-O2 -ftz' \
ICON_CFLAGS='-O3 -ftz' \
ICON_ECRAD_FCFLAGS='-qno-opt-dynamic-align -no-fma -fpe0' \
ICON_FCFLAGS="-DDO_NOT_COMBINE_PUT_AND_NOCHECK -O3 -ftz \
-qoverride-limits -assume realloc_lhs \
-align array64byte -fma -ip \
-D__SWAPDIM -DOCE_SOLVE_OMP" \
ICON_OCEAN_FCFLAGS="-O3 -assume norealloc_lhs -reentrancy threaded \
-qopt-report-file=stdout -qopt-report=0 \
-qopt-report-phase=vec" \
ICON_YAC_CFLAGS='-O2 -ftz' \
LDFLAGS="-L/sw/spack-levante/hdf5-1.12.1-tvymb5/lib \
-L/sw/spack-levante/netcdf-c-4.8.1-2k3cmu/lib \
-L/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
-L/sw/spack-levante/netlib-lapack-3.9.1-rwhcz7/lib64 \
-L/sw/spack-levante/eccodes-2.21.0-3ehkbb/lib64" \
LIBS="-Wl,--disable-new-dtags -Wl,--as-needed -lxml2 -leccodes \
-llapack -lblas -lnetcdff -lnetcdf -lhdf5" \
MPI_LAUNCH=mpiexec \
--enable-intel-consistency \
--enable-vectorized-lrtm \
--enable-parallel-netcdf \
--enable-grib2 \
--enable-fcgroup-OCEAN=src/hamocc:src/ocean:src/sea_ice \
--enable-yaxt \
--enable-art \
--enable-ecrad \
--disable-mpi-checks
Repeatedly composing such a command is exhausting and error-prone. Therefore, each team involved in the development of ICON is encouraged to implement and maintain configure wrappers, which would simplify the configuration stage for their users. The wrappers should be put in a subdirectory with a relevant name of the config
directory. Although there are no hard requirements on how the scripts should be implemented, we recommend considering the following features:
Ensure that the wrapper script passes command-line arguments to the
configure
script, allowing users to override default configuration options. For example:./config/dkrz/levante.intel --enable-openmp
Also, account for the case of calling the wrapper with the
--help
argument.Account for out-of-source building:
the wrapper script should be able to locate the
configure
script when called from a directory other than the source root directory, for example:script_dir=$(cd "$(dirname "$0")"; pwd) icon_dir=$(cd "${script_dir}/../.."; pwd) "${icon_dir}/configure" <a list of predefined arguments> "$@"
the wrapper script should prepare the current working directory for the following runscript generation.
Prepend the
LIBS
variable with-Wl,--as-needed
flag so that the actual list of the libraries the ICON executable depends on would include only those required for the particular configuration of the model (see the man pages of the linker for more details:man ld
).Allow for running the
make check
command (see section Building of the bundled packages for more details) by extending theLD_LIBRARY_PATH
environment variable in theBUILD_ENV
script, instead of doing so in the wrapper script itself.
Generic configure wrapper#
The subfolder config/generic
contains generic configure wrappers. These generic configure wrappers assume that all required Software libraries are installed under the same prefix. The prefix defaults to /opt/local
with a fallback to /opt/homebrew
on macOS and to /usr
on other platforms. The default values can be overridden by setting the environment variable ICON_SW_PREFIX
:
export ICON_SW_PREFIX='/path/to/icon/prerequisites'
The following sections provide a list of software required for building and running ICON. Users can build and install (to the same prefix) the listed packages manually or use the package managers available for their platform. Basic instructions on how to do it on several popular platforms are provided in section Supported Hardware.
Building tools#
Software libraries#
MPICH, OpenMPI or any other MPI implementation that provides compiler wrappers
mpicc
,mpicxx
andmpif90
for C, C++ and Fortran, respectively, as well as the job launchermpiexec
HDF5 with high-level interface (for NetCDF-С), thread-safety (for CDO), and szlib filtering support (only C interface required, not a direct dependency of ICON)
NetCDF-C with NetCDF-4 support
ecCodes with JPEG2000 and AEC support (only C interface required)
See section ICON dependencies for more details.
Optional tools#
CDO for pre- and post-processing, also used by some of the generated runscripts
rsync for the generated runscipts in the case of out-of-source building
Out-of-source configuration (building)#
The building system of ICON supports so-called out-of-source builds. This means that you can build ICON in a directory other than the source root directory. The main advantage of this is that you can easily switch between several different configurations (each in its own build directory) of the model while working on the same source code. Hence, it is possible to introduce changes into the source code and test them with different compilers, flags and options without having to re-configure the model or copy the updated source files to other directories.
The configure
script (also when called via a configure wrapper) prepares the current working directory for the following building. The particular case of the current working directory being the source root directory is called in-source configuration.
The following example shows how ICON can be configured on Levante@DKRZ in two different directories using Intel and GCC compiler toolchains (assuming that the source root directory of ICON is /path/to/icon-srcdir
):
mkdir intel && cd intel
/path/to/icon-srcdir/config/dkrz/levante.intel
cd ..
mkdir gcc && cd gcc
/path/to/icon-srcdir/config/dkrz/levante.gcc
cd..
After executing the commands above, you will have two directories named intel
and gcc
, each ready for building ICON with Intel and GCC compilers, respectively.
Building#
The building stage is done with GNU make upon successful completion of the configuration stage. The oldest supported version of make
is 3.81, however, it has significant limitations and it is recommended to use version 4.1 or later.
The building step is done by running make
command with an optional argument specifying the number of jobs to run simultaneously. For example:
make -j8
By default, make
reads the instructions from a file called Makefile
, which in the case of ICON is generated based on the Makefile.in
template and is mainly responsible for the initialization of the building environment. The initialization is performed by the shell script provided to the configure
script as the BUILD_ENV
argument (see section Configuration and building environments). The script is saved in the Makefile
. Each time make
is executed in the root of the ICON build directory, it reads the Makefile
, runs the initialization script and recursively executes itself in the same directory but with another input makefile called icon.mk
. The latter is generated based on the icon.mk.in
template and contains the instructions on how to perform the following building steps required to generate the ICON executable.
The following sections provide information on some features and implementation details of the building process.
Source file collection#
The list of source files that need to be compiled to produce the ICON executable is generated dynamically each time the make
command is executed. This is done using find
command (both GNU and BSD versions are supported) with the assumption that the source files have the following filename extensions:
.f90
— Fortran source files, regardless of whether they contain Fortran preprocessor directives;.inc
— Fortran header files included with the quoted form of the Fortran preprocessor#include
directives, e.g.#include "filename.inc"
;.incf
— Fortran header files included with the FortranINCLUDE
statements, e.g.include 'filename.incf'
(these files are not allowed to have Fortran preprocessor directives);.c
— C source files;.cu
— CUDA source files;.hip.cc
— HIP source files.
The list of source files is a result of the recursive search for files that have the aforementioned extensions and reside in the src
and support
subdirectories of the source root directory of ICON. Additionally, depending on whether the corresponding components of the model were enabled at the configuration stage, the list is extended with Fortran source files from the ./externals/jsbach/src
, ./externals/dace_icon/src_for_icon
, ./externals/emvorado
and ./externals/art
subdirectories.
Preprocessing#
Depending on the configuration, Fortran source files undergo one or more of the following preprocessing procedures.
Fortran source files residing in the
./externals/jsbach/src
are preprocessed with thedsl4jsb.py
script. This is done only if the JSBACH component has been enabled at the configuration stage (--enable-jsbach
). Otherwise, the source files of the component are completely ignored. The output files of this procedure have the same name as the input files plus an additional infix.pp-jsb
.Depending on whether the explicit Fortran preprocessing is enabled (
--enable-explicit-fpp
), the results of the actual previous preprocessing step, together with Fortran source files that have not been preprocessed yet, are preprocessed with the standard Fortran preprocessor. The output files of this procedure have the same name as the input files plus an additional infix.pp-fpp
.If the Serialbox2 serialization is enabled (
--enable-serialization
), the results of the actual previous preprocessing step, together with Fortran source files that have not been preprocessed yet, are preprocessed with the corresponding script of the Serialbox2 toolkit. The output files of this procedure have the same name as the input files plus an additional infix.pp-sb2
.
The output directories of the preprocessing steps have the same layout as the directories containing their input files and the output files have the same prefixes and suffixes (i.e. extensions) as the corresponding input files. For example, if the source file of JSBACH ./externals/jsbach/src/base/mo_jsb_base.f90
is preprocessed by each of the preprocessing steps, the corresponding output files are saved as follows:
./externals/jsbach/src/base/mo_jsb_base.pp-jsb.f90
— JSBACH preprocessing output;./externals/jsbach/src/base/mo_jsb_base.pp-jsb.pp-fpp.f90
— explicit Fortran preprocessing output;./externals/jsbach/src/base/mo_jsb_base.pp-jsb.pp-fpp.pp-sb2.f90
— Serialbox2 preprocessing output.
Source dependency tracking#
Before the compilation can take place, it is required to identify the exact list of source files that need to be compiled to produce the ICON executable. The actual content of the list depends not only on how the code is configured, but possible modifications of the source code made since the last call of make
need to be taken into account as well. Moreover, in the case of Fortran source files, the compilation order becomes important since a source file declaring a Fortran module must be compiled before any other source file using that module. Both tasks are accomplished as follows.
Once the Preprocessing is finished, all source files of ICON (including the enabled components) or their final preprocessed versions are processed with the dependency generator (depgen.py
). The tool parses each source file, detects which header and module files are required for its successful compilation, and stores this information in the form of a makefile. The makefiles are then read by make
and the dependency listing script deplist.py
. The former makes sure that the source files are compiled in the right order, and the latter identifies the list of source files that need to be compiled.
The dependency generator (depgen.py
) recognizes preprocessor #include
, #if
and the associated directives as well as Fortran INCLUDE
, USE
and MODULE
statements. If the usage of a module or a header file is surrounded by the #ifdef SOME_MACRO
and #endif
directives, it will be put on the list of files required for the compilation only if the macro SOME_MACRO
is defined. The list of macro definitions enabling various features of the model is generated at the configuration stage in the form of compiler flags, e.g. -DSOME_MACRO -DSOME_OTHER_MACRO
, which are appended to FCFLAGS
. This way, some Fortran modules do not need to be generated in particular configurations of the model and, therefore, the source files declaring them are not compiled.
The dependency listing script deplist.py
reads the dependency makefiles, builds a source dependency graph and traverses it starting with the vertex associated with the src/drivers/icon.o
object file. Each vertex of the graph accessible from the starting one is printed to the output. The output is then filtered by make
to generate the list of object files required for the ICON executable. This is the main purpose of the listing script. Additionally, the tool can run Code consistency checks described in the following subsection.
Code consistency checks#
Each failed code consistency check run by the dependency listing script deplist.py
is reported to the standard error stream. The identified problems are expressed in terms of files and makefile dependencies and, therefore, require additional explanation provided in this subsection. Normally, the codebase is kept consistent and users do not see the messages described below until they introduce a modification to the source code that breaks the consistency. Currently, the dependency listing script deplist.py
checks the source dependency graph for the following problems:
Two or more Fortran source files declare modules with the same name.
This type of inconsistency is reported as follows:
deplist.py: WARNING: target 'mod/some_module.mod.proxy' has more than one immediate prerequisite matching pattern '*.o': some/dir/some_file.o some/other/dir/some_other_file.o
This means that the Fortran module
some_module
is declared twice. The first declaration is found in the filesome/dir/some_file.f90
and the second declaration is found insome/other/dir/some_other_file.f90
.Two or more Fortran modules circularly depend on each other.
This type of inconsistency is reported as follows:
deplist.py: WARNING: the dependency graph has a cycle: src/drivers/icon.o ... mod/some_module.mod.proxy some/dir/some_file.o mod/some_module_1.mod.proxy <- start of cycle some/other/dir/some_file_1.o mod/some_module_2.mod.proxy some/other/dir2/some_file_2.o mod/some_module_1.mod.proxy <- end of cycle
This reads as that the module
some_module_1
(declared insome/dir/some_file_1.f90
) uses modulesome_module_2
(declared insome/other/dir/some_file_2.f90
), which in turn usessome_module_1
. Usually, this means that the compilation ofsome/dir/some_file_1.f90
will fail.A Fortran module is used but not declared.
This problem is reported by the dependency listing script with the following message:
deplist.py: WARNING: target 'mod/missing_module.mod.proxy' does not have an immediate prerequisite matching any of the patterns: '*.o'
This means that the module
missing_module
is used in one of the source files but there is no Fortran source file in the ICON codebase that declares it. It might be the case, however, that the module is not missing but just not part of the ICON codebase, e.g.mpi
,sct
,yaxt
, etc. Such modules are external to ICON and need to be explicitly specified as such in the filedepgen.f90.config
residing in the current build directory (the file is generated at configuration time based on a template file residing in the source directory. Therefore, to make the modifications persistent, you need to introduce them in the filedepgen.f90.config.in
.Two or more source files have the same basename.
The problem is reported as follows:
deplist.py: WARNING: the dependency graph contains more than one target with basename 'some_file.o': some/dir/some_file.o some/other/dir/some_file.o
This message reports about two (or more) source (not necessarily Fortran) files
some/dir/some_file.f90
andsome/other/dir/some_file.f90
that compile into objects with the same basename. Although handled by the building system in most cases, having several source files with the same basename in a project is considered bad practice, potentially has negative side effects, and, therefore, is deprecated.
Compilation cascade prevention#
It is important, especially for the development process, that the modifications of the source code done after the initial compilation trigger as few recompilations as possible. One of the basic features of make
is to keep track of the file modification timestamps. Based on the information from the makefiles generated by the dependency generator depgen.py
, the tool triggers recompilation of a source file only if the file itself or a header file it includes, or a Fortran module file it uses has been modified since the last execution. Unfortunately, most of the Fortran compilers update the module files even if their relevant contents do not change, i.e. the modification timestamp of a module file gets updated even if the declaration of the associated Fortran module in the source file remains the same. This leads to so-called compilation cascades.
Partially, this issue is circumvented in the building system of ICON as follows.
If a Fortran source file
filename.f90
uses a modulemodulename
, the corresponding dependency makefilefilename.f90.d
(created by the dependency generatordepgen.py
) gets an entry declaring the dependency of the respective object file on a module proxy file:filename.o: mod/modulename.mod.proxy
When the compilation of the file declaring the module
modulename
takes place for the first time, the original module filemod/modulename.mod
generated by the compiler is backed up under the namemod/modulename.mod.proxy
.When
make
checks whether the object filefilename.o
needs to be updated (i.e. the source filefilename.f90
needs to be recompiled), it compares the potentially updated original module filemod/modulename.mod
with the proxy filemod/modulename.mod.proxy
and triggers the recompilation only if they are significantly different. The latter fact is determined in two steps: first, the files are compared for binary identity with thecmp
command, second, if the first check shows the difference, the contents of the files are compared with the/utils/mkhelper/fortmodcmp.py
script, which employs compiler-specific heuristics.Each time the proxy file
mod/modulename.mod.proxy
is detected to be significantly different from the original module filemod/modulename.mod
, it is replaced with a new copy of the latter.
The described mechanism helps to avoid compilation cascades in many cases. However, the structure of the module files generated by most of the compilers is usually not documented, which makes the comparison of the module files difficult. Thus, the redundant recompilations are not guaranteed to be eliminated entirely.
Building of the bundled packages#
The building of the bundled packages is based on the makefiles generated by their configure and CMake scripts. The makefiles are put to the corresponding subdirectories of the ./externals
directory residing in the root build directory of ICON. The packages are built before the compilation of any Fortran source file of ICON takes place. This is done to make sure that the interface Fortran modules of the libraries are available in advance.
Once called in the build directory of ICON, make
recursively runs itself in the build directories of the bundled packages. The list of targets passed to the instances of make
running in the directories of the bundled packages depends on the list of targets specified by the user when calling make
in the build directory of ICON. The targets all
, mostlyclean
(for Autotools-based packages), clean
, distclean
, or check
(for Autotools-based packages) and test
(for CMake-based packages) are preserved and passed over. All other targets are filtered out.
Source provenance collection#
Source provenance information is collected at the building stage and injected into the ICON executable. This information is saved at runtime in the output files of the model so that the latter can be matched with the exact version of ICON that was used to produce them. The information is collected automatically with the help of the pvcs.py
script. The script generated a source file version.c
containing the URL of the git repository, the name of the git branch, and the hash of the git commit. The source file is then treated by make
as part of the ICON codebase.