Building ICON with ICON4Py

ICON4Py provides Python-based substitutions for the dynamical core and horizontal diffusion components of ICON. ICON4Py is mainly tested on NVIDIA GPUs; it is not recommended for CPU-only runs. AMD GPU support is experimental.

Limited feature support

The ICON4Py granules do not offer full support of all ICON options. See section Features for details.

Features

Not supported with ICON4Py

The following features are not supported when using ICON4Py:

• Nesting (neither horizontal nor vertical)

• Multi-block runs: ICON must be set up to use only one block (nblocks_e=1, i.e., nproma == number of edges).

• Only a subset of namelist parameters is supported, see below.

Dynamical core (solve_nh in mo_solve_nonhydro.f90)

ICON4Py replaces the dynamical core of ICON with a Python/GT4Py implementation. The substitution covers the non-hydrostatic solver, including velocity advection.

The following table lists the supported namelist options:

Parameter	Supported Value(s)	Description
itime_scheme	4 only (most efficient)	Time stepping scheme
iadv_rhotheta	2 only (Miura)	Rho-theta advection
igradp_method	3 only (Taylor + hydrostatic approx.)	Pressure gradient method
rayleigh_type	2 only (Klemp)	Rayleigh damping type
divdamp_order	2, 4, or 24	Divergence damping order
divdamp_type	3 (3D) or 32 (combined 3D/2D) — not 2	Divergence damping type
lvert_nest	.FALSE. only	Vertical nesting
ldeepatmo	.FALSE. only	Deep atmosphere
lextra_diffu	.TRUE. only	Extra diffusion near CFL limit for vertical advection

Diffusion (diffusion in mo_nh_diffusion.f90)

ICON4Py replaces the horizontal diffusion module.

The following table lists the supported namelist options:

Parameter	Supported Value(s)	Description
hdiff_order	5 only (Smagorinsky + 4th order)	Diffusion type
itype_vn_diffu	1 only (diamond on vertices)	Smagorinsky stencil type
itype_t_diffu	2 only (heterogeneous)	Temperature discretization
lhdiff_smag_w	.FALSE. only	Smagorinsky diffusion for w
lsmag_3d	.FALSE. only	3D Smagorinsky coefficient computation
l_zdiffu_t	.TRUE. or .FALSE.	Truly horizontal temperature diffusion over steep slopes
itype_sher	0, 1, or 2	Shear type
loutshs	.TRUE. or .FALSE.	Output flag for horizontal shear
iforcing (from run_nml)	0 (none), 2 (AES), or 3 (NWP)	Type of physics forcing

Dependencies

For communication (halo exchanges), ICON4Py uses GHex, which requires mpi4py.

Installing ICON4Py

ICON4Py requires Python 3.12 or newer.

The following instructions describe a manual installation. We recommend installing the required Python packages in a virtual environment in three steps:

1. Install mpi4py, which must be compatible with the MPI implementation used for ICON.

2. Install GHex with the flags relevant for the target architecture.

3. Install icon4py.

Installing mpi4py

python3 -m pip install mpi4py

See the mpi4py documentation and the ComIn documentation for mpi4py for details. Make sure mpi4py is working as expected, e.g. by inspecting:

ldd $(python -c "import mpi4py.MPI; print(mpi4py.MPI.__file__)")

Depending on your MPI setup, you might need to install from source:

MPI4PY_BUILD_MPICC=<path to mpi wrapper> python3 -m pip install --no-binary=mpi4py --no-cache-dir mpi4py

Installing GHex

GHex requires build-time flags to select the variant (e.g. GPU support). For a GPU-enabled build, the proper compiler must be available, e.g. nvcc for CUDA compilation.

export GHEX_USE_GPU=ON
export GHEX_GPU_TYPE=NVIDIA
export GHEX_GPU_ARCH=90  # GPU architecture for packing/unpacking kernels
export GHEX_TRANSPORT_BACKEND=MPI
python3 -m pip install ghex

GHex’s Python package uses CMake-based scikit-build-core. Additional build configuration options can be passed via environment variables:

export CXX=<path to C++ compiler> # GCC or Clang recommended
export CC=<path to C compiler> # GCC, Clang or NVHPC
export CUDA_PATH=<path to CUDA installation>
export BOOST_ROOT=<path to boost installation> # header-only install is sufficient; see https://github.com/ghex-org/GHEX for the minimum required version
export MPI_HOME=<path to MPI installation>

Installing ICON4Py

Install ICON4Py, picking the extra that matches your build. externals/icon4py/constraints.txt pins the ICON4Py and DaCe versions tested with the current ICON sources.

python3 -m pip install icon4py-bindings           -c externals/icon4py/constraints.txt   # CPU only
python3 -m pip install icon4py-bindings[cuda12]   -c externals/icon4py/constraints.txt   # GPU, CUDA 12.x
python3 -m pip install icon4py-bindings[cuda13]   -c externals/icon4py/constraints.txt   # GPU, CUDA 13.x

Configuring and Building ICON with ICON4Py

Configure ICON as usual (see Building ICON) with the additional option --enable-icon4py:

./configure --enable-icon4py <other arguments>

At configure time point to the Python interpreter with ICON4Py in its environment.

By default the resulting binary runs with ICON4Py enabled; see icon4py_mode in &run_nml for other modes (off, substitute, verify).

Running ICON with ICON4Py

ICON4Py will just-in-time (JIT) compile optimized programs for the selected ICON setup which will have significant performance impact on the first run. On subsequent runs, the programs will be used from cache. Programs are specialized for the local domains; changing the decomposition will trigger a new compilation.

Source run.env before launching ICON

configure writes a run.env file under externals/icon4py/ in the build directory. Source it in the shell that launches ICON:

. <build-dir>/externals/icon4py/run.env

The file groups its exports in three categories. Override individual variables after sourcing, or edit run.env directly, if you need to deviate.

Compilers

Sets CC, CXX, CUDACXX, and CUDAHOSTCXX to the compilers picked at configure time so GT4Py’s JIT toolchain stays aligned with the rest of ICON. If CMake fails to locate the CUDA toolkit from CUDACXX (rare), add export CUDA_PATH=<path to CUDA toolkit>.

Cache variables

Control where JIT-compiled libraries are stored and how long they survive between runs:

export GT4PY_BUILD_CACHE_LIFETIME=persistent
# export GT4PY_BUILD_CACHE_DIR=<path>   # defaults to cwd

persistent keeps the cache across runs so stencils don’t have to be recompiled every launch. GT4PY_BUILD_CACHE_DIR defaults to the current working directory; set it explicitly to share a cache across runs launched from different working directories.

Performance settings

export PYTHONOPTIMIZE=2

Always apply for best performance, set PYTHONOPTIMIZE=0 for debugging.

Troubleshooting

Runtime error: No module named ‘_cffi_backend’

If you see errors like:

0: Could not find platform dependent libraries <exec_prefix>
0: Failed to initialize the Python-CFFI embedding logic:
0: ModuleNotFoundError: No module named '_cffi_backend'

the Python environment containing ICON4Py is not available at runtime. Make sure the correct Python environment is activated and takes precedence in your PATH.

Selecting the ICON4Py backend at runtime

The GT4Py backend used by the ICON4Py granules is selected by the icon4py_backend switch in &run_nml; the default (0) lets ICON4Py pick automatically. See icon4py_backend in &run_nml for details.