Chilitest#

chilitest is a python interface for ICON model testing (main focus is infrastructure testing through ensemble-generated statistics) which encapsulates runscript generation, experiment execution, output postprocessing and testing protocols. It is designed for creating custom applications/workflows.

By request, chilitest can be extended for testing with other executables (e.g. standalone physics modules), for the use of different perturbation methods, and for other types of testing (e.g. scientific tests that do not require ensemble statistics).

Available ensemble perturbation methods:

Available spatial averages:

  • 2D output:

    • Global mean

  • 3D output:

    • Surface mean per p-level

Available tests:

  • extreme values test (check that each variable does not take more extreme values than the maximums and minimums of that variable found at the ensemble)

  • z-score test

Usage#

Required files:

  • mkexp <exp_name>.config file (no need to declare EXP_ID, it will be set automatically to exp_name)

  • Application configuration <exp_name>_<application_name>.json file. Defines the input parameters for each of the main chilitest objects. Each top-level section corresponds to one object and is passed directly to that object when it is created (i.e. the object constructor defines supported configuration options). Applications validate each object input before doing any work (see validate_config).

  • A section may declare "include": "<file>" (path relative to the declaring config) whose contents serve as defaults for that section. Use this for the analysis definition (variables, spatial_average_type) that a tolerance test and its reference update must agree on, e.g. a shared <exp_name>_analysis.json included by both application configs.

Pre-defined applications#

chilitest contains pre-defined applications like tolerance testing in utils/chilitest/applications. Each application can be executed directly from the command line after setting a mkexp file, and a application configuration file (one json per application script).

Notes#

  • All chilitest experiments read RBF coefficients (per resolution) from Swift server during initialization, to improve reproducibility across model versions

  • In case of zero ensemble spread of a statistical quantity, the corresponding tolerance is set to machine epsilon (~2.22e-16) scaled to the reference value of that quantity

  • Only compatible with mkexp for runscript generation

  • Only atmosphere experiments tested at the moment

  • Only double precision experiments at the moment

  • Only machines with slurm client installed

  • All automatic mkexp paths are generated inside the given build directory. This cannot be modified.

Installation#

Install chilitest in your local python environment in editable mode with pip install -e <path-to-local-chilitest-folder>.

Main files#

  • <exp_name>.config: mkexp configuration. required_options.config shows the required experiment options that must be added to any exp config running with chilitest

  • <exp_name>_analysis.json: the analysis definition shared (via include) by the application configs of one experiment…

    • variables: the variables used at the tests, per dimension (2d/3d). The dimensions to process are derived from these keys

    • spatial_average_type: the type of postprocessing applied to the output variable fields, per dimension (must declare the same dimensions as variables)

  • <exp_name>_<application_name>.json: the per-application configuration, e.g. the statistics list of metrics calculated over ensemble members for the reference update, or the test tolerances (extreme_values.factor, z_score.tolerance) for the tolerance test

Main objects#

  • Ensemble: contains all the information related to an ensemble

  • FileInfo: responsible for the file system specification and path generation

  • ApplicationConfig: responsible for reading the application configuration json files (as well as command line arguments).

    • load: generates a dict from the application config json file

    • load_with_cli: same than load, but the config file and the run-time information (mkexp config, build/statistics directories, SLURM account, mkexp environment/options) are given as regular command line options – see the application’s --help. Only these values are settable on the command line; application-internal config keys cannot be (mis)spelled there. Example: python chili_application.py <config>.json --mkexp-config <exp>.config --build-dir <dir> --stats-dir <dir> --account eg0000

    • validate_config: checks a loaded config against the constructor/method signatures its sections are passed to. The legal values of config strings that select behaviour by name (spatial_average_type, perturbation_method) are declared in chilitest/registry.py

  • Experiment: prepares the file system and runs (ensemble) experiments. Stores model version used in the runs. Allows passing options to mkexp, which will override the values in <exp_name>.config (except for EXP_OPTIONS, where it add to the <exp_name>.config values). Some values (Experiment._mkexp_default_args) cannot be overridden.

  • Statistics: responsible for output postprocessing

  • EnsembleStatistics: responsible for ensemble statistics (npz binary files)

  • Test: tests experiment output against reference ensemble statistics

Plotting#

It is possible to make plots of test and reference ensemble experiments with plot.plot_spatial_average_differences. Only for global means at the moment.

Example#

(after running the AMIP tolerance test example

from chilitest.file_info import FileInfo
from chilitest.application_config import ApplicationConfig
from chilitest.ensemble import Ensemble
from chilitest.statistics import EnsembleStatistics, Statistics
from chilitest.plot import plot_spatial_average_differences

# Only `size` relevant for this example
ensemble = Ensemble(
    perturbation_method="pinit_nml_params",
    size=10,
    perturbation_amplitude=1e-07,
)

# stats-dir = where ref statistics are read from
test_file_info = FileInfo(
    mkexp_config_path="<path-to-icon>/test/tolerance-tests/aes_amip/aes_amip.config",
    build_dir="<path-to-test-build-dir>",
    stats_dir="<path-to-stats-dir>"
)

ref_file_info = FileInfo(
    mkexp_config_path="<path-to-icon>/test/tolerance-tests/aes_amip/aes_amip.config",
    build_dir="<path-to-test-build-dir>",
    stats_dir="<path-to-stats-dir>",
    ensemble=ensemble
)

test_cfg = ApplicationConfig.load("<path-to-icon>/test/tolerance-tests/aes_amip/aes_amip_tolerance_test.json")
ref_cfg = ApplicationConfig.load("<path-to-icon>/test/tolerance-tests/aes_amip/aes_amip_update_ref_statistics.json")

test_stats = Statistics(
    file_info=test_file_info,
    **test_cfg["statistics"]
)

ref_stats = EnsembleStatistics(
    file_info=ref_file_info,
    **ref_cfg["ensemble_statistics"]
)

test_stats.open_dataset()
test_stats.compute_spatial_averages()

ref_stats.open_dataset()
ref_stats.compute_spatial_averages()

plot_spatial_average_differences(
    test_statistics=test_stats,
    ref_statistics=ref_stats,
    ref_file_info=ref_file_info,
    dim="2d"
)