identibench 0.3.1

Downloads and prepares various system identification benchmark datasets

IdentiBench

IdentiBench is a Python library designed to streamline and standardize the benchmarking of system identification models. Evaluating and comparing dynamic models often requires repetitive setup for data handling, evaluation protocols, and metrics implementation, making fair comparisons and reproducing results challenging. IdentiBench tackles this by offering a collection of pre-defined benchmark specifications for simulation and prediction tasks, built upon common datasets. It automates data downloading and processing into a consistent format and provides standard evaluation metrics via a simple interface (run_benchmark). This allows you to focus your efforts on developing innovative models, while relying on IdentiBench for robust and reproducible evaluation.

Key Features

• Access Many Benchmarks from different systems: Instantly utilize pre-configured benchmarks covering diverse domains like electronics (Silverbox), mechanics (Industrial Robot), process control (Cascaded Tanks), aerospace (Quadrotors), and more, available for both simulation and prediction tasks.
• Automate Data Management: Forget manual downloading and processing; the library handles fetching data from various sources (web, Drive, Dataverse), extracting archives (ZIP, RAR, MAT, BAG), converting to a standard HDF5 format, and caching locally.
• Integrate Any Model to evaluate on all benchmarks: Plug in your custom models, regardless of the Python framework used (NumPy, SciPy, PyTorch, TensorFlow, JAX, etc.), using a straightforward function interface (build_model) that receives all necessary context.
• Capture Comprehensive Results: Obtain detailed evaluation reports including standard metrics (RMSE, NRMSE, FIT%, etc.), task-specific scores, execution timings, configuration parameters (hyperparameters, seed), and raw model predictions for thorough analysis.
• Easily Define New Benchmarks: Go beyond the included datasets by creating your own benchmark specifications (BenchmarkSpecSimulation, BenchmarkSpecPrediction) for private data or unique tasks, leveraging the library’s structure and transparent data format.

Installation

You can install identibench using pip:

pip install identibench

To install the latest development version directly from GitHub, use:

pip install git+https://github.com/daniel-om-weber/identibench.git

For development:

git clone https://github.com/daniel-om-weber/identibench.git
cd identibench
uv sync --extra dev

Quickstart

A model is a build_model(context) function that trains on the benchmark's training data and returns a predictor callable. For a simulation benchmark the predictor is called as model(u, y_init, attrs) and returns the simulated output. That's the whole interface — IdentiBench handles downloading the data, running your predictor over every test sequence, and scoring it.

import numpy as np
import identibench as idb

def build_model(context):
    # Train on the benchmark's training split. This trivial baseline just
    # predicts the mean of the training output — replace it with your own model.
    y_train = np.concatenate([seq.y for seq in context.get_train_sequences()])
    y_mean = y_train.mean(axis=0)

    def model(u, y_init, attrs):
        # Called once per test sequence; returns an array of shape (len(u), n_y).
        return np.tile(y_mean, (len(u), 1))

    return model

# The first call downloads and caches the dataset under ~/.identibench_data
# (override with the IDENTIBENCH_DATA_ROOT environment variable). WH is small
# (a few MB), so it's a good first benchmark.
result = idb.run_benchmark(idb.BenchmarkWH_Simulation, build_model)
print(result["metric_score"])   # primary metric (RMSE in mV) on the test set

For complete, runnable models see the examples/ notebooks: 00_getting_started fits a linear ARX baseline on a simulation benchmark, and 01_riann_orientation scores an orientation model on the IMU benchmarks.

Datasets download on first use and are cached locally, so you only pay for them once. The classic simulation/prediction benchmarks are small (a few MB each); the orientation/IMU and quadrotor datasets are larger (hundreds of MB — e.g. BROAD is ~0.8 GB), so start with a small benchmark such as WH_Sim.

Training a model with hyperparameters

build_model receives the full context, so you can fit any model and read settings from context.hyperparameters. This example trains a NARX model with sysidentpy (install it first: pip install sysidentpy):

from sysidentpy.model_structure_selection import FROLS
from sysidentpy.parameter_estimation import LeastSquares

def build_frols_model(context):
    u_train, y_train, _ = next(context.get_train_sequences())

    ylag = context.hyperparameters.get('ylag', 5)
    xlag = context.hyperparameters.get('xlag', 5)
    n_terms = context.hyperparameters.get('n_terms', 10)
    estimator = context.hyperparameters.get('estimator', LeastSquares())

    _model = FROLS(xlag=xlag, ylag=ylag, n_terms=n_terms, estimator=estimator)
    _model.fit(X=u_train, y=y_train)

    def model(u_test, y_init, attrs):
        yhat_full = _model.predict(X=u_test, y=y_init[:_model.max_lag])
        return yhat_full[_model.max_lag:]

    return model

hyperparams = {
    'ylag': 2,
    'xlag': 2,
    'n_terms': 10,            # number of terms for FROLS
    'estimator': LeastSquares(),
}

results = idb.run_benchmark(
    spec=idb.BenchmarkWH_Simulation,
    build_model=build_frols_model,
    hyperparameters=hyperparams,
)

Simulation Benchmarks

Key	Benchmark Name
WH_Sim	BenchmarkWH_Simulation
Silverbox_Sim	BenchmarkSilverbox_Simulation
Tanks_Sim	BenchmarkCascadedTanks_Simulation
CED_Sim	BenchmarkCED_Simulation
EMPS_Sim	BenchmarkEMPS_Simulation
NoisyWH_Sim	BenchmarkNoisyWH_Simulation
RobotForward_Sim	BenchmarkRobotForward_Simulation
RobotInverse_Sim	BenchmarkRobotInverse_Simulation
Ship_Sim	BenchmarkShip_Simulation
QuadPelican_Sim	BenchmarkQuadPelican_Simulation
QuadPi_Sim	BenchmarkQuadPi_Simulation

Orientation (IMU) Benchmarks

Estimate orientation (a unit quaternion) from 6-axis IMU data and score it with the inclination (tilt) error in degrees. These are BenchmarkSpecSimulation tasks — plug in any model via build_model (neural network, complementary filter, …).

The RIANN benchmarks port the six datasets from Weber et al. 2021. The combined riann benchmark reproduces the paper's pooled-train / cross-dataset-test protocol; the per-source benchmarks let you download and evaluate a single dataset in isolation. Each is downloaded from its original public source on first use.

Key	Benchmark Name	Files (role)
RIANN_Inclination	BenchmarkRIANN_Inclination	36 train / 6 valid / 119 test (all sources)
BROAD_Inclination	BenchmarkBROAD_Inclination	39 test
TUMVI_Inclination	BenchmarkTUMVI_Inclination	6 test
OxIOD_Inclination	BenchmarkOxIOD_Inclination	71 test
EuRoC_Inclination	BenchmarkEuRoC_Inclination	6 test
RepoIMU_Inclination	BenchmarkRepoIMU_Inclination	21 test
Caruso_Inclination	BenchmarkCaruso_Inclination	18 test
DFJIMU_Inclination	BenchmarkDFJIMU_Inclination	Weygers & Kok (2020)
DFJIMU_Relative	BenchmarkDFJIMU_Relative	Weygers & Kok (2020)

All of these live in identibench.datasets.orientation and are grouped in the idb.orientation_benchmarks registry. Inputs are u_cols = [acc_x, acc_y, acc_z, gyr_x, gyr_y, gyr_z, dt] (acc m/s², gyr rad/s, dt the per-sample sampling interval in seconds; the dfjimu benchmarks omit dt) and the target is y_cols = [q_w, q_x, q_y, q_z] (quaternion w, x, y, z).

import numpy as np
import identibench as idb

def build_model(context):
    # A trivial baseline that always predicts the identity orientation.
    def model(u, y_init, attrs):
        return np.tile([1.0, 0.0, 0.0, 0.0], (len(u), 1))
    return model

# Evaluate on a single small dataset...
result = idb.run_benchmark(idb.BenchmarkEuRoC_Inclination, build_model)

# ...or reproduce the full RIANN cross-dataset protocol in one run.
result = idb.run_benchmark(idb.BenchmarkRIANN_Inclination, build_model)
print(result["metric_score"])          # aligned (unmasked) inclination RMSE, deg
print(result["custom_scores"])         # faithful masked RMSE + 99th pct, per source

Which number to report. metric_score is the first-sample-aligned but unmasked inclination RMSE. RIANN's published numbers are the masked RMSE and its 99th percentile, per dataset — these live in custom_scores (<source>/incl_rmse_deg, <source>/incl_p99_deg, and pooled all/…), surfaced as cs_* columns by benchmark_results_to_dataframe.

Prediction Benchmarks

Key	Benchmark Name
WH_Pred	BenchmarkWH_Prediction
Silverbox_Pred	BenchmarkSilverbox_Prediction
Tanks_Pred	BenchmarkCascadedTanks_Prediction
CED_Pred	BenchmarkCED_Prediction
EMPS_Pred	BenchmarkEMPS_Prediction
NoisyWH_Pred	BenchmarkNoisyWH_Prediction
RobotForward_Pred	BenchmarkRobotForward_Prediction
RobotInverse_Pred	BenchmarkRobotInverse_Prediction
Ship_Pred	BenchmarkShip_Prediction
QuadPelican_Pred	BenchmarkQuadPelican_Prediction
QuadPi_Pred	BenchmarkQuadPi_Prediction

Workflow Details

This section provides more detail on the core concepts and components of the identibench workflow.

Benchmark Types

identibench defines two main types of benchmark tasks, specified using different classes:

• Simulation (BenchmarkSpecSimulation):
  • Goal: Evaluate a model’s ability to perform a free-run simulation, predicting the system’s output over an extended period given the input sequence.
  • Typical Input to Predictor: The full input sequence (u_test) and potentially an initial segment of the output sequence (y_test[:init_window]) for warm-up or state initialization.
  • Expected Output from Predictor: The predicted output sequence (y_pred) corresponding to the input, usually excluding the warm-up period.
  • Use Case: Assessing models intended for long-term prediction, control simulation, or understanding overall system dynamics.
• Prediction (BenchmarkSpecPrediction):
  • Goal: Evaluate a model’s ability to predict the system’s output k steps into the future based on recent past data.
  • Typical Input to Predictor: Often involves windows of past inputs and outputs (e.g., u[t:t+H], y[t:t+H]).
  • Expected Output from Predictor: The predicted output at a specific future time step (e.g., y[t+H+k]). The pred_horizon parameter defines ‘k’, and pred_step defines how frequently predictions are made.
  • Use Case: Evaluating models focused on short-to-medium term forecasting, state estimation, or receding horizon control.
• init_window: Both benchmark types often use an init_window. This specifies an initial number of time steps whose data might be provided to the model for initialization or warm-up. Importantly, data within this window is typically excluded from the final performance metric calculation to ensure a fair evaluation of the model’s predictive capabilities beyond the initial transient.

Model Interface (build_model)

The core of integrating your custom logic is the build_model function you provide to run_benchmark.

• Purpose: This function is responsible for defining your model architecture, training it using the provided data, and returning a callable predictor function.

• Input (context: TrainingContext): Your build_model function receives a single argument, context, which is a TrainingContext object. This object gives you access to:

  • context.spec: The full specification of the current benchmark being run (including dataset paths, input/output columns, init_window, etc.).
  • context.hyperparameters: A dictionary containing any hyperparameters you passed to run_benchmark. Use this to configure your model or training process.
  • context.seed: A random seed for ensuring reproducibility.
  • Data Access Methods: Functions like context.get_train_sequences() and context.get_valid_sequences() provide iterators over the raw, full-length data sequences as Sequence(u, y, attrs) named tuples, where u and y are NumPy arrays and attrs is a dict of that file's HDF5 attributes (e.g. the sampling rate fs). Note: You need to handle any batching or windowing required for your specific training algorithm within your build_model function.

• Output (Predictor Callable): build_model must return a callable with the signature model(u, y_init, attrs) that returns the predicted output as a NumPy array of shape (len(u), len(y_cols)):

  • u: the input sequence to predict over — the full test signal for a simulation benchmark, or a single window for a prediction benchmark.
  • y_init: the first init_window ground-truth output samples, provided for warm-up / state initialization.
  • attrs: a dict of the test file's HDF5 attributes (e.g. the sampling rate fs).

  run_benchmark calls this predictor on each test sequence and scores the returned predictions against the held-out targets.

Running Multiple Benchmarks

To evaluate a model across several scenarios efficiently, use the run_benchmarks function:

# Example: Run on a subset of benchmarks
specs_to_run = {
    'WH_Sim': idb.simulation_benchmarks['WH_Sim'],
    'Silverbox_Sim': idb.simulation_benchmarks['Silverbox_Sim']
}

# Assume 'my_build_model' is your defined build function
all_results = idb.run_benchmarks(specs_to_run, build_model=build_frols_model,n_times=3)

all_results

--- Starting benchmark run for 2 specifications, repeating each 3 times ---

-- Repetition 1/3 --

[1/6] Running: BenchmarkWH_Simulation (Rep 1)
  -> Success: BenchmarkWH_Simulation (Rep 1) completed.

[2/6] Running: BenchmarkSilverbox_Simulation (Rep 1)
  -> Success: BenchmarkSilverbox_Simulation (Rep 1) completed.

-- Repetition 2/3 --

[3/6] Running: BenchmarkWH_Simulation (Rep 2)
  -> Success: BenchmarkWH_Simulation (Rep 2) completed.

[4/6] Running: BenchmarkSilverbox_Simulation (Rep 2)
  -> Success: BenchmarkSilverbox_Simulation (Rep 2) completed.

-- Repetition 3/3 --

[5/6] Running: BenchmarkWH_Simulation (Rep 3)
  -> Success: BenchmarkWH_Simulation (Rep 3) completed.

[6/6] Running: BenchmarkSilverbox_Simulation (Rep 3)
  -> Success: BenchmarkSilverbox_Simulation (Rep 3) completed.

--- Benchmark run finished. 6/6 individual runs completed successfully. ---

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style>

	benchmark_name	dataset_id	hyperparameters	seed	training_time_seconds	test_time_seconds	benchmark_type	metric_name	metric_score	cs_multisine_rmse	cs_arrow_full_rmse	cs_arrow_no_extrapolation_rmse
0	BenchmarkWH_Simulation	wh	{}	2406651230	4.944649	1.012850	BenchmarkSpecSimulation	rmse_mV	42.161572	NaN	NaN	NaN
1	BenchmarkSilverbox_Simulation	silverbox	{}	3813113752	2.839149	1.246224	BenchmarkSpecSimulation	rmse_mV	10.732386	8.501941	16.154317	7.5409
2	BenchmarkWH_Simulation	wh	{}	1950649438	4.801520	1.034119	BenchmarkSpecSimulation	rmse_mV	42.161572	NaN	NaN	NaN
3	BenchmarkSilverbox_Simulation	silverbox	{}	1560698088	2.880391	1.217932	BenchmarkSpecSimulation	rmse_mV	10.732386	8.501941	16.154317	7.5409
4	BenchmarkWH_Simulation	wh	{}	3258007268	4.916941	1.021927	BenchmarkSpecSimulation	rmse_mV	42.161572	NaN	NaN	NaN
5	BenchmarkSilverbox_Simulation	silverbox	{}	4194043971	2.937101	1.231710	BenchmarkSpecSimulation	rmse_mV	10.732386	8.501941	16.154317	7.5409

This function iterates through the provided list or dictionary of benchmark specifications, calling run_benchmark for each one using the same build_model function and hyperparameters.

#calculate mean and std of the results
idb.aggregate_benchmark_results(all_results,agg_funcs=['mean','std'])

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead tr th { text-align: left; } .dataframe thead tr:last-of-type th { text-align: right; } </style>

	training_time_seconds		test_time_seconds		metric_score		cs_multisine_rmse		cs_arrow_full_rmse		cs_arrow_no_extrapolation_rmse
	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
benchmark_name
BenchmarkSilverbox_Simulation	2.885547	0.049179	1.231955	0.014147	10.732386	0.0	8.501941	0.0	16.154317	0.0	7.5409	0.0
BenchmarkWH_Simulation	4.887703	0.075912	1.022966	0.010673	42.161572	0.0	NaN	NaN	NaN	NaN	NaN	NaN

Data Handling & Format

Understanding how identibench organizes and stores data is helpful for direct interaction or adding new datasets.

• Directory Structure: Datasets are stored under a root directory (default: ~/.identibench_data, configurable via the IDENTIBENCH_DATA_ROOT environment variable). The structure follows: DATA_ROOT / [dataset_id] / [subset] / [experiment_file.hdf5].
• Subsets: Standard subset names are train, valid, and test. An optional train_valid directory might contain combined data.
• Download & Cache: Data is downloaded automatically when a benchmark requires it and cached locally to avoid re-downloads. The identibench.datasets.download_all_datasets function can fetch all datasets at once.
• File Format: Processed time-series data is stored in the HDF5 (.hdf5) format.
• HDF5 Structure:
  • Each .hdf5 file typically represents one experimental run.
  • Signals (inputs, outputs, states) are stored as separate 1-dimensional datasets within the file, named conventionally as u0, u1, …, y0, y1, …, x0, …
  • Data is usually stored as float32 NumPy arrays.
  • Metadata like sampling frequency (fs) and suggested initialization window size (init_sz) are stored as attributes on the root group of the HDF5 file.
  • Example Structure: my_dataset/ └── train/ └── train_run_1.hdf5 ├── u0 (Dataset: shape=(N,), dtype=float32) ├── y0 (Dataset: shape=(N,), dtype=float32) └── Attributes: └── fs (Attribute: float)
• Extensibility: Adhering to this HDF5 format ensures compatibility when adding new dataset loaders. Helper functions like identibench.utils.write_array facilitate creating files in the correct format.

Understanding Benchmark Results

The run_benchmark function returns a dictionary containing detailed results of the experiment. Key entries include:

• benchmark_name (str): The unique name of the benchmark specification used.
• dataset_id (str): Identifier for the dataset source.
• hyperparameters (dict): The hyperparameters dictionary passed to the run.
• seed (int): The random seed used for the run.
• training_time_seconds (float): Wall-clock time spent inside your build_model function.
• test_time_seconds (float): Wall-clock time spent evaluating the returned predictor on the test set.
• benchmark_type (str): The type of benchmark run (e.g., 'BenchmarkSpecSimulation').
• metric_name (str): The name of the primary metric function defined in the spec.
• metric_score (float): The calculated score for the primary metric on the test set (aggregated if multiple test files).
• custom_scores (dict): Any additional scores calculated by custom evaluation logic specific to the benchmark.
• model_predictions (list): A list containing the raw outputs. For simulation, it’s typically [(y_pred_test1, y_true_test1), (y_pred_test2, y_true_test2), ...]. For prediction, the structure might be nested reflecting windowed predictions.