modacor 1.0.0

A new modular data correction software for any neutron or X-ray technique that produces 1D or 2D scattering, diffraction, or imaging data.

MoDaCor (v1.0.0)

Overview

New modular data corrections for any neutron or X-ray technique that produces 1D or 2D scattering/diffraction/imaging data.

Summary

MoDaCor is a library for traceable, stepwise data corrections, with propagation of units as well as (multiple) uncertainties. The data sources can be files or data streams. The correction follows an optionally branching and merging graph that is configured per application. The computational steps can reuse pre-calculated information produced during a first run to minimise unnecessary overhead. The modular approach allows detailed introspection into the effect of each step on the data, units, and uncertainties.

This implements modular data correction workflow concepts discussed in https://doi.org/10.1107/S1600576717015096. It can be used either directly or as a reference to validate faster, more integrated data-correction implementations.

Installation

pip install modacor

Optional extras

Support for the Tiled-backed IoSource is provided via an optional dependency set. Install it with:

pip install modacor[tiled]

To install the in-development version:

pip install git+https://github.com/BAMresearch/MoDaCor.git@main

Documentation structure

The published documentation at https://BAMresearch.github.io/MoDaCor is organised around three main tracks:

• Getting started – a Quickstart walkthrough that runs a sample pipeline with the bundled MOUSE dataset and highlights pipeline tracing.
• Pipeline operations – guidance for building and maintaining YAML pipeline configurations, understanding branching/merging graphs, and using tracing exports.
• Extension development – tutorials for creating new correction modules, tests, and IO sources, plus an auto-generated reference page per module.

Instrument-specific walkthroughs (for example the MOUSE and SAXSess instruments) show how to combine these resources for real experiments, with links to example pipelines and metadata files.

Planned documentation stubs now live in the repository to signal the intended structure:

• docs/getting_started/ – Quickstart guides (populated with quickstart.md).
• docs/pipeline_operations/ – pipeline basics, configuration reference, and tracing/debugging placeholders.
• docs/extending/ – module-author guide, IO extension notes, and contribution checklist placeholders.
• docs/examples/ – instrument-specific walkthrough placeholders for MOUSE and SAXSess pipelines.

These pages currently contain explicit TODO notes and will be expanded during the documentation refresh.

Documentation contributors can follow docs/README.md for local build instructions and authoring notes.

Development

For coding contributions, we strongly recommend:

• using flake8 and/or black for consistent formatting;
• writing tests for every added functionality to encourage test-driven development practices.

Testing

See which tests are available (arguments after -- get passed to pytest which runs the tests):

tox -e py -- --co

Run a specific test only:

tox -e py -- -k <test_name from listing before>

Run all tests with:

tox -e py

Package Version

Get the next version number and how the GIT history would be interpreted for that:

pip install python-semantic-release
semantic-release -v version --print

This prints its interpretation of the commits in detail. Make sure to supply the --print argument to not raise the version number which is done automatically by the release job of the GitHub Action Workflows.

Project template

Update the project configuration from the copier template and make sure the required packages are installed:

pip install copier jinja2-time
copier update --trust --skip-answered

Changelog

%

v1.0.0 (2025-06-16)

Unknown

• fix pytest warning: rename TestProcessingStep -> TESTProcessingStep (b19e701)

• adapt PoissonUncertainties unit test to use processing data (86c719b)

• modification of the poisson uncertainties module for pipeline use (ce86eae)

• pipeline with PoissonUncertainties step is now running (594315b)

• adjust names of bundles and BaseData keys to contain "signal" (e8c342f)

• add license text to license.py (fdd7ddf)

• add to last commit (648cb25)

• adding a central license to shorten headers (2f88d39)

• begin adapting the calculate function (7cf56a1)

• update pipeline integration test to use new ProcessStep (61724fd)

• update to new BaseData unit handling (06f8763)

• use modacor's unit registry (540ce4a)

• adapt PoissonUncertainties unit test to use processing data (bf20968)

• modification of the poisson uncertainties module for pipeline use (ee3e940)

• pipeline with PoissonUncertainties step is now running (663fee6)

• adjust names of bundles and BaseData keys to contain "signal" (ff9a21a)

• begin adapting the calculate function (cc5f122)

• update pipeline integration test to use new ProcessStep (aec63ce)

• update to new BaseData unit handling (ebdceeb)

• use modacor's unit registry (5fe0eda)

• arithmetic: data is divided by normalization (98f9389)

• BaseData no longer has a .data property (5a53260)

• remove test of display_data property: display_units not defined (19d0a66)

• basedata: use simplified call signature (619368a)

• import uncertainties.unumpy for variance calculation tests (7a55d68)

• add license text to license.py (0f5cfb9)

• add to last commit (441068d)

• adding a central license to shorten headers (939516c)

• add note to self: need ProcessStep registry to populate pipeline (b8ba70b)

• test and draft config format for pipeline import from yaml (4a40871)

• initial yaml loader for pipelines (a12d58e)

• add hash function to make test run (already in draft PR #33) (4e03a00)

• add pyyaml to requirements for yaml definition of pipelines (0f76ece)

• Delete src/modacor/modules/base_modules/poisson_uncertainty.py (09def97)

• Updating the ProcessStep definition (#34) (605c3f6)

• changing unit handling to central pint unit registry (ee3f2f5)

• adding a multiply by variable processing step (fa04cd7)

• import central unit registry (cfdd867)

• change internal_units to signal_units and removing ingest and display units (adfa551)

• minor polishing (e9f9083)

• provide azimuthal variance (93d0dae)

• First uncertainties commit, test is broken (29bec04)

• fixing error (986f10b)

• Starting on Poisson Testing (04eaa24)

• implement the test, still broken (2b787b7)

• polish import_tester (683a0f8)

• make an integrated dataset behave like a normal one (49e8412)

• feed-back of flake8 (07cdc34)

• Allow longer lines (5b74de3)

• configuration for flake8 (dc45d13)

• increase line length (d56fc36)

• fix PoissonUncertainties module: needs to return outputs (4e4eeea)

• use smaller array for testing (2b57000)

• test runs until PoissonUncertainties are actually calculated (7c91102)

• prepare test running PoissonUncertainties module in a pipeline (537ccdf)

• adapt: DataBundle no longer has a data attribute (7a33677)

• first integration test: run pipeline with dummy processsteps (58e1c5d)

• add hash function to ProcessStep (15125a9)

• remove DataBundle from the arguments for running the pipeline (def0dba)

• fixing file reference in documentation (5d337fd)

• removing unused fields (4b12b8f)

• Tidy up (312cb82)

• HDF Loader Tests Now Passing (d17e05d)

• adding a new poisson uncertainties estimator (22c2233)

• Delete src/modacor/dataclasses/pipelinedata.py (5045ae9)

• renaming pipelinedata to processingdata (be76a32)

• Removing an outdated module (cbd7f87)

• Updated formatting with black, flake8 and isort (c89718b)

• add init.py in runner submodule (d995aff)

• modify branch addition methods to operate on another Pipeline (3dcdce7)

• use the same syntax for incoming and outgoing branch addition (ee8ada7)

• add separate methods for adding in- and outgoing branches (a635247)

• fix reference to self (06c2173)

• prepare running the pipeline - not yet tested (f544fea)

• add functionality for branching the pipeline, with simple tests (e8093f8)

• test addition of a simple hashable object (e9bdcbf)

• initial pipeline draft based on graphlibs TopologicalSorter (7c5cfc5)

• fix a bunch of tests (503d1e0)

• flake8 config file (2d34c5d)

• cleanup of databundle and adding pipelinedata (2015907)

• adding a header and cleanung up datbundle. (7c4e7e3)

• renaming raw_data to signal in BaseData (2f6429f)

• HDF Testing (d874571)

• HDF IO Test Commit (920b02f)

• Update pyproject.toml (7f8d1c7)

• Update pyproject.toml (acf17b9)

• Update pyproject.toml (70eed8a)

• fix importing BaseData (a104dce)

• work on integration (125dacd)

• Attempt to fix tox.ini (9a25376)

• Removed unused imports (c5bf606)

• Updating package metadata (eb4b8c8)

• Added tests for io sub-package (e0f3218)

• WIP on azimuthal integration (ca15a97)

• updating the pre-commit config (6e247b4)

• updated BaseData (72912e7)

• modifications to basedata and databundle (a50204c)

• clean-up (5435da4)

• put back scalers are normalization factor (8ad59ff)

• updating databundle (f56265a)

• Modified HDF Loader (1fa7fd0)

• Start of HDF5 Loader (3397b14)

• Error Propagation (9e17100)

• Adding units to normalization (1ef17ef)

• Requirements Changes (d7041d5)

• Updated requirements (1c2b91c)

• Updates to init files (663ede2)

• Added file headers to io and updated definitions (fa7996b)

• Fixing linting (6bd4918)

• Populated io (4faa1db)

• Error Propagation (6f2854c)

• Error Propagation (d0b87f2)

• Error Propagation (d0a040b)

• Missed comma (09fecf1)

• Updates to the logger (f09b5df)

• fix renaming of ProcessStep (0de2352)

• fix relative import error (6fa77f4)

• dynamic requirements.txt (90b233b)

• fix scatteringdata duplicate and databundle import (0a02cbd)

• fix imports (886002c)

• Modified imports (7993f93)

• Modifed ProcessStep (22d05d1)

• Tox modifications (b8ee1aa)

• databundle (1d18bca)

• One step further (c9e0fd0)

• Changes for tox (142582e)

• fix import all modules (5561826)

• fix all relative imports in tests (675e504)

• move test into project ... (ad171a4)

• License modification (5d331e4)

• ScatteringData to DataBundle (108e8a4)

• Small updates (d176c49)

• Updated processstep naming and content (7ae6ca7)

• adding the flow charts (e07cb55)

• typos (fb85395)

• migrate dask array to numpy (62fd3b2)

• implement base dataclass (8049030)

• implement normalization and variances in dict (f41c0ab)

• modified to use the actual datapoint in basedata now. (d9bd951)

• Getting towards a working concept (0772d80)

• something like this (dfd97bf)

• Getting towards a proof-of-principle (2a77207)

• more experimentation to explore delayed execution (5f365a1)

• Making changes to classes based on experimentation... (1710f40)

• Added some flow diagrams to help clarify... (b17ff22)

• Update README.rst (1e658e2)

• Changed name of note to step_note (accd141)

• fixed tests and validator. (4b768e6)

• Changed naming slightly, and added a message handler placeholder (0880270)

• changes broke tests (e81e369)

• test implementation of basedata and processstep dataclasses (eb6dcc4)

v0.0.0 (2025-02-13)

Unknown

• Initial commit (4753f2a)