dispcraft

Physics-first, then hybrid physics+ML modeling of the Euclid/NISP grism spectral dispersion — from a geometrical-optics instrument model, through per-dataset and joint multi-dataset calibration against ground-test data, to an ML residual corrector and a field-dependent extension, benchmarked against a published reference result (arXiv:2506.08378).

Project status: exploratory research prototype, ongoing — not production-ready. This is a proof-of-concept pipeline, not a finished deliverable. The best model produced so far (hybrid physical + ML residual) is 1.7x worse on y and 8.4x worse on z than the reference paper's own held-out accuracy (see Results below and docs/Status_Report.md §1) — it does not currently reach the accuracy this instrument needs. Development is ongoing; open gaps and next steps are tracked in docs/Status_Report.md §9-10.

If you just want to use the calibrated models (predict where a spectrum lands, given field position and wavelength), see the User Guide — but read the status note above first: these are the best models this exploratory project has produced to date, not models validated as meeting an accuracy requirement. This README covers the project itself.

What's here

• dispcraft/ — the library: instrument optics (optics/), the ground-test forward model and joint-fit machinery (calibration.py), the field-dependent correction tier (field_calibration.py), the 0th-order dispersion model (zeroth_dispersion.py), the Chebyshev polynomial residual alternative (chebyshev_residual.py), data loading (measurement.py), the residual-MLP building block (ml.py), GitLab model-registry helpers (model_registry.py), and the unified 1st-/0th-order prediction entry point (prediction.py).
• scripts/ — register_models.py trains and publishes the recommended residual/NN models to GitLab's model registry (see the User Guide); freeze_stage5_fits.py, freeze_chebyshev_fits.py, and freeze_best_models.py replay already-validated fits to produce the corresponding models/*.toml files.
• models/*.toml — frozen, versioned calibration results (one file per fit). Every tunable physical parameter used in a real result is recorded here, not hardcoded in code or notebooks.
• notebooks/ — one notebook per project stage/phase, prototyping and producing every result before (validated bits) get promoted to dispcraft/.
• data/ — ground-test PSF datasets (docs/internship/2-Intro_data/index.html is the authoritative description of what's in here and how it was built; see also data/README.md).
• docs/ — User_Guide.md (how to use the calibrated models), Status_Report.md (full write-up of methodology and results), reference material (Euclid-NISP-Specs.md, the benchmark paper summary), internship/ (the slide decks, see below), and the source for the published documentation site (mkdocs.yml at the repo root).
• docs/internship/0-Intro_generale/ … 4-Projet/ — the slide decks that define this project's scope and order (see Roadmap below).
• webapp/ — a self-contained, static JS visualization of the physical grism model (no build step, no data dependency) — an interactive, outreach-friendly complement to the notebooks.
• tests/ — one test module per dispcraft/ module.

Roadmap

This project follows a fixed sequence of stages, each driven by a slide deck, plus three further stages added by explicit request once the deck's own scope closed. Status and detailed findings for every stage are tracked in CLAUDE.md; the short version:

Getting started

This project uses pixi for environment management — don't use a bare venv/pip install.

pixi install        # create/sync the environment from pixi.toml/pixi.lock
pixi run test        # run the test suite (pytest tests)
pixi run mlflow-ui    # browse logged experiment runs (sqlite:///mlflow.db)

To work in a notebook or script interactively:

pixi shell
jupyter lab notebooks/

Data files (data/*.csv) are already provided; data/README.md documents how they were extracted from the ground-test database, if you need to regenerate them.

Results at a glance

Final held-out accuracy (mean over the 6 RGS ground-test configs), compared to the reference paper's own held-out (Argon) figure — full detail in docs/Status_Report.md:

This does not yet reach the reference's accuracy (1.7x worse on y, 8.4x worse on z) — closing that gap is ongoing work, not a solved problem. See docs/Status_Report.md §9-10 for the open items.

See the User Guide for how to reproduce and use this (and the BGS / 0th-order counterparts) yourself — as a snapshot of current, unfinished work, not a validated production model.

License

GNU General Public License v3.0 — see LICENSE.