Navette - Weaving thin-film systems that perform

Navette is a high-performance, physically rigorous 1D optical engine designed for the simulation of light propagation in stratified media. Built on a modern Scattering Matrix (S-matrix) architecture, it offers a numerically stable and vectorized alternative to traditional Transfer Matrix Methods (TMM).

1. Unconditional Numerical Stability

Traditional TMM suffers from numerical divergence (exponentially growing evanescent waves) when dealing with thick layers or highly absorbing materials. Navette utilizes the Redheffer Star Product to propagate scattering matrices, ensuring that all matrix elements remain bounded and physically meaningful, regardless of layer thickness.

2. High-Concurrency Performance

As a Principal Performance Engineer, you need tools that scale. Navette is built for speed:

• Parallel Execution: Utilizes Rust + rayon data-parallelism across wavelengths/angles to saturate all available CPU cores.

• Vectorized Engine: Operations are performed across the entire (wavelength × angle) coordinate space in a single pass, eliminating Python's loop overhead.

• Memory Efficiency: Collapses multi-layer stacks into a compact global S-matrix to minimize cache misses.

Measured, not claimed. All numbers below are --release builds on a 32-core Windows box, taken by the scripts named beside them under validation/benches/; each of those scripts refuses to run against a debug build (see the profile note under Getting started). "vs numba" compares against the numba reference implementation this engine replaced.

Two caveats kept deliberately visible: small grids (under ~2 000 points) are dispatch-bound and still behind the numba kernel, and bench_refold does not exercise the needle insertion path, which is the expensive part of synthesis.

3. Partial Coherence Support

Real-world systems often involve thick substrates (like a 1mm glass slide) where phase information is lost. Navette features a Hybrid Coherence Engine:

• Coherent Blocks: Preserves phase for thin-film interference.

• Incoherent Interfaces: Switches to intensity-based propagation for thick layers, preventing the "unphysical ringing" caused by assuming perfect coherence across a macroscopic substrate.

4. Advanced Physics Modeling

Navette goes beyond simple Fresnel equations to provide research-grade accuracy:

• Interface Roughness: Implements the Névot-Croce model, providing superior accuracy for high-frequency or X-ray reflectometry compared to standard Gaussian approximations.

• Ellipsometric Rigor: Outputs (Ψ,Δ) parameters that strictly follow the Azzam & Bashara convention, ensuring direct compatibility with commercial ellipsometers (e.g., Woollam, Horiba).

5. Automated Coating Design

Navette doesn't just simulate — it synthesizes, with the classic needle method running natively on the same engine:

• Needle Insertion: Probes every candidate position with an infinitesimal test layer and inserts real material where the merit function improves most — the Tikhonravov needle algorithm, merit-driven and target-aware.

• Thickness Optimization: Levenberg-Marquardt refinement over free layers with bounds and clamping, interleaved with insertion passes and impact-ranked cleanup (merge, thin-layer removal, re-optimization).

• Multi-domain Targets: One joint merit over spectral, angular, and CIE color demands — multiple angles, illuminants with own-white metamerism control, and per-target wavelength windows — all folded into the needle gradient with analytic chain-rule terms, so a single run designs for daylight and showroom light at once.

• Graded Media: Mixture gradients are a first-class layer property, not a hand-built stack of sublayers. A gradient spec names the two endpoint materials, the EMA kernel (any of the six in-tree mixing rules) and the profile mode — FixedSpan, where composition runs f_start → f_end across the film, or RateCapped, a thickness-relative slope clamped to [f_min, f_max] so a thick film saturates into a pure-material tail. Single-material drift (InhMode) has the same two modes. Graded layers still serve as pinned background (substrate diffusion gradients, rugate foundations) while the needle designs around them — what is new is that a graded span's thickness can itself be an optimizer parameter.

• One Physical Layer, One Parameter: A profiled film is one layer, so the optimizer moves its total thickness as a single number rather than one parameter per sublayer, and a thickness-relative profile is re-derived at the construction points when its span rescales instead of carrying stale nk. For the same reason the thin-layer floor, the thickness ceiling and the layer budget are span quantities: they count design layers, not solver rows.

• Thin-Layer Policy: A film driven below the minimum thickness you can actually deposit need not be deleted. thin_layer_policy decides: 'remove' (the default, and bit-for-bit the historical behaviour), 'clamp_up_final' — the search runs exactly as before and only the final pass lifts a surviving sub-minimum film to the floor — or 'clamp_up_always'. A film that carries an interface slice clamps up like any other single layer.

• Saved Designs Stay Readable: state files carry a schema version and the reader accepts a range ([1, 2]), not a point. A state written by an older build is reconstructed from its defaults rather than refused; one written by a newer build is refused with a message that says so, because the remedy there is upgrading, not hand-editing the file.

Technical Specifications

Project layout

Navette/
├── Cargo.toml                # Rust workspace (cargo check/test --workspace)
├── pyproject.toml            # maturin project: builds the `navette` wheel (src layout)
├── src/navette/              # unified Python package
│   ├── __init__.py           # version + public surface
│   ├── color/                # wrapper over native `navette._color`
│   ├── interpolate/          # wrapper over native `navette._interpolate`
│   ├── smatrix/              # ScatterMatrix + needle (native `navette._smatrix`)
│   ├── spectralweave/        # weavers + merit (native `navette._spectralweave`)
│   ├── materials/            # dispersion models (native `navette._materials`)
│   ├── _*.py                 # shims re-exporting the `navette._navette` submodules
│   ├── structure/            # stacks, architect (native model + thin wrappers)
│   ├── synthesis/            # needle pipeline driver (native DesignStack)
│   ├── config/               # native-validated holders, program documents
│   └── data/CIE/             # bundled reference spectra
├── rust/                     # Rust sources: one engine crate + bindings
│   ├── navette/              # pure-Rust engine (color/interpolate/materials/
│   │                         # smatrix/spectralweave/structure modules;
│   │                         # published as `navette` on crates.io)
│   └── navette-py/           # PyO3 aggregator -> navette._navette (one wheel)
├── validation/               # tests, parity, benches, goldens + references (see validation/README.md)
├── tools/check_exposure.py   # bidirectional exposure lint (CI)
├── examples/  docs/plans/  benchmarks/

Install & build

# Single aggregated native extension (navette._navette, all engines):
maturin develop --release
# checks
cargo check --workspace
cargo test --workspace     # everything (needs Python for binding crates)
cargo test-pure            # pure-Rust gate (no Python needed)
cargo fmt --all            # rustfmt defaults; CI fails on any diff
python tools/check_toolchain.py   # is your clippy as new as CI's?
pytest validation

Lint on the toolchain CI uses. cargo clippy only reports the lints its own version knows. Between 0.6.13 and 0.6.30 the local toolchain was one minor version behind CI's stable, the local run was clean, and CI was red for 17 consecutive pushes on a lint the local clippy did not have. tools/check_toolchain.py fails when that gap reopens.

Run this once per clone so git blame skips the tree-wide reformat commit (0.6.32) and points at whoever actually wrote each line:

git config blame.ignoreRevsFile .git-blame-ignore-revs

Always pass --release. Plain maturin develop builds with the dev profile: the extension imports and computes correctly, but runs several times slower, so every timing taken against it is meaningless. This is not hypothetical — a whole round of committed benchmark results (and the conclusions drawn from them) had to be discarded for exactly this reason. navette.build_profile() reports which profile is installed, and the benches under validation/benches/ exit rather than time a "debug" one.

Architecture: Rust core, Python addon

All logic and all validation live in the navette Rust crate — it runs fully standalone (file → design → solve → report, no interpreter). The Python package is a thin addon: validated config holders, YAML→dict parsing, result reshapes, and re-exports. Conversely every feature-level Rust function is exposed via PyO3, so Python can drive the whole engine. tools/check_exposure.py enforces this both ways in CI (see docs/plans/exposure_audit.md).

CI

.github/workflows/ci.yml runs on every push and pull request: cargo test --workspace, a zero-compiler-warnings check (-D warnings), pytest validation on Windows and Linux (pinned runner images, so the recorded fingerprints stay platform-stable), four blocking lints — exposure, CIE sync, .pyi surface sync, and message hygiene (tools/check_message_whitespace.py: space runs and console-unencodable characters inside message literals) — and an assertion that the installed extension is a release build. cargo clippy -D warnings (since 0.6.6) and cargo fmt --all --check (since 0.6.32) are blocking; nothing in the workflow is advisory any more.

Layout notes

• rust/ holds the Cargo workspace (the single navette engine crate plus the navette-py PyO3 aggregator) — the idiomatic Rust layout, publishable to crates.io.
• src/navette/ is the Python package in src-layout — the idiomatic Python layout, which maturin detects automatically for mixed projects.

Release & publish

Release automation: tag vX.Y.Z (must match pyproject.toml, workspace Cargo.toml, its internal navette dependency, __about__.py and both Cargo.lock entries — all six enforced by CI) → .github/workflows/release.yml builds wheels (Linux/Windows/macOS) and publishes to PyPI (trusted publisher) + crates.io (token), leaf crates first.

maturin build --release   # -> target/wheels/navette-0.6.44-*.whl (single wheel, all engines)

Optimizer backends

LmConfig(optimizer=...) chooses which least-squares solver runs. navette._smatrix.available_optimizers() reports what the installed wheel actually has; a name it lacks is refused with the rebuild command, never quietly replaced by a different solver.

Optional cargo features

Off by default, so a standard wheel pulls no extra dependencies.

maturin develop --release --features opt-minpack-lm
maturin develop --release --features opt-argmin

Manual fallback: cargo publish -p navette; maturin upload target/wheels/navette-0.5.0-*.whl.