svc-processing 0.1.1

Pure-Python SVC HR-1024i hyperspectral .sig processing pipeline

SIG Processing Pipeline

A pure-Python pipeline for processing SVC HR-1024i field hyperspectral .sig files. Reads raw multi-detector spectra, performs sensor stitching and radiometric correction, applies resolution-matched Gaussian smoothing, and resamples onto a uniform 1 nm grid from 400–2500 nm. Numerically verified against the legacy R/spectrolab reference to better than 1 × 10⁻⁶ absolute reflectance.

This README is the canonical entry point for both humans and LLMs. Every directory in the repo has its own README.md describing its contents in detail. Follow the links below — do not assume anything that is not stated here or in the linked docs.

---

Repository map

Path	Purpose	Read me first
TUTORIAL.md	Beginner walkthrough — starts in the Jupyter notebook (gentle path), with an advanced CLI track.	Start here if new
pyproject.toml	Package metadata and svc-pipeline console script.	This file ↓
pipeline/	Core Python package: CLI, SigFileProcessor, resample_spectra, SVCDataProcessor.	pipeline/README.md
tests/	Pytest suite, including the R-vs-Python parity test.	tests/README.md
config/	Run configs + instrument calibration JSONs.	config/README.md
docs/	Manuscript-grade methods, parity reports, LLM re-test prompt.	docs/README.md
archived_r_scripts/	Frozen R/spectrolab reference (Pipeline A) — kept only for parity verification.	archived_r_scripts/README.md
notebooks/	Demo and visualization notebooks (not on the production path).	notebooks/README.md
naming_ids/	Private CSV lookup tables for grouping scans into samples; only the schema README is tracked.	naming_ids/README.md
FOLDER_STRUCTURE.md	Authoritative tree + reading order.	—
ACKNOWLEDGMENTS.md	Attribution, third-party licenses, and citation.	—
LICENSE	GNU General Public License v3.0.	—

Generated outputs (gitignored): pipeline_outputs/sig_processed/<run>/, pipeline_outputs/sig_resampled/<run>/.

---

Quick start

New to the pipeline? TUTORIAL.md starts with the interactive Jupyter notebook (the gentle path) and has an advanced track for the CLI below.

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev,demo]"

# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline config.json

Outputs land under pipeline_outputs/ by default. See config/README.md for every supported key.

---

Pipeline architecture

   raw *.sig files (sig_input_dir)
            │
            ▼
   ┌────────────────────────────────────┐
   │ Stage 1: SigFileProcessor          │  pipeline/sig_processor.py
   │   - instrument-consistency check   │
   │   - truncate at calibration        │
   │     end-line wavelength            │
   │   - write summary CSV              │
   └────────────────────────────────────┘
            │
            ▼  processed *.sig + *_processed_sig_summary.csv
   ┌────────────────────────────────────┐
   │ Stage 2: resample_spectra()        │  pipeline/resampler.py
   │   - detect sensor segments         │
   │   - guess_splice_at + trim         │
   │   - match_sensors (iter = 1)       │
   │   - smooth_fwhm (k=3 kmeans)       │
   │   - Gaussian resample fwhm=10      │
   │     onto 400–2500 nm @ 1 nm        │
   └────────────────────────────────────┘
            │
            ▼  <run>_merged_spectra.csv
   ┌────────────────────────────────────┐
   │ Stage 3 (post-hoc, optional):      │  pipeline/processor.py
   │   SVCDataProcessor /               │  + notebooks/
   │   SigSpectraAverager — group &     │
   │   average scans into samples.      │
   └────────────────────────────────────┘

The svc-pipeline console script glues Stages 1 and 2 together through pipeline.cli; Stage 3 is invoked from notebooks against the Stage 2 output. The pipeline is single-pass and idempotent per input directory — previous processed .sig files in the target directory are deleted at the start of each run.

For the formal algorithmic spec (every constant, every formula), read docs/supplementary_methods.md. That document — not this README — is the source of truth for what the code does.

---

Running the pipeline

svc-pipeline [config] [options]

Argument	Meaning
config	Run-config JSON (positional, optional; default config/config.json). Bare names resolve under config/, so config.json, config, and config/config.json are equivalent. See config/README.md for schema.
--input-dir <path>	Override sig_input_dir and process only this directory.
--step {1,2,all}	1 = process + summary CSV only; 2 = resample only (requires Stage 1 to have been run); all = both (default).
--verbose	Print INFO/DEBUG logs before and after each stage.

Expected output layout

pipeline_outputs/
├── sig_processed/<input_dir_name>/
│   ├── <truncated *.sig files>
│   └── <input_dir_name>_processed_sig_summary.csv
└── sig_resampled/<input_dir_name>/
    └── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)

summary_csv_name and merged_csv_name in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in config/README.md.

---

Verification (R-vs-Python parity)

The Python resampler is verified against the legacy R/spectrolab script (archived_r_scripts/merge_resample_sig.R). Acceptance threshold: 1 × 10⁻³ absolute reflectance (0.1 %, well below the HR-1024i radiometric noise floor). Historical results:

Dataset	Samples	Max abs diff	Mean abs diff
Silver instrument (Serial 1202103)	66	1.10 × 10⁻⁶	4.0 × 10⁻⁸
Bronze instrument (Serial 2212118), a4any_sb_2025-cn_ch-svc-aviris_bottom	15	9.4 × 10⁻⁷	3.4 × 10⁻⁸

Run the parity test (skipped automatically without reference data):

pytest tests/test_resampler_parity.py \
    --r-reference-csv=/path/to/r_output/merged_spectra.csv \
    --r-input-dir=/path/to/processed_sig_files/

To re-run parity on a new dataset, hand the prompt at docs/parity_retest_prompt.md to any capable coding LLM. Full details in tests/README.md and docs/README.md.

---

Public Python API

The most-used entry points (all importable from their concrete modules — pipeline/__init__.py is intentionally empty):

from pipeline.sig_processor import SigFileProcessor   # truncation + instrument inspection
from pipeline.resampler      import process_sig_file, resample_spectra
from pipeline.processor      import (
    SVCDataProcessor,      # chainable load/group/average
    SigSpectraAverager,    # facade — pass a DataFrame, get aggregated DataFrame back
    GroupSpec,             # GroupSpec.from_csv("naming_ids/<file>.csv")
    find_spectra_by_name,  # cross-DataFrame name search
)

Detailed signatures and behavioural notes in pipeline/README.md.

---

For LLMs working in this repo

1. Read FOLDER_STRUCTURE.md first — it lists every directory and the reading order.
2. Before modifying pipeline/resampler.py, re-read docs/supplementary_methods.md. The constants _FWHM_NM, _SIGMA_NM, _INTERP_WVL, _FIXED_SENSOR, _BAND_MIN, _BAND_MAX, and the algorithm steps are load-bearing for the parity claim. Changing any of them requires re-running the parity test and writing a new docs/parity_<dataset>_<date>.md.
3. The R script at archived_r_scripts/merge_resample_sig.R is frozen. Treat it as a behavioural reference, not as live code to edit.
4. Never commit machine paths or private data. Use the placeholders already present in config/config.json and the notebooks.
5. Run pytest after any change to pipeline/. The parity test will skip if reference data is unavailable; the rest of the suite still runs.

---

Demo Notebook

The demo notebook uses an external 15-file .sig artifact because raw headers contain GPS/location metadata. Prepare local demo data with:

python3 scripts/prepare_demo_data.py \
  --source-dir data/a4any_sb_2025-cn_ch-svc-aviris_bottom

The notebook is config-driven: edit the settings cell (DATA_FOLDER, OUTPUT_FOLDER, INSTRUMENT) to run it on your own data and instrument. See notebooks/pipeline_demo/README.md.

---

Requirements

Python 3.11 is the supported runtime. Runtime, demo, and development dependencies are declared in pyproject.toml.

• numpy, scipy, pandas — numerical core.
• matplotlib — demo notebook plotting.
• pytest>=8.3.0 — test runner for local verification.

Install with python -m pip install -e ".[dev,demo]" for local development and use svc-pipeline = "pipeline.cli:main" as the command-line entry point.

R is not required for the production pipeline. It is needed only to regenerate Pipeline A parity references; see archived_r_scripts/README.md.

---

License

Released under the GNU General Public License v3.0 © 2026 Cole Regnier (regs08, [e-mail]), Gold Lab (GrapeSPEC project), Cornell University. The package uses the SPDX expression GPL-3.0-only, chosen for compatibility with the GPL-3 spectrolab reference implementation.

---

Acknowledgments & Citation

This project is an independent reimplementation of the spectrolab algorithm (no source copied) and builds on prior field-spectroscopy work. See ACKNOWLEDGMENTS.md for full attribution, third-party licenses, and how to cite this software and spectrolab.