microstructure-tpw 0.3.0

Ocean microstructure turbulence processing: TKE dissipation and chi from Rockland Scientific instruments

microstructure-tpw

Python tools for reading Rockland Scientific microprofiler data and computing turbulent dissipation rates from VMP (Vertical Microstructure Profiler) and MicroRider instruments.

Overview

microstructure-tpw provides a complete processing pipeline for ocean turbulence measurements from Rockland Scientific instruments equipped with shear probes and fast thermistors (FP07). The package reads proprietary .p binary data files, converts channels to physical units, detects profiles, and computes both the rate of dissipation of turbulent kinetic energy (epsilon) and the rate of dissipation of thermal variance (chi), following the methods described in the Rockland Scientific ODAS MATLAB Library and associated Technical Notes.

• Epsilon (TKE dissipation rate) from shear probe spectra (detailed mathematics), including Goodman coherent noise removal, Nasmyth spectrum fitting, and Macoun & Lueck wavenumber correction.

• Chi (thermal variance dissipation rate) from FP07 thermistor spectra (detailed mathematics), including Batchelor/Kraichnan spectrum models, FP07 transfer function correction, and spectral fitting (iterative Peterson & Fer 2014, the default, or MLE).

The package is organized into six subpackages under odas_tpw:

• rsi — Rockland Scientific instrument I/O, NetCDF conversion, profiles, epsilon/chi orchestration
• chi — Chi (thermal variance dissipation) calculation
• scor160 — ATOMIX shear-probe benchmark processing and shared physics modules
• perturb — Full campaign processing pipeline (trim, merge, calibrate, compute, bin)
• processing — Instrument-agnostic profile processing (top trim, bottom-crash, CT alignment, multi-probe epsilon combining)
• pyturb — Third-party code (oceancascades/pyturb) hosted in-repo; maintained upstream, not refactored here

Installation

pip install -e ".[dev]"    # editable install with dev dependencies
pip install .              # standard install

See docs/installation.md for more options.

Quick Start

Several CLIs ship with the package (rsi-tpw, perturb, perturb-plot, perturb-diag, scor160-tpw, pyturb-cli); the two primary ones, for different workflows:

• rsi-tpw — single-file or short batch processing on the command line.
• perturb — full campaign pipeline driven by a YAML config file (trim → merge → profiles → epsilon → chi → bin → combo), with parallel workers and stage-aware logging. Use this for cruise-scale runs.

The two CLIs use different spectral defaults (rsi-tpw: fft_length=1024 with a 4096-sample dissipation window; perturb: duration-based fft_sec=1.0 — one second, resolved per instrument sampling rate, = 512 samples on a 512-Hz VMP-250 — for both epsilon and chi), so their outputs differ in vertical resolution and noise behavior.

rsi-tpw (single files / short batches)

# Full pipeline: .p files → epsilon → chi
rsi-tpw pipeline VMP/*.p -o results/

# Or run individual stages
rsi-tpw eps VMP/*.p -o epsilon/                       # writes into epsilon/eps_00/
rsi-tpw chi VMP/*.p --epsilon-dir epsilon/ -o chi/    # finds eps_00/ automatically

# Pre-deployment bench test (dummy probes, instrument at rest): raw-count
# figures + spectra + an auto-evaluated Rockland checklist (see docs/rsi-tpw/bench.md)
rsi-tpw bench VMP/*.p -o bench/

Note: eps writes into a hash-tracked subdirectory (epsilon/eps_00/, see output directories). The chi command searches the given --epsilon-dir and its eps_* subdirectories (most recently modified first), and matches both single-profile ({stem}_eps.nc) and per-profile ({stem}_prof001_eps.nc, ...) file names, concatenating the latter along time. Only when no matching epsilon files exist anywhere does chi fall back to Method 2 for that file, with a console warning.

from odas_tpw.rsi.pipeline import run_pipeline
from pathlib import Path

# Full pipeline: .p → profiles → epsilon → chi → binning → combine
run_pipeline([Path("VMP/file.p")], Path("results/"))

# Or use the modular API
from odas_tpw.rsi.dissipation import compute_diss_file
from odas_tpw.rsi.chi_io import compute_chi_file

compute_diss_file("VMP/file.p", "epsilon/")
compute_chi_file("VMP/file.p", "chi/")

Note: get_diss() and get_chi() still work for backward compatibility but are deprecated in favor of run_pipeline() or the modular compute_diss_file() / compute_chi_file() functions.

perturb (campaign-scale batch pipeline)

# Write a template config you can edit (cruise paths, GPS file,
# instrument-specific overrides, parallelism, etc.).
perturb init my-cruise.yaml

# Run the full pipeline — outputs land in <output_root>/{profiles_NN,
# diss_NN, chi_NN, ctd_NN, *_binned_NN, combo}, with a per-run log under
# <output_root>/logs/.
perturb run -c my-cruise.yaml

# Or run individual stages against the same config:
perturb trim -c my-cruise.yaml
perturb profiles -c my-cruise.yaml
perturb diss     -c my-cruise.yaml
perturb chi      -c my-cruise.yaml
perturb bin      -c my-cruise.yaml
perturb combo    -c my-cruise.yaml

See docs/perturb/pipeline.md for the full stage list and docs/perturb/configuration.md for the config schema.

Documentation

rsi-tpw (science library)

Document	Description
CLI Reference	All rsi-tpw subcommands and flags
Configuration	YAML config file format and all parameter defaults
Pipeline	Processing stages and data flow
Python API	Using microstructure-tpw from Python code
Output Directories	Sequential hash-tracked output scheme
Vectorization	Vectorized compute internals
Bench Test	Pre-deployment bench check (bench): quick_bench figures + auto checklist

perturb (batch pipeline)

Document	Description
Pipeline	Batch processing stages and data flow
CLI Reference	All perturb subcommands and flags
Configuration	YAML config file format for perturb
Parallel Scaling	Benchmark results for multi-core scaling
Modules	Module-level reference
pyturb Comparison	Comparison with oceancascades/pyturb

scor160 (ATOMIX benchmark)

Document	Description
CLI Reference	All scor160-tpw subcommands and flags

Shared

Document	Description
Epsilon Mathematics	TKE dissipation algorithm details
Chi Mathematics	Thermal dissipation algorithm details
Mixing Efficiency	Mathematics of the dissipation ratio
Best Practices	Guidance for comparing turbulence measurements
Bibliography	Consolidated references
Installation	Installation options
Changelog	Version history

References

This package is a Python implementation derived from the Rockland Scientific ODAS MATLAB Library (v4.5.1) and associated Technical Notes (TN-028, TN-051, TN-061).

See docs/bibliography.md for the full list of references.

Testing

python -m pytest                          # run all tests
python -m pytest tests/test_epsilon.py    # epsilon pipeline tests only

Development

This project was developed in collaboration with Claude Code.

Citation

If you use this software in published work, please cite it. Citation metadata lives in CITATION.cff — GitHub renders a "Cite this repository" button from it (BibTeX/APA export). Each tagged release is archived on Zenodo with a DOI; cite the concept (all-versions) DOI so the reference always resolves to the latest release. The DOI badge and identifier are added here once the first release is minted.

License

GPLv3 — see LICENSE for details.