🌊 caldip
Calibration dip analysis for oceanographic instruments
Caldip provides tools for processing, analyzing, and visualizing data from calibration dips performed with CTD profiles and multiple instrument types including MicroCATs and RBR thermistors. The package enables comparison and calibration analysis between reference CTD measurements and deployed oceanographic instruments during calibration dips.
🚀 Quick Start
Installation
pip install caldip
# or, with uv:
uv pip install caldip # into the active environment
uv tool install caldip # as a standalone command-line tool
For development (editable install with the docs and test tooling):
git clone https://github.com/ocean-uhh/caldip.git
cd caldip
pip install -e ".[dev]"
# or, with uv:
uv pip install -e ".[dev]"
Basic Usage
# Scaffold a configuration file for a new cast directory
caldip init data/proc_calib/msm142_2026/cal_dip/castM4/
# Pre-process the CTD file (normalize, wild-edit, 1 Hz resample, save NetCDF)
caldip ctd data/proc_calib/msm142_2026/cal_dip/castM4/
# Generate interactive plots
caldip plot data/proc_calib/msm142_2026/cal_dip/castM4/
# Generate statistics (use secondary CTD sensor as reference)
caldip stats data/proc_calib/msm142_2026/cal_dip/castM4/ --ctd-sensor 2
# Build a per-cruise HTML report from the stats output
caldip report data/proc_calib/msm142_2026/cal_dip/
📁 Project Structure
caldip/
├── caldip/ # Python package
│ ├── __init__.py # Public API: plot, stats, find_bottle_stops, load_config
│ ├── core.py # Core algorithms
│ ├── readers.py # Data loading and normalization
│ ├── scaffold.py # Stub YAML generator (caldip init)
│ ├── tools.py # Shared utilities
│ ├── parameters.py # Canonical variable names (synced from seasenselib)
│ ├── sbe_hex_reader.py # SBE hex format reader
│ ├── _plot.py # Plotly implementation (internal)
│ ├── _writers.py # Output formatting and NetCDF saving (internal)
│ ├── report/ # Per-cruise HTML report builder (caldip report)
│ └── cli/ # CLI entry points
│ ├── __init__.py # `caldip` dispatcher
│ ├── init.py # `caldip init` subcommand
│ ├── ctd.py # `caldip ctd` subcommand
│ ├── instrument.py # `caldip instrument` subcommand
│ ├── plot.py # `caldip plot` subcommand
│ ├── stats.py # `caldip stats` subcommand
│ └── report.py # `caldip report` subcommand
├── scripts/ # Batch and utility scripts
└── pyproject.toml # Package configuration and dependencies
🏗️ Workflow
The recommended workflow runs each step once per cast:
Step 1 — Scaffold the configuration
caldip init data/proc_calib/cruise_year/cal_dip/castM4/
Scans the cast directory for CTD and instrument files and writes a stub
castM4.caldip.yaml. Edit the file to set deployment_time, recovery_time,
and any clock_offset values before proceeding.
Step 2 — Pre-process the CTD
caldip ctd data/proc_calib/cruise_year/cal_dip/castM4/
Normalizes variable names, applies wild-edit spike removal, resamples to 1 Hz,
and saves the result as <ctd_file_stem>.nc alongside the source CNV file.
Also generates a comparison plot of raw vs processed data. Subsequent
caldip plot and caldip stats calls load this .nc file automatically.
Step 3 — Cache instrument data (optional but recommended)
# Save all records to NetCDF for fast re-use
caldip instrument castM4/castM4.caldip.yaml --serial 013874
caldip instrument castM4/castM4.caldip.yaml --serial 13840
Produces caldip_{type}_{serial}_raw.nc (full normalized record) and
caldip_{type}_{serial}_use.nc (trimmed to deployment_time/recovery_time).
If you skip this step, caldip plot and caldip stats will create these files
automatically on the first run.
Cache priority: caldip plot and caldip stats load from _use.nc if it
exists, otherwise from _raw.nc, otherwise from the source file. To force a
re-read from the source (e.g. after editing the source file or changing
clock_offset), either re-run caldip instrument for that serial or delete
the cached .nc files:
# Force regeneration of one instrument
caldip instrument castB1/castB1.caldip.yaml --serial 26269
# Or delete the cache files manually
rm castB1/caldip_microcat_26269_raw.nc castB1/caldip_microcat_26269_use.nc
Step 4 — Analyse
# Interactive plot of all instruments vs CTD
caldip plot castM4/castM4.caldip.yaml
# Per-bottle-stop statistics
caldip stats castM4/castM4.caldip.yaml --ctd-sensor 2 -o outputs/
Once a cruise's casts have been processed, build a browsable report of them all:
# Per-cruise HTML report (index + per-cast pages) from the stats output
caldip report data/proc_calib/msm142_2026/cal_dip/
Configuration file
caldip init writes a stub; here is a minimal example of the finished file:
name: castM4
ctd_file: 'msm_142_1_032_1sec.cnv'
ctd_sensor: 2
deployment_time: '2026-04-03T03:05:36'
recovery_time: '2026-04-03T05:08:24'
directory: 'data/proc_calib/msm142_2026/cal_dip/castM4/'
instruments:
- position: '1'
serial: 013874
label: 'TR1050'
instrument: tr1050
file_type: 'rbr-matlab-legacy'
filename: '013874_20260403_1302.mat'
clock_offset: 7175 # seconds; positive = add to instrument time
- position: '2'
serial: 13840
label: 'SBE37'
instrument: MicroCAT
file_type: 'sbe-cnv'
filename: '13840_cal_dip_data_time.cnv'
Instrument selection
Two optional keys control which instruments are recorded and processed:
# All instruments physically on the rosette during this cast (serial numbers only).
# "Dipped" means the instrument went into the water, even if no data file is available yet.
# This is a record-keeping field — it has no effect on processing.
dipped_serials: [2942, 25586, 2941, 5367, 7507, 3026, 26202, 26269]
# Subset of serials from `instruments` to actually load and process.
# Omit this key (or leave it empty) to process all entries in `instruments`.
# Use it to isolate one instrument, or to skip serials whose data files are missing.
process_serials: [2942, 2941, 7507]
If process_serials is absent, all entries in instruments are loaded. If it is
present, only those serials are loaded — useful for a quick single-instrument check
or when some data files have not yet been downloaded.
MicroCAT and CTD data must be in *.cnv (or *.asc) format — *.hex must be
converted to *.cnv using SBEDataProcessing first. For RBR data, *.rsk is
supported; the older *.hex format must be converted to legacy *.mat using
Ruskin.
🔬 Core Algorithm: Bottle Stop Detection
The bottle stop detection algorithm in caldip/core.py:find_bottle_stops():
- Pressure variable detection — searches for
pressurein the dataset - Search region — starts detection from max_pressure − 10 dbar
- Rate-based detection — 60-second sliding window; flags periods where pressure change rate < threshold (default: 10 dbar/min)
- Boundary refinement — refines start/end to within 2 dbar of the median pressure
- Merging — merges stops within 10 samples of each other
- Duration filter — keeps only stops ≥ minimum duration (default: 180 s)
🎯 Supported Instruments
| Instrument | Formats | file_type |
|---|---|---|
| Sea-Bird CTD (SBE9) | .cnv |
ctd-cnv |
| Sea-Bird MicroCAT (SBE37) | .cnv, .asc |
sbe-cnv, sbe-asc |
| RBR solo/duet/concerto | .rsk |
rbr-rsk |
| RBR (legacy Ruskin export) | .mat |
rbr-matlab-legacy |
📊 CLI Reference
| Command | Description |
|---|---|
caldip init <dir> |
Write a stub .caldip.yaml for a cast directory |
caldip ctd <yaml> |
Normalize, wild-edit, resample CTD; save .nc + comparison plot |
caldip instrument <yaml> --serial N |
Save one instrument to _raw.nc / _use.nc and generate a time-series plot |
caldip plot <yaml> |
Interactive Plotly plot of instruments vs CTD |
caldip stats <yaml> |
Per-bottle-stop statistics; write CSV files |
caldip report <dir> |
Per-cruise HTML report (index + per-cast pages) from stats output |
Both caldip ctd and caldip instrument accept --format to control what outputs are
produced. Valid values (comma-separated): nc and html.
# CTD: save NetCDF only, skip the plot
caldip ctd castB1/castB1.caldip.yaml --format nc
# CTD: regenerate the comparison plot without re-saving the NetCDF
caldip ctd castB1/castB1.caldip.yaml --format html
# instrument: save NetCDF files only (no plot)
caldip instrument castB1/castB1.caldip.yaml --serial 7507 --format nc
# instrument: plot only, skip NetCDF
caldip instrument castB1/castB1.caldip.yaml --serial 240230 --format html
The default for both commands is --format nc,html (produce both outputs).
All subcommands accept --help for full option details.
🐍 Python API
For use in Jupyter notebooks or custom scripts. Prefer import caldip over
from caldip import plot, stats to avoid shadowing common names like
scipy.stats.
Typical notebook workflow
import caldip
from pathlib import Path
# 1. Load configuration
config = caldip.load_config("castM4/castM4.caldip.yaml")
data_dir = Path("castM4/")
# 2. Load instrument and CTD data
instruments = caldip.load_instruments_from_config(config, data_dir)
reference = caldip.load_reference_data(config, data_dir)
# 3. Trim to deployment window (optional — uses deployment_time/recovery_time from YAML)
instruments, reference = caldip.trim_to_deployment(instruments, reference, config)
# 4. Interactive plot (opens in browser or notebook)
fig = caldip.plot(instruments, reference, config=config)
fig.show()
# 5. Per-bottle-stop statistics
df = caldip.stats(instruments, reference, config)
print(df[["serial", "bl_press", "temp_diff", "cond_diff"]])
Lower-level access
import caldip
# Detect bottle stops from a CTD xarray Dataset
reference = caldip.load_reference_data(config, data_dir)
ctd_ds = list(reference.values())[0]["data"]
stops = caldip.find_bottle_stops(ctd_ds)
for stop in stops:
print(f" {stop['pressure']:.0f} dbar — {stop['duration_seconds'] / 60:.1f} min")
See the API documentation for full details.
🛠️ Dependencies
Core Scientific: numpy, pandas, xarray, scipy, netcdf4
Configuration: pyyaml
Visualization: plotly
Oceanographic Data: seabirdscientific, seasenselib (optional)
🔧 Advanced Usage
Custom bottle stop detection
caldip plot config.yaml --threshold 25.0 --min-duration 90
caldip plot config.yaml --no-bottle-stops
Override CTD sensor for a quick check
# Use primary CTD sensor
caldip stats config.yaml --ctd-sensor 1
# Use secondary CTD sensor (overrides YAML ctd_sensor setting)
caldip stats config.yaml --ctd-sensor 2
Batch processing
bash scripts/generate_all_caldip_plots.sh
The package is designed for oceanographic researchers performing instrument calibration checks and requires familiarity with CTD operations and oceanographic data formats.
🙏 Acknowledgements
Development was assisted by Claude Code (Anthropic) and GitHub Copilot code review.
Metadata
Release files for caldip 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| caldip-0.2.1.tar.gz | 138.1 kB | With provenance | Details for caldip-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| caldip-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for caldip-0.2.1-py3-none-any.whl |
Total release size: 260.7 kB
Release files / caldip-0.2.1.tar.gz
| Download URL | caldip-0.2.1.tar.gz |
|---|---|
| Size | 138.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
48fbb0531434f68104bec04f12af61bb8d65c304f9abd9902e88f6e9bd34b4b7
|
|
BLAKE2b-256 checksum How to use checksums |
89d8afe46a9d52f3e89ba01c1cbe3874520feadf0dd740e87eefe55bdcb103de
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logRelease files / caldip-0.2.1-py3-none-any.whl
| Download URL | caldip-0.2.1-py3-none-any.whl |
|---|---|
| Size | 122.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f947419b3970bb279e95f9b28564535453e2456644b45c2e5a07e77bf7371421
|
|
BLAKE2b-256 checksum How to use checksums |
3e143929ea748f245dd5ceec9314f42baa5f3c8c6e1440fe565afe26b1b307c0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logSecurity for caldip 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for caldip-0.2.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.