SciTeX IO (scitex-io)

Universal scientific data I/O with plugin registry

Full Documentation · uv pip install scitex-io[all]

Problem and Solution

Quick Start

import scitex_io as sio
import pandas as pd
import numpy as np

# Demo Data
df_orig = pd.DataFrame({"x": [1, 2, 3]})
arr_orig = np.array([1, 2, 3])
params_orig = {"lr": 1e-3, "epochs": 10}

# Unified Saving API
sio.save(df_orig, "data.csv")
sio.save(arr_orig, "data.npy")
sio.save(params_orig, "config.yaml")

# Unified Loading API
df_loaded = sio.load("data.csv")
arr_loaded = sio.load("data.npy")
params_loaded = sio.load("config.yaml")

# Round-trip check
assert df_loaded.equals(df_orig)
assert np.array_equal(arr_loaded, arr_orig)
assert params_loaded == params_orig

from scitex_io import register_saver, register_loader

@register_saver(".custom")
def save_custom(obj, path, **kw):
    open(path, "w").write(str(obj))

@register_loader(".custom")
def load_custom(path, **kw):
    return open(path).read()

sio.save("hello", "data.custom")
assert sio.load("data.custom") == "hello"

Installation

uv pip install "scitex-io[all]"

uv pip install "scitex-io[scientific]"   # HDF5 / zarr / parquet stack
uv pip install "scitex-io[mcp]"          # MCP server only
uv pip install -e ".[dev]"               # editable install for contributors

How it works

1. Format detection by extension

save() / load() pick the right reader/writer from the file extension via a plugin registry — just as OS does. Custom handlers can be available using register_saver / register_loader.

%%{init: {'flowchart': {'nodeSpacing': 20, 'rankSpacing': 40, 'curve': 'linear'}, 'themeVariables': {'fontSize': '12px'}}}%%
flowchart LR
    A["save(obj, 'x.ext')"] --> B{Registry}
    L["load('x.ext')"] --> B
    B -->|.csv .parquet .feather| C[pandas]
    B -->|.npy .npz| D[numpy]
    B -->|.h5 .zarr| E[h5py / zarr]
    B -->|.pkl .joblib| F[pickle / joblib]
    B -->|.pt .pth| G[torch]
    B -->|.png .jpg .svg| H[Pillow]
    B -->|.yaml .json| I[PyYAML / json]
    B -->|.bib .pdf .docx ...| J[30+ handlers]
    B -.->|register_*| K[Custom format]

2. save(obj, out.ext) in /path/to/script.py → /path/to/script_out/out.ext

Relative paths in save() resolve relative to the calling script / notebook, not the working directory. Scripts and outputs are tied as locations.

/path/to/project/
├── config/                              # see §3
│   └── ...
└── scripts/
    └── xxx/
        ├── filename.py                  # sio.save(df, "results.csv")
        └── filename_out/                # auto-created sibling of the script
            └── results.csv              # output lands here

Caller	sio.save(df, "sub/dir/results.csv") writes to
/path/to/analysis.py (script)	/path/to/analysis_out/sub/dir/results.csv
/path/to/exp.ipynb (notebook)	/path/to/exp_out/sub/dir/results.csv
python -i / IPython / REPL	~/.scitex/io/runtime/cache/sub/dir/results.csv

Bare filename or any relative path — "results.csv", "sub/dir/results.csv", and "./sub/dir/results.csv" all work; the whole path is appended under the caller's output anchor.

Intermediate directories created automatically — no os.makedirs() / Path.mkdir() calls needed on the caller side.

sio.save(df, "/data/x.csv")                            # absolute → used as-is

sio.save(df,  "results.csv", symlink_from_cwd=True)
sio.save(fig, "fig1.png",    symlink_to="/data/latest/fig1.png")
sio.save(df,  "results.csv", use_caller_path=True)     # resolve from caller script
sio.save(df,  "results.csv", dry_run=True)             # print path, don't write

3. Centralized project configuration

Scientific projects benefit from keeping parameters — hyperparameters, paths, thresholds — out of the scripts that consume them, as a single source of truth.

CONFIG = load_configs() collects every YAML under <project-root>/config/ into one nested DotDict. Parameters are then accessible as CONFIG.YAML_FILE_NAME.FIELD_NAME.

UPPER_CASE normalisation. YAML filenames and field names are recognised in UPPER_CASE, following Python's convention for user-defined parameters. model.yaml with hidden_dim: 256 lands at CONFIG.MODEL.HIDDEN_DIM regardless of source casing.

Conflict handling. When an UPPER/lower pair collide (e.g. MODEL.yaml next to model.yaml, or HIDDEN_DIM next to hidden_dim), the UPPER variant is prioritised and a UserWarning is emitted pointing at the conflict.

/path/to/project/
├── config/
│   ├── PATHS.yaml                       # DATA_DIR: /data/experiment_01
│   ├── PREPROCESS.yaml                  # SAMPLE_RATE: 1000, BANDPASS: [0.5, 40]
│   ├── MODEL.yaml                       # HIDDEN_DIM: 256, DROPOUT: 0.3
│   └── IS_DEBUG.yaml                    # IS_DEBUG: true
└── scripts/
    └── xxx/
        └── filename.py                  # CONFIG = sio.load_configs()
                                         #   CONFIG.MODEL.DROPOUT       → 0.3
                                         #   CONFIG.PREPROCESS.SAMPLE_RATE → 1000

CONFIG = sio.load_configs()           # loads ./config/*.yaml
CONFIG.PREPROCESS.SAMPLE_RATE            # 1000

# Debug mode: DEBUG_ prefixed keys override their counterparts
# In MODEL.yaml: { HIDDEN_DIM: 256, DEBUG_HIDDEN_DIM: 32 }
CONFIG = sio.load_configs(IS_DEBUG=True)
CONFIG.MODEL.HIDDEN_DIM                  # 32 (debug value promoted)

4. Linter for Migration and Hooks

scitex-io ships 14 IO-specific (STX-IO001..014) and 5 path-handling (STX-PA001..005) lint rules. They are detected automatically by scitex-dev's linter, which is already a hard dependency of scitex-io — no extra install needed.

scitex-dev linter check-files src/           # lint a tree
scitex-dev linter list-rules --category io   # show live rule definitions

cp examples/scitex_io_lint.sh ~/.claude/hooks/post-tool-use/
chmod +x ~/.claude/hooks/post-tool-use/scitex_io_lint.sh

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write|MultiEdit",
        "hooks": [
          { "type": "command",
            "command": "~/.claude/hooks/post-tool-use/scitex_io_lint.sh" }
        ]
      }
    ]
  }
}

5. Etc.

paths = sio.glob("data/**/*.csv")                        # natural sort: 1, 2, 10
paths = sio.glob("results/{exp1,exp2}/*.npy")            # brace expansion
paths, parsed = sio.parse_glob("sub_{id}/ses_{session}/*.vhdr")
# parsed = [{'id': '001', 'session': 'pre'}, ...]

dfs = sio.load("results/*.csv")                          # list of DataFrames
data = sio.load("large.hdf5"); data = sio.load("large.hdf5")  # 2nd call: cache hit

sio.embed_metadata("figure.png", {
    "experiment": "exp_042", "model": "resnet50",
    "accuracy": 0.94, "timestamp": "2026-03-11",
})
meta = sio.read_metadata("figure.png")
meta["experiment"]              # "exp_042"

Four Interfaces

from scitex_io import save, load, list_formats, register_saver, register_loader
from scitex_io import load_configs, DotDict
from scitex_io import embed_metadata, read_metadata, has_metadata

save(obj, "path.ext")        # Save any object
data = load("path.ext")      # Load any file
fmts = list_formats()        # Show all registered formats
cfg  = load_configs()        # Load ./config/*.yaml as DotDict
embed_metadata("fig.png", d) # Embed provenance into figure

scitex-io --help-recursive          # Show all commands
scitex-io info                      # Show registered formats
scitex-io configs                   # Load and display project configs
scitex-io configs -d ./my_configs   # Custom config directory
scitex-io configs --json            # Output as JSON
scitex-io list-python-apis -vv      # List Python APIs with signatures
scitex-io --version                 # Show version
scitex-io mcp start                 # Start MCP server
scitex-io mcp doctor                # Check MCP health
scitex-io mcp list-tools -vv        # List MCP tools with parameters

scitex-io mcp start

scitex-io skills list              # List available skill pages
scitex-io skills get save-and-load # Get detailed save/load documentation
scitex-io skills get glob          # Get glob/parse_glob patterns
scitex-io skills get supported-formats  # Get all format tables

Part of SciTeX

scitex-io is part of SciTeX. Install via the umbrella with pip install scitex[io] to use as scitex.io (Python) or scitex io ... (CLI).

import scitex

@scitex.session
def main(CONFIG=scitex.INJECTED):
    data = scitex.io.load("input.csv")     # auto-tracked by clew
    result = process(data)
    scitex.io.save(result, "output.csv")   # auto-tracked by clew
    return 0

scitex.io delegates to scitex_io — they share the same API and registry.

The SciTeX system follows the Four Freedoms for Research below, inspired by the Free Software Definition:

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.