mod-trace

Inspect ML model artifacts without loading the framework.

mod-trace is a small Rust CLI for answering a practical question:

What is inside this model file?

It can inspect real artifacts such as CatBoost .cbm files, LightGBM .txt/.lgb text models, ONNX .onnx graphs, and PyTorch .pt/.pth checkpoints, then report structure, size, parameters, operator mix, rough inference cost, and changes between versions. All formats are read natively — no Python, framework, or runtime needed (CatBoost --deep is the one optional exception). The PyTorch reader is static: it sizes/names tensors and fingerprints weights without decoding exact shapes.

The most useful command is explain-diff, which says in plain English what changed between two model versions:

mod-trace explain-diff old_model.onnx new_model.onnx

(A secondary "tensor lab" for handcrafted JSON plans lives in docs/tensor-lab.md.)

Install

mod-trace is a single self-contained binary (written in Rust, published to PyPI as a wheel). The core commands need no Python and no ML framework at runtime.

pipx install mod-trace      # recommended: isolated, puts `mod-trace` on your PATH
# or
pip install mod-trace

mod-trace doctor            # shows which formats and helpers are available

• Prebuilt wheels for Apple Silicon macOS, Linux x86_64, and Windows x64 — pip just downloads the binary, no build step.
• Intel macOS has no prebuilt wheel yet, so pip install there builds from source (needs a Rust toolchain).
• Optional: CatBoost --deep (decoded tree/leaf diffs) shells out to Python catboost; point MODELLENS_PYTHON at a Python that has it if needed. Everything else — including all of LightGBM, ONNX, and PyTorch — needs nothing extra.

Core Commands

mod-trace doctor                                  # what formats/helpers are available
mod-trace inspect model.cbm                       # one model (also .lgb, .onnx, .pt)
mod-trace inspect --json model.onnx               # machine-readable
mod-trace inspect --deep model.cbm                # CatBoost: decoded splits (needs python catboost)
mod-trace diff old.cbm new.cbm                    # compare two versions
mod-trace explain-diff old.onnx new.onnx          # plain-English what changed
mod-trace check --max-size-growth 20% --fail-on-feature-change old.lgb new.lgb   # CI gate (exit != 0 = fail)
mod-trace convert model.pkl                       # un-pickle a model -> native (needs python)

Works on CatBoost .cbm, LightGBM .txt/.lgb, ONNX .onnx, and PyTorch .pt/.pth/.bin.

Why This Exists

Data and ML engineers often inherit model artifacts:

• fraud_model.cbm
• ranking_model.onnx
• model_v17.cbm
• candidate_model.onnx

Before running them, it is useful to know:

• what type of model it is
• how large it is
• how many trees, parameters, nodes, or operators it contains
• what the rough per-row or per-forward-pass cost looks like
• what changed between two versions

mod-trace is not a model runtime. It is an artifact inspector.

Doctor

mod-trace doctor lists which inspectors and optional helpers are available (each built-in format, the Python/catboost helper for --deep/convert, and which commands are usable). Add --json for CI/setup scripts to check availability programmatically.

JSON Output

Use JSON when mod-trace is part of CI, release checks, or model registry automation:

mod-trace inspect --json path/to/model.cbm
mod-trace inspect --json path/to/model.onnx
mod-trace diff --json path/to/old_model.cbm path/to/new_model.cbm
mod-trace diff --json path/to/old_model.onnx path/to/new_model.onnx

The JSON diff is designed for checks such as:

• fail if file size, parameter memory, or estimated ops grows too much
• fail if CatBoost feature names, training config, or learned-state fingerprint changes unexpectedly
• fail if ONNX operator counts or initializer tensors change

--deep CatBoost reports are text-only for now because they are diagnostic dumps from CatBoost's native Python parser.

CI Checks

Use check when a model artifact should fail promotion if it changes too much:

mod-trace check path/to/old_model.cbm path/to/new_model.cbm \
  --max-size-growth 20% \
  --fail-on-feature-change \
  --fail-on-training-config-change

mod-trace check path/to/old_model.onnx path/to/new_model.onnx \
  --max-size-growth 20% \
  --max-ops-growth 25% \
  --max-parameter-growth 30% \
  --fail-on-new-op

Check rules:

check prints a short PASS/FAIL report and exits nonzero when a rule fails. Any number of rules can be combined; any one failing fails the whole check.

Explain Diff

explain-diff is the plain-English version of diff — it reports what actually changed between two model versions, not just raw numbers:

mod-trace explain-diff old_model.onnx new_model.onnx

Model Change Explanation
------------------------
Type: ONNX
Old: old_model.onnx
New: new_model.onnx

Architecture:
  Attention layers:  12 -> 24
  Hidden size:       768 -> 1024
  Parameters:        110.0M -> 220.0M (+100%)
  Nodes:             420 -> 820 (+95%)

Estimated inference cost (static op proxy): +94%

New operators introduced:
  LayerNormalization

Summary:
  Grew from ~12 to ~24 attention layers; parameters +100%, estimated cost +94%.

Works for ONNX, CatBoost, and LightGBM (tree models report trees / leaves / learned-state instead of attention layers).

Supported formats

Per-format commands and full example output live next to the sample models:

• CatBoost → examples/catboost/README.md (incl. --deep)
• LightGBM → examples/lightgbm/README.md
• ONNX → examples/onnx/README.md
• PyTorch → examples/pytorch/README.md

Notes per format

• CatBoost --deep is optional and needs Python catboost; it decodes exact splits, leaf values, feature typing, and float borders. Plain inspect/diff need nothing.
• PyTorch is read statically from the torch.save zip (param counts, names, dtype, sampled weight fingerprint; ZIP64-safe). Shapes are not decoded. Legacy-pickle .pt/.bin give names-only; .safetensors is not supported (export to ONNX instead).
• PyTorch → ONNX: for graph-level detail or a .safetensors/legacy file, torch.onnx.export(model, dummy, "model.onnx", opset_version=18) then mod-trace inspect model.onnx. See examples/pytorch/README.md.

Convert (pickled / MLflow models)

mod-trace reads native model formats, but models are often stored as a pickle (model.pkl, e.g. logged by MLflow) — which is Python-only and can't be read without the framework. convert does the one-time translation:

mod-trace convert model.pkl                 # -> model.txt (LightGBM) or model.cbm (CatBoost)
mod-trace convert ./mlflow-model-dir --out /tmp/v3.txt
mod-trace inspect model.txt                 # now readable natively

This is the one command that needs Python (it loads the pickle with lightgbm/catboost, then re-saves natively). Point MODELLENS_PYTHON at an interpreter that has the model's library. Supported: LightGBM → .txt, CatBoost → .cbm. After converting, everything else is framework-free.

Tensor Lab

A secondary lab for explaining transformer internals on small handcrafted JSON plans (trace, compare, why, validate, quiz, demo). It is not the primary product surface. See docs/tensor-lab.md.

What It Does Not Do

mod-trace does not:

• run inference
• train models
• load PyTorch directly
• require CatBoost, PyTorch, or ONNX Runtime for inspection
• provide GPU kernels
• replace framework-native debugging tools

Privacy

Do not commit real model weights or private business artifacts. The repository ignores:

• models/
• examples/*.onnx
• examples/*.cbm

Use examples/make_sample_catboost.py when you need a shareable .cbm demo.

Architecture

mod-trace has small, native inspection paths per format:

• CatBoost .cbm metadata scanner
• LightGBM .txt/.lgb text parser
• ONNX protobuf graph scanner
• PyTorch .pt/.pth zip/pickle scanner
• JSON tensor-plan analyzer