andamento 0.3.3

Model-relative training data diagnostics for fine-tuned ML models

Andamento

Model-relative training data diagnostics for fine-tuned ML models

Training data effectiveness is relational, not intrinsic. The same dataset produces different diagnostics depending on which base model you measure against.

---

The Problem

You fine-tuned a model. Benchmarks improved overall. But specific capabilities degraded and you don't know why.

Count-based analysis says "add more data." Andamento surfaces which regions look wasted, harmful, or high-impact under a specific base model so you know where to inspect first.

Quickstart

pip install 'andamento[models]'

# Zero-config demo: downloads a small model (~1 GB), runs curvature on 400 examples
andamento quickstart

An interactive HTML report opens showing which task categories helped, which hurt, and why. No data, no models, no configuration needed.

# Then try on your own data:
andamento audit your_data.jsonl                    # Structural audit (no GPU)
andamento curvature your_data.jsonl \
    --base mistralai/Mistral-7B-v0.1 \
    --ft ./my-fine-tuned-model                     # Full curvature analysis

Recommended workflow: quickstart → validate → plan → audit → curvature

Choose Your Analysis Depth

Andamento has four named analysis paths. Pick the one that matches what you need:

Path 1: Structural Audit

What you need: CPU only. No models, no GPU. What you get: Coverage gaps, near-duplicate clusters, topic coherence, size imbalance, topology.

andamento validate your_data.jsonl        # Check format
andamento audit your_data.jsonl           # Structural findings + HTML report

When to use: First pass on any dataset. Answers "is this data structurally sound?" before spending GPU time.

Path 2: Curvature Diagnosis

What you need: GPU + base model + fine-tuned model (or LoRA adapter). What you get: Everything in Path 1, plus: per-cell curvature (helped/hurt/wasted), bootstrap CIs, repair queue ranked by impact, waste classification, content vs format decomposition.

andamento curvature your_data.jsonl --base org/model --ft org/fine-tuned

When to use: You want to know which regions of your training data are helping, which are hurting, and what to fix first.

Path 3: Curvature + Benchmark Coverage

What you need: GPU + models + a benchmark dataset (evaluation JSONL). What you get: Everything in Path 2, plus: benchmark alignment per cell, benchmark-dark detection (training regions with no external validation), delta agreement between curvature and benchmark performance.

andamento curvature your_data.jsonl --base org/model --ft org/fine-tuned \
    --benchmark eval_set.jsonl

When to use: You have benchmark results and want to cross-check whether curvature agrees with downstream performance.

Path 4: Remote Analysis

What you need: A GPU machine (cloud VM, Colab, lab server) + your laptop. What you get: Everything in Paths 2-3, but heavy computation runs on the GPU machine.

# On the GPU machine:
pip install 'andamento[server]' && andamento serve --port 8080

# From your laptop:
andamento curvature data.jsonl --base X --ft Y \
    --provider http --provider-url http://gpu:8080

When to use: Your data is small enough to send over the network, but your laptop has no GPU.

---

Why Not Just Use Cleanlab / FiftyOne / Manual Inspection?

Those tools measure data quality -- duplicates, label errors, outliers. Andamento measures data effectiveness -- whether a clean, well-labeled example actually produces a measurable loss change relative to a specific base model.

What It Measures

Two failure modes invisible to standard evaluation:

1. Over-tessellation -- cells where the curvature CI includes zero (no measurable training signal). Fine-tuning produced no measurable change, usually because the base model already handled this region.
2. Negative curvature -- cells where fine-tuning increased loss (CI entirely negative). The training data made the model worse here.

Plus: near-duplicate detection with per-cell adaptive thresholds (a separate diagnostic layer that catches corpus-internal redundancy curvature can't see), and a pre-flight calibration that adapts decision thresholds to your specific model and data.

⚠ The most dangerous failure mode: false all-clear

There's a regime where curvature will tell you everything is fine when it isn't. If your adapter is rank-constrained (LoRA, QLoRA) and the training data is semantically homogeneous (a cultural-voice corpus, a function-calling corpus, anything where every example serves the same functional role), the adapter can run out of capacity while every cell shows positive curvature. Curvature reports loss-per-region; it doesn't see gradient-subspace collisions inside the adapter. We hit this exactly once in the case studies (System B — every cell helped while the adapter regressed because its 64 rank dimensions were collapsed onto 17 distinct learning directions).

Two partial guardrails ship in the tool:

• global_uniformity_suspect flags reports where one curvature class dominates >80% of cells with abnormally low between-cell variation. Treat this flag as a stop sign, not a feature: it means you cannot trust an all-clear from curvature alone. Verify with --gradient.
• --gradient opts in to per-cell gradient direction analysis (~2× cost). Surfaces effective adapter rank, pairwise cosines, and collision pairs.

If global_uniformity_suspect fires on your run, do not take "every cell is positive" as confirmation. The only definitive answer is gradient-subspace inspection.

Key Observation: Effectiveness Depends on the Base Model

The same dataset produces different diagnostics depending on which base model you measure against:

	Raw base (Mistral-7B-v0.1)	Instruct base (Mistral-7B-Instruct-v0.2)
Helped	100%	71.9%
Wasted	0%	9.4%
Hurt	0%	18.8%

Against a raw pretrained model, every example produces positive curvature. Against the instruct model, 28.2% shows zero or negative curvature -- regions where the base model's existing distribution already covers these examples. (Note: this is a cross-model comparison. The fine-tune was trained from the raw base, not the instruct model.)

A dataset isn't effective or wasteful in isolation. It depends on what the base model already does.

How It Works

Andamento partitions your training data into cells, then measures how fine-tuning changed the model's loss in each region.

curvature = loss_base - loss_ft

• Positive = training helped (loss went down)
• Negative = training hurt (loss went up)
• CI includes zero = training was wasted (indistinguishable from noise)

Classification uses bootstrap 95% CIs per cell, not a global threshold. Each cell has its own effective decision boundary based on its sample size and loss variance. An optional pre-flight calibrates a fallback threshold from the base model's loss distribution when CIs can't be computed.

The repair queue ranks cells by |curvature| x example_count -- highest impact fixes first.

The Repair Loop

andamento curvature v1.jsonl --base X --ft Y --output v1/   # Diagnose
# Fix training data based on repair queue
andamento curvature v2.jsonl --base X --ft Y --output v2/   # Re-diagnose
andamento compare --before v1/report.json --after v2/report.json    # Track progress

What It Found (Airoboros Case Study)

We ran Andamento on Airoboros 3.2 (58K examples, Mistral 7B). Against an instruction-tuned base model, the diagnostic found:

• 6 categories where the fine-tune increased loss relative to the reference (creative writing + coding + chain-of-thought — the instruct model already handles these)
• 3 categories with no distinguishable loss shift (curvature CI includes zero)
• 92% of multiturn examples are near-duplicates (2,740 examples, 212 unique)
• 22 of 32 categories are novel_effective — genuine training value
• 2 incoherent topological components — category boundaries that don't map to real structural differences

Downstream intervention check: 6 pruning strategies × LoRA r=32 on raw Mistral-7B-v0.1, 1 epoch, lr 2×10⁻⁴, 10 tasks, 3 random seeds (42/123/456), bootstrap 95% CIs paired at the per-item level. This is not a claim that benchmarks are ground truth for curvature; it tests whether the diagnostic signals can guide useful interventions. Headline results vs. random pruning of the same size:

• Pruned + dedup → +10.41pp on GSM8K (CI [+9.00, +11.89]). Curvature flags loss-space waste; near-duplicate detection finds corpus-internal redundancy. Layering them is what produces this; neither alone does. Recovers about half of the −30.5pp regression that full fine-tuning costs.
• Topology-informed pruning (10% removed) → +1.95pp on TruthfulQA (CI [+1.18, +2.74]) and +1.92pp on Winogrande (CI [+0.45, +3.39]). H₀ persistence on the cell adjacency graph identifies which cell boundaries are structural; only structurally distinct negatives get cut.
• Category pruning alone: not significant on TruthfulQA. Quadrant-informed pruning: significantly hurts (random wins).
• The honest tradeoff: 5 of 6 pruning variants significantly hurt GSM8K. Pruning is targeted, not uniformly better.

See Methodology, Interpreting Results, and the reproducibility docs for the full context, caveats, and reproduction steps.

Evidence and Scope

Andamento is an exploratory diagnostic workbench, not a certified data-quality oracle. The directly measured quantity is the paired loss delta between two fixed models. The interpretation layer — cells, topology, repair queues, and pruning choices — is evidence-informed but not universal truth.

The strongest public anchor is the reproducible Airoboros/Mistral audit. The broader methodology is informed by additional case studies, including a public Pragma function-calling corpus and two proprietary systems that are documented as regime checks and failure cases. These case studies show where the diagnostic helped, where it diverged from benchmarks, and where curvature alone failed. They do not prove generalization across every model family, language, domain, or scale.

If Andamento gives a surprising result on your corpus, treat it as a prioritized inspection signal. Reports, failures, and replications are welcome; the goal of the open-source release is to grow shared evidence around when model-relative curvature is useful and when it is not.

Commands

Setup & validation:

Command	Path	Description
andamento validate	All	Check dataset format and schema before running anything
andamento plan	All	Dry-run preflight: shows what will happen, catches bad paths
andamento doctor	All	Check environment, GPU, packages, providers
andamento quickstart	--	Zero-config demo: auto-downloads model, runs curvature

Analysis:

Command	Path	Description
andamento audit	1	Structural audit: coverage, duplication, coherence (no GPU)
andamento curvature	2-4	Full curvature analysis: what's helping, hurting, and wasted
andamento analyze	2-4	Power-user entry point (all flags, checkpoint trajectories)
andamento saturate	2	Compare curvature across multiple base models

Post-analysis:

Command	Path	Description
andamento compare	2-4	Compare two reports to track repair progress
andamento dashboard	All	View a saved report interactively
andamento serve	4	Start GPU compute server for remote analysis

Key Flags

Flag	Description
--base / --ft	Base and fine-tuned model (HuggingFace ID or local path)
--prefilter	Pre-flight: base perplexity, calibrated threshold, sequence length stats
--gradient	Gradient analysis: effective adapter rank, pairwise cosines (~2x cost)
--tessellate-by	Metadata-based tessellation (e.g., domain,tier)
--stability-check	Test repair queue robustness under threshold perturbation
--format-type json|xml|yaml	Decompose curvature into format vs content tokens
--no-cache	Skip stage checkpoints, recompute everything
--provider http	Offload to remote andamento serve instance
--provider-url URL	URL for HTTP provider (persists in config.toml)
--provider-api-key KEY	API key for authenticated remote provider
--benchmark PATH	Benchmark dataset for alignment coverage
--loss-samples auto	Adaptive sqrt scaling (default), or integer for fixed
--load-in-4bit	4-bit quantization (default)
--no-open	Skip auto-opening the HTML report in your browser

See CLI Reference for all flags.

Installation

Pick the path that matches what you're trying to do:

"I just want a structural audit (no GPU)"

pip install andamento
andamento validate your_data.jsonl
andamento audit your_data.jsonl

Coverage, duplication, coherence, topology — all run on CPU. Downloads a ~90MB embedding model on first run (sentence-transformers). If your data has metadata fields (category, source, etc.), embedding is skipped entirely.

"I have HuggingFace models and a local GPU"

pip install 'andamento[models]'
andamento plan your_data.jsonl --base org/model --ft org/fine-tuned
andamento curvature your_data.jsonl --base org/model --ft org/fine-tuned

Loads models via Transformers. Quantizes to 4-bit by default (fits 7B models in ~6GB VRAM). 4-bit/8-bit quantization requires bitsandbytes (Linux only): pip install 'andamento[quantize]'. On macOS/Windows, use --no-quantize (needs more VRAM).

"I have a LoRA adapter (local directory)"

pip install 'andamento[models]'
andamento curvature data.jsonl --base org/base-model --ft ./my-lora-adapter/

Andamento auto-detects adapter_config.json. The --base flag must point at the full base model, not the adapter.

"I have no GPU"

pip install 'andamento[remote]'
# On a GPU machine (yours, a cloud VM, Colab):
pip install 'andamento[server]' && andamento serve --port 8080

# From your laptop:
andamento curvature data.jsonl --base X --ft Y \
    --provider http --provider-url http://gpu-machine:8080

All heavy computation runs remotely. Your laptop only needs CPU.

Cloud deployment

On the GPU machine:

pip install 'andamento[server]'
andamento serve --port 8080 --host 0.0.0.0 --api-key YOUR_SECRET

Security:

• Always use --api-key when binding to 0.0.0.0. Without it, anyone with network access can load models and run inference on your GPU.
• For internet-facing deployments, put the server behind a reverse proxy (nginx, Caddy) with TLS. andamento serve does not handle HTTPS itself.
• On private networks (VPC, tailnet, SSH tunnel), --api-key alone is sufficient.

HuggingFace auth for gated models:

# On the GPU machine, before starting the server:
huggingface-cli login
# or
export HF_TOKEN=hf_...

Persistent HF cache (Docker / ephemeral VMs):

# Mount the host cache so model downloads survive container restarts
docker run -v ~/.cache/huggingface:/root/.cache/huggingface ...

Preflight check (from your laptop):

andamento doctor --remote http://gpu-machine:8080 --api-key YOUR_SECRET

This checks GPU, CUDA, installed packages, HuggingFace auth, and cache directory on the remote server.

"I use GGUF / Ollama / llama.cpp"

Your dataset works fine — andamento audit and andamento validate need no model at all. For curvature analysis, point --base at the original HuggingFace model your GGUF was quantized from. You don't need to stop using Ollama for inference; Andamento just needs the HF weights for a one-time diagnostic pass.

How to find the HF source model:

1. Go to the GGUF repo on HuggingFace (e.g., TheBloke/Mistral-7B-v0.1-GGUF)
2. Look at the model card — it almost always links the source under "Original model" or "Base model"
3. That link is your --base (e.g., mistralai/Mistral-7B-v0.1)

Common examples:

You're running	Use as --base
mistral:7b in Ollama	mistralai/Mistral-7B-v0.1
llama3:8b in Ollama	meta-llama/Meta-Llama-3-8B
TheBloke/...-GGUF	Check "Original model" link on the model card
Your own GGUF quant	Use the HF ID you quantized from

# Structural audit — works immediately, no model needed:
andamento audit your_data.jsonl

# Curvature — point at the HF source, not the GGUF:
pip install 'andamento[models]'
andamento curvature your_data.jsonl \
    --base mistralai/Mistral-7B-v0.1 \
    --ft ./your-fine-tune

andamento plan will catch GGUF paths and Ollama names before you download anything, and suggest the HF equivalent.

Quick reference

pip install andamento                  # Structural audit only (CPU)
pip install 'andamento[models]'        # + Curvature analysis (GPU)
pip install 'andamento[remote]'        # + HTTP provider client
pip install 'andamento[server]'        # + GPU server (andamento serve)
pip install 'andamento[quantize]'      # + 4-bit/8-bit (bitsandbytes, Linux)
pip install 'andamento[all]'           # Everything

Input Formats

• JSONL (messages, text, or prompt/completion) -- auto-detected
• CSV/TSV -- --input-format csv
• HuggingFace datasets -- pass dataset ID, use --split for custom splits

Reproduce the Airoboros Audit

The Airoboros numbers above come from public data and public models. Step-by-step reproducibility scripts are included:

pip install 'andamento[models]'

# Full analysis with all diagnostics
andamento curvature jondurbin/airoboros-3.2 \
    --base mistralai/Mistral-7B-Instruct-v0.2 \
    --ft jondurbin/airoboros-m-7b-3.1.2 \
    --tessellate-by category \
    --prefilter --gradient --stability-check

# See reproducibility/README.md for step-by-step scripts

Python API

Every CLI command has a typed Python equivalent:

from andamento import run_audit, run_validate, run_compare

result = run_audit("your_data.jsonl")
print(result.verdict, result.n_cells)

v = run_validate("your_data.jsonl")
print(v.valid, v.n_examples)

cmp = run_compare("v1/report.json", "v2/report.json")
for d in cmp.summary_deltas:
    print(f"{d.label}: {d.delta:+.1f}")

All result types are frozen dataclasses. See CLI Reference — Python API for full documentation.

Documentation

Doc	Description
Methodology	Measurement model, assumptions, and limits
Interpreting Results	How to read the report and act on it
CLI Reference	Full flag documentation, examples, Python API

Privacy

By default, all analysis runs locally and your training data never leaves your machine. Network activity is limited to downloading public artifacts:

• Model weights from HuggingFace Hub (when using a HF model ID for --base or --ft)
• Embedding model (~90MB, sentence-transformers, first run only — skipped with --tessellate-by)
• HuggingFace datasets (when you pass a HF dataset ID instead of a local file)
• --provider http sends raw example texts and chat messages to your own remote andamento serve instance (your server, your network — nothing leaves it)

No telemetry. No phoning home. No third-party API calls. No cloud dependencies.

Attribution

This release is centered on the package, articles, and reproducibility docs. If Andamento helps your work, please link the repository and, when useful, the methodology or reproducibility docs:

• Repository: https://github.com/Autarkis/andamento
• Methodology: docs/methodology.md
• Reproducibility: reproducibility/README.md

License

Apache 2.0. See LICENSE.