data morph
Open Source File Data Migration with Fine-tuned Small Language Model
Knowledge distillation from a large-model agent (Claude Opus + Agent Skill) into a fine-tuned Gemma 2B, so developers can convert between file formats locally for free instead of paying for frontier-LLM API calls.
AI Builders 2026 · Track: Agentic AI + NLP
Problem
Rule-based parsers can't handle messy, context-dependent file conversions. Frontier LLMs can, but they're expensive at scale. This project distills that capability into a 2B-parameter model that runs locally.
Approach
- Teacher: Claude Opus + Claude Code + Agent Skill generates 500–1000 verified training pairs.
- Student: Gemma 2B, fine-tuned with LoRA / QLoRA.
- Target: ≥80% of teacher accuracy across 4 metrics — Format Validity, Schema Compliance, Loadability, Content Accuracy.
Pipeline architecture
Conversion is a five-stage pipeline, not a single end-to-end model call. The model only ever sees a small structured metadata envelope, never the full source file:
[source file]
│
├─→ [1. Metadata extractor] deterministic — schema + samples + warnings
├─→ [2. Context summarizer] Gemma 2B base — short NL summary
↓
[3. Script generator] Claude Opus (training) → Gemma 2B fine-tuned (inference)
↓ outputs an executable Python script
[4. Sandbox executor] deterministic — runs the script
↓ converted output file
[5. Validator] the 4 W2 metrics — format, schema, load, content
↓
[output file]
Why this shape: distillation target narrows from "transform a whole file" (impractical for a 2 B model) to "read metadata, write a script" (realistic). The model never sees full file content, so the pipeline scales to arbitrary file sizes. Failures are debuggable — the script is a readable intermediate artefact.
Status
W1–W6 complete; W7 model surgery done — a 2.0 GB single-file student is production-validated.
-
Data (W3): 800 verified teacher pairs (100% accept), split into
data/processed/{train,val,test}.jsonl(650 / 80 / 70, content-disjoint). -
EDA (W4):
notebook/w4_eda.ipynb— training-readiness audit (balance, leakage, sequence-length budget). -
Fine-tune (W5): Gemma-4 E2B distilled via LoRA (
mlx_vlm.lora, SFT) on the envelope→script task. Best checkpoint (iter-400) selected by held-out eval. -
Eval (W6): on the held-out 70-case test set, through the full pipeline (envelope → script → sandbox → 4 metrics), the fine-tuned student reaches 65/70 one-shot and 68/70 (0.971) at production retry≤3 — already ≥80%-of-teacher.
-
Shrink (W7): the multimodal base is mostly dead weight for this task. A three-step surgery (
scripts/build_textonly_student.py+prune_vocab.py) fuses the adapter, strips the unused vision + audio towers, prunes the 262 k vocab → 16 k (the corpus uses ~4.5 k tokens; the vocab indexes the two biggest tensors), then re-quantizes — all on a puregemma4_textmodel loaded viamlx_lm:Artifact params size retry≤3 % teacher fine-tuned bf16 (runtime adapter) 5.12 B 9.6 GB — — prior 8-bit (full model) 5.1 B 5.5 GB 68/70 ~97% fused + text-only + vocab-16k, bf16 2.05 B 3.8 GB 69/70 (0.986) ~99% + 8-bit (final ship artifact) 2.05 B 2.0 GB 67/70 (0.957) ~96% 9.6 GB → 2.0 GB (−79%) with accuracy still well above the ≥80%-of-teacher target on every metric. Each cut is lossless-by-construction (strip/prune, guarded by a tokenizer round-trip verification gate) or a small retry-recoverable numerical cost.
Next (W7 deployment): push the 2.0 GB model to Hugging Face Hub with a model card,
ship the pip-installable pipeline wrapper. See docs/progression.md for the live tracker.
Supported formats
CSV, JSON, TXT — in 5 use cases (CSV→JSON nested, JSON→CSV flattening, TXT log→CSV, CSV→TXT report, schema migration).
Setup
Requires Python 3.12 (chosen for stronger MLX support). Project is
managed by uv.
uv sync # creates .venv from pyproject.toml + uv.lock
source .venv/bin/activate # macOS / Linux
# .venv\Scripts\activate # Windows
Add a new dependency: uv add <pkg> (or uv add --dev <pkg> for dev-only).
Hardware / framework
- Primary target: MacBook Pro M5 Max (40 GPU cores, 120 GB unified memory) with MLX.
- Fallback: Google Colab + PyTorch + Unsloth (used when MLX is unavailable, e.g. on Windows).
Repo structure
data/
raw/ # synthetic corpus from seeded generators (regenerable, gitignored)
interim/ # verified teacher pairs (envelope + analysis + script + scores)
processed/ # train/val/test chat JSONL for fine-tuning
test_set/ # 15 hand-crafted W2 baseline cases
notebook/ # EDA (w4_eda), fine-tune scaffold (w5_finetune), experiments
src/
extractor/ # Stage 1: deterministic metadata extractor — CSV, JSON, TXT (done)
evaluation/ # Stage 5: the 4 W2 metrics + Opus-baseline runner (DO NOT EDIT)
data/ # generators (oracle), sandbox (Stage 4), teacher_script + collect (Stage 3)
features/ # format_pairs: verified pairs → chat JSONL + disjoint split
models/ # LoRA/QLoRA fine-tune + inference (W5)
scripts/ # generate_corpus, collect_pairs, collect_all_parallel, build_dataset, baseline, plotting
skills/ # Agent-Skill prompts read by `claude -p` (file conversion + script generation)
tests/ # unit tests (metrics, extractor, data, features) + fixtures
models/ # Gemma-4 E2B (local, gitignored) + fine-tuned checkpoints
results/ # baseline run artefacts (per-run summary.json + plots)
docs/ # specs, plans, weekly reports (gitignored)
Timeline (8 weeks)
| Week | Focus | Points |
|---|---|---|
| 1 | Problem statement + use cases | 15 |
| 2 | Metrics + Claude Opus baseline | 15 |
| 3 | Teacher-generated training pairs | 15 |
| 4 | EDA + data cleaning | 20 |
| 5 | Fine-tune Gemma 2B (LoRA) | — |
| 6 | Evaluation + error analysis | 20 |
| 7 | Deployment (pip + HF Hub) | 15 |
| 8 | Blog, slides, poster | — |
| Total | 100 (≥70 to pass) |
Deliverables
- GitHub repo (this one)
- Hugging Face Hub model + model card
pip install-able Python package- Medium blog post
- Presentation slides + A1 poster
- Facebook post (100–200 words)
Ethics
- Converted files may contain personal data → no uploads of user input.
- Teacher bias propagates to student — documented in model card.
- Hallucination risk mitigated by automated format/schema validation at inference time.
Metadata
Release files for data-morph-gemma 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| data_morph_gemma-0.1.0.tar.gz | 509.0 kB | Without provenance | Details for data_morph_gemma-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| data_morph_gemma-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for data_morph_gemma-0.1.0-py3-none-any.whl |
Total release size: 568.7 kB
Release files / data_morph_gemma-0.1.0.tar.gz
| Download URL | data_morph_gemma-0.1.0.tar.gz |
|---|---|
| Size | 509.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
321400be6c76182115afda3eccd968f00dfebd23502209087fd619738eb45380
|
|
BLAKE2b-256 checksum How to use checksums |
a7ad2f65fa1f8906cf1d4d6124017b746fe2358c9e7113d597613800722a637e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.21 {"installer":{"name":"uv","version":"0.11.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / data_morph_gemma-0.1.0-py3-none-any.whl
| Download URL | data_morph_gemma-0.1.0-py3-none-any.whl |
|---|---|
| Size | 59.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
18816a2ac9c1d06e2a92b6bed28763462adcc56082f222592cd6fcb353ef497d
|
|
BLAKE2b-256 checksum How to use checksums |
3a314a2ec19b851a037407cac6018fb54fa6d32e11333c09025cb7c971a363b3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.21 {"installer":{"name":"uv","version":"0.11.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for data-morph-gemma 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.