truva 0.1.0

Data curation engine for LLM fine-tuning

Truva

Truva curates your fine-tuning data so you train on signal, not noise.

A CLI-first data curation engine for ML engineers who fine-tune language models. Truva takes a messy dataset and produces a smaller, higher-quality "gold" dataset by removing redundancy, scoring information density, and detecting contradictions.

Goal: Reduce dataset size by 50–80% while maintaining or improving downstream model accuracy, cutting GPU training costs proportionally.

Quick Install

pip install truva

30-Second Example

# Deduplicate a dataset with default settings
truva dedupe ./data.jsonl --output ./deduped.jsonl

# Deduplicate with a custom threshold and generate a report
truva dedupe ./data.jsonl --threshold 0.9 --output ./deduped.jsonl --report ./report.json

# Generate embeddings for a dataset
truva embed ./data.jsonl --output ./embeddings.npy

What It Does

Before	After
50,000 rows	12,000 rows
Redundant examples	Unique, representative samples
Unknown quality	Scored and filtered
Hidden contradictions	Flagged for review

Features

Semantic Deduplication

Removes near-duplicate rows using embedding similarity and Union-Find clustering. Each cluster keeps the single most representative example (closest to centroid).

truva dedupe ./data.jsonl --threshold 0.95

• --threshold 0.95 (default): Aggressive but safe for most fine-tuning datasets
• --threshold 0.85: More aggressive, catches paraphrases
• --threshold 1.0: Only removes exact semantic matches

Embedding Generation

Compute vector embeddings for your dataset using local models or the OpenAI API.

# Local (free, no API key needed)
truva embed ./data.jsonl --provider local --model all-MiniLM-L6-v2

# OpenAI API
truva embed ./data.jsonl --provider api --model text-embedding-3-small

Supported Formats

• JSONL — One JSON object per line (.jsonl, .json)
• CSV — Auto-detects the text column or use --text-field
• Hugging Face Datasets — Pass a dataset identifier like username/dataset

Configuration

All options are available as CLI flags:

--threshold FLOAT       Cosine similarity threshold for dedup (0.0–1.0)
--provider [local|api]  Embedding provider
--model TEXT            Model name
--text-field TEXT       Column/field to use (auto-detected if not set)
--format TEXT           Input format: auto, jsonl, csv, hf
--output, -o TEXT       Output file path
--report TEXT           Path for JSON report

Requirements

• Python 3.10+
• Works on macOS (Apple Silicon), Linux

License

TBD