curatorkit 0.1.18

Data curation and synthetic data generation for LLM post-training.
Ingest from any source, generate with any LLM, verify every sample against its origin,
recover what fails, and export trainer-ready datasets with full provenance.

Documentation · Quickstart · Tutorials · Ecosystem · Contributing

Why CuratorKIT

Synthetic data is the fastest way to build a post-training dataset. It is also the fastest way to poison a model: generated samples hallucinate beyond their source material, drift off-distribution, duplicate each other, and quietly carry secrets, PII, and toxicity into training runs. Most teams find out after fine-tuning.

CuratorKIT treats dataset construction as a pipeline with quality gates, not a script with a prompt. Every generated answer is checked for grounding against the exact source chunk it came from. Rejected samples get diagnosed rather than discarded, and the fixable ones are repaired. Every run emits a provenance manifest, a dataset card, and checksums, so any sample can be audited back to its source.

Key features

• Grounded hallucination gate. Verifies each generated answer against the source passage it was generated from, not against the judge model's general knowledge.
• Reward and diversity gates. Multi-dimension LLM-judge scoring plus embedding-based filtering against low-quality and near-duplicate samples.
• Adaptive recovery. An inline diagnostic probe classifies each rejection into a failure-mode taxonomy and repairs the recoverable ones.
• Data hygiene. Secrets detection, PII pseudonymization (Presidio), and toxicity filtering as pipeline stages.
• Any source. JSONL, JSON, CSV, Parquet, HuggingFace datasets, and PDFs with layout-aware parsing. Multi-source runs with per-source field mapping.
• Eight generation tasks. QA, preference pairs, GRPO rollouts, multi-turn, Evol-Instruct, chain-of-thought, and adversarial variants, on any LiteLLM-compatible API or local Ollama/vLLM.
• Trainer-ready exports. Alpaca, ShareGPT, DPO, GRPO, and PPO formats with train/val/test splits, consumed directly by TRL and AlignTune.
• Provenance by default. Every run writes manifest.json, rejected.jsonl with structured reasons, dataset_card.md, and SHA-256 checksums.txt.

How it works

flowchart LR
    A["Ingest<br/>PDF · JSONL · CSV<br/>Parquet · HF Hub"] --> B["Clean + dedup<br/>exact · minhash · embedding"]
    B --> C["Hygiene<br/>secrets · PII · toxicity"]
    C --> D["Generate<br/>QA · DPO · GRPO · CoT"]
    D --> E{"Quality gates<br/>hallucination · reward · diversity"}
    E -->|pass| F["Export<br/>Alpaca · ShareGPT · DPO · GRPO · PPO"]
    E -->|reject| G["Adaptive recovery<br/>diagnose · repair · re-gate"]
    G -->|recovered| E
    G -->|unrecoverable| H["rejected.jsonl<br/>+ diagnostics"]
    F --> I["manifest · dataset card · checksums"]

Install

pip install "curatorkit[all]"        # connectors + generation + embedding + hygiene (recommended)
pip install curatorkit               # core only: cleaning and dedup

Requires Python 3.11+ on Linux, macOS, or Windows.

pip install "curatorkit[generation]"       # LLM generation only
pip install "curatorkit[generation-full]"  # generation + embedding + FAISS
pip install "curatorkit[connectors]"       # pyarrow + datasets (no LLM)
pip install "curatorkit[hygiene]"          # secrets / PII / toxicity gates
pip install "curatorkit[pdf]"              # layout-aware PDF parsing (MinerU)

# From source
pip install "curatorkit[all] @ git+https://github.com/Lexsi-Labs/CuratorKIT.git"

Quickstart

Clean and deduplicate an existing dataset (no LLM, no API key):

from curatorkit import Curator, CuratorConfig

result = Curator(CuratorConfig(
    dataset        = {"name": "tatsu-lab/alpaca", "max_samples": 2000},
    dedup          = "minhash",
    clean          = True,
    export_formats = ["alpaca", "sharegpt"],
    output_dir     = "output/clean",
)).run()

result.print_summary()

HF Hub sources need the connectors extra (included in all); local JSONL/CSV files run on the core install.

Generate gated synthetic QA data from a document:

export OPENAI_API_KEY=sk-...   # any LiteLLM backend works; local Ollama/vLLM too

result = Curator(CuratorConfig(
    dataset                 = "handbook.pdf",          # needs the [pdf] extra
    llm_model               = "openai/gpt-4o-mini",
    generation_task         = "qa",
    num_questions           = 3,
    hallucination_threshold = 0.7,                     # grounding gate
    reward_threshold        = 0.7,                     # LLM-judge gate
    export_formats          = ["alpaca", "sharegpt"],
    output_dir              = "output/qa",
)).run()

Or run a declarative YAML pipeline from the CLI:

curatorkit run examples/quickstart/pipeline.yaml --output-dir output/

Every run writes manifest.json, rejected.jsonl, dataset_card.md, and checksums.txt alongside the export files. The quickstart example runs end to end without an API key.

What it generates

Output files

Documentation

Full documentation lives at lexsi-labs.github.io/CuratorKIT.

Tutorials

Nine runnable notebooks cover the feature set end-to-end.

#	Notebook	Links
01	Generate an SFT dataset from a PDF
02	Generate DPO preference pairs
03	Generate GRPO rollouts
04	Ingest multiple sources
05	Clean and deduplicate a dataset
06	Adaptive recovery
07	Adversarial generation
08	Data hygiene pipeline
09	Filtered vs unfiltered fine-tuning

Full descriptions and prerequisites are in the tutorials index.

Contributing

The connector, generator, gate, and exporter layers are designed as plugin points. Read the contributing guide and the architecture reference, then open an issue or PR. Questions go to GitHub Discussions.

Support

• Documentation: lexsi-labs.github.io/CuratorKIT
• GitHub Issues: github.com/Lexsi-Labs/CuratorKIT/issues
• Discussions: github.com/Lexsi-Labs/CuratorKIT/discussions
• Email: pratinav.seth@lexsi.ai

Citation

If you use CuratorKIT in your research, please cite it (see CITATION.cff):

@software{curatorkit2026,
  author    = {Bhattacharjee, Soham and Sharma, Karun and Sankarapu, Vinay Kumar and Seth, Pratinav},
  title     = {CuratorKIT: Data Curation and Synthetic Data Generation for LLM Post-Training},
  year      = {2026},
  publisher = {Lexsi Labs},
  url       = {https://github.com/Lexsi-Labs/CuratorKIT}
}

@misc{bhattacharjee2026curatorkitdatacuration,
      title={CuratorKIT : Data Curation and Synthetic Data Generation for LLM Post-Training}, 
      author={Soham Bhattacharjee and Karun Sharma and Vinay Kumar Sankarapu and Pratinav Seth},
      year={2026},
      eprint={2606.21631},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.21631}, 
}

@misc{bhattacharjee2026provenancegroundedgatingadaptiverecovery,
      title={Provenance-Grounded Gating and Adaptive Recovery in Synthetic Post-Training Data Curation}, 
      author={Soham Bhattacharjee and Karun Sharma and Vinay Kumar Sankarapu and Pratinav Seth},
      year={2026},
      eprint={2606.11127},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.11127}, 
}

License

LSAL v1.1; see LICENSE. Free for research, education, and non-commercial use. Commercial use requires a separate license — contact support@lexsi.ai. The optional pdf extra installs MinerU, which is licensed AGPL-3.0. Install it only if that suits your use.

Contact

https://www.lexsi.ai

Paris 🇫🇷 · Mumbai 🇮🇳 · London 🇬🇧

Ecosystem

CuratorKIT is part of the Lexsi Labs open-source stack:

• AlignTune fine-tunes with the data you curate here. CuratorKIT's Alpaca, DPO, GRPO, and PPO exports are AlignTune's native input formats.
• TabTune is a unified library for tabular foundation models.
• DLBacktrace and xai_evals cover model interpretability and explanation evaluation.