RupuData

Follow the path of your data.

RupuData answers an uncomfortable question: does my training data overlap with data I use to evaluate my model?

Local CLI for duplicates, train/eval overlap, and benchmark overlap signals, with deterministic JSON reports. Technical signals, not legal certification.

Install

Requires Python 3.9+.

pip install rupudata
rupudata --help

Quick start

rupudata scan dataset.jsonl
rupudata compare train.jsonl eval.jsonl
rupudata compare train.jsonl eval.jsonl --text-field question
rupudata compare train.jsonl eval.jsonl --fail-on-overlap
rupudata benchmark-check train.jsonl --benchmark gsm8k
rupudata benchmark-check train.jsonl --benchmark gsm8k --fail-on-overlap

compare defaults to full records. Use --text-field when only one column should participate in matching.

--fail-on-overlap exits 2 when exact or normalized overlap is found (CI gate). Exit 1 is reserved for errors. The JSON report is written either way.

Real-world example (GSM8K)

Export train/test from Hugging Face (openai/gsm8k) to JSONL. Each line looks like:

{"question": "Natalia sold clips to 48 of her friends in April…", "answer": "72"}

pip install datasets
python - <<'PY'
from datasets import load_dataset
import json

def write_jsonl(path, split, key="question"):
    rows = load_dataset("openai/gsm8k", "main", split=split)
    with open(path, "w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps({key: row[key]}, ensure_ascii=False) + "\n")
    print(split, len(rows), "->", path)

write_jsonl("gsm8k_train.jsonl", "train")
write_jsonl("gsm8k_test.jsonl", "test")
PY

Path A — inject three test questions (overlap)

head -n 3 gsm8k_test.jsonl > leak.jsonl
cat gsm8k_train.jsonl >> leak.jsonl

rupudata benchmark-check leak.jsonl \
  --benchmark gsm8k \
  --reference gsm8k_test.jsonl

╭───────────────────────────────╮
│ RupuData v0.7.0               │
│ Follow the path of your data. │
╰───────────────────────────────╯

Benchmark check: leak.jsonl vs GSM8K

Benchmark
──────────────────────────────
  Benchmark                GSM8K
  Reference                user_reference
  Benchmark records        1,319
  Dataset rows             7,476
  Benchmark fingerprint    rupu:a75016197e210681

Overlap
──────────────────────────────
  Exact matches                  3
  Normalized matches             3
  Status                         OVERLAP_DETECTED
  Evidence pairs (exact)         3
  Evidence pairs (normalized)    3

Evidence (sample)
──────────────────────────────
  dataset_row    reference_row    field
  0              0                question
  1              1                question
  2              2                question

3 exact GSM8K test-set overlaps detected under the configured matching methodology.

Path B — clean train (no leak)

rupudata compare gsm8k_train.jsonl gsm8k_test.jsonl --text-field question
rupudata benchmark-check gsm8k_train.jsonl --benchmark gsm8k --reference gsm8k_test.jsonl
rupudata scan gsm8k_train.jsonl

→ compare / benchmark-check: 0 overlaps, NO_OVERLAP_DETECTED.

→ scan (7,473 rows): 0 exact duplicates; 2 lexical near-dupe pairs. JSON evidence:

"near_duplicates": {
  "pairs": 2,
  "records_flagged": 4,
  "evidence": [
    { "left": 1174, "right": 7233, "jaccard": 0.8688, "field": "question" },
    { "left": 2483, "right": 6691, "jaccard": 0.9262, "field": "question" }
  ],
  "evidence_truncated": false
}

Record 2483
  Josie received $50 as a gift. She plans to buy two cassette tapes…

Record 6691
  Amanda received $50 as a gift. She plans to buy two cassette tapes…

Similarity (Jaccard on character shingles): 0.93

That is lexical near-duplication. RupuData does not claim the questions are the same math problem for a student, nor plagiarism — only that the strings share enough character shingles under the configured threshold.

Benchmark reference: demo sample vs real audit

Warning: Without --reference, default gsm8k uses a tiny packaged sample. A NO_OVERLAP_DETECTED against that sample does not mean your dataset is free of GSM8K. For an actual audit, pass the real export with --reference.

rupudata benchmark-check train.jsonl --benchmark gsm8k --reference /path/to/gsm8k.jsonl

Try the packaged examples

git clone https://github.com/EmanuelCorreaAR/rupudata.git
cd rupudata
pip install -e .

rupudata compare examples/train.jsonl examples/eval.jsonl
rupudata benchmark-check examples/train_with_gsm8k_overlap.jsonl --benchmark gsm8k
rupudata scan examples/near_dupes.jsonl --near-duplicate-threshold 0.85

Status

0.8.1 — clearer CLI help; --fail-on-overlap on compare / benchmark-check.

Not in this release (on purpose): semantic / paraphrase matching, streaming multi-GB scans, provenance/license detectors.

Next: driven by real usage.

Development

git clone https://github.com/EmanuelCorreaAR/rupudata.git
cd rupudata
pip install -e ".[dev]"
pytest
python -m build

Technical audit contract

Methodology, matching units, and evidence shapes: docs/AUDIT.md.

License

Apache License 2.0