decontaminate 0.3.0.post4

Fast contamination detection for ML training data - Python bindings for decon

decontaminate

Fast contamination detection for ML training data. Python bindings for decon.

Installation

pip install decontaminate

Usage

import decon

config = decon.Config(
    training_dir="/path/to/training/data",
    evals_dir="/path/to/eval/references",
    report_output_dir="/path/to/output",
)
report_dir = decon.detect(config)

API

The Python API is a thin PyO3 wrapper over the Rust implementation. See src/lib.rs for all Config parameters and available functions:

• detect(), review(), compare(), evals(), server()
• Tokenizer (encode/decode with cl100k, o200k, etc.)
• clean_text() (text normalization)

Documentation

Full documentation: https://github.com/vincentzed/decon