text-dedup 0.2.1

All-in-one text deduplication

text-dedup

Features

• Hash-based methods such as SimHash, MinHash + LSH for near deduplication.
• SuffixArray-based method from Deduplicating Training Data Makes Language Models Better for substring exact deduplication.
• In-memory or Redis/KeyDB-cached index to handle larger than memory datasets.

Documentation

Github Pages

Todos

• Memory benchmark for streaming processing
• Speed benchmark for in-memory processing
• Inter-dataset deduplication
• Rewrite suffix array in Python

Thanks

• seomoz/simhash-cpp
• datasketch
• google-research/deduplicate-text-datasets
• Developed with OSS license from JetBrains
• This project is heavily influenced by the deduplication work at BigScience workshop. The original code can be found at bigscience-workshop/data-preparation.