Key Features

• Easy to use deduplication algorithms for any vector array
• Suite of tolerance tuning algorithms to help you find the right tolerance value for your system
• Suite of benchmarking tools to ensure rigor, accuracy, and speed (not yet implemented)
• Factory Plugin architecture, for easy extensibility and modification

Implemented Algorithms

• Distance Matrix (Simple, accurate, expensive): Computes the distance matrix for all vectors and determines duplicates by finding those that fall below a given distance
• Multi Hashing (Fast): Smears and rounds the vectors using a normal distribution and computes the hashes for each which are then used to determine duplicates by proportion of hash clashes.

Quick Start

install using pip

pip install deduplicate_lib

load your data into python

from deduplicate_lib.plugins.deduplication_algorithms.multi_hash import MultiHash

# define your paramerters in the MultiHash object
dda=MultiHash(dataset_array=your_data_numpy_array)

# return a list of all unique values
print(dda.deduplicate())

A more detailed example can be seen in the examples directory

Dependencies

• Python 3.9+
• numpy
• numba
• scipy

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

# Clone the repository
git clone https://github.com/julianholland/deduplicate.git
cd deduplicate

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check .
ruff format .

Running Tests

# Run all tests
pytest

# Run specific test categories
pytest tests/core/
pytest tests/plugins/
pytest tests/plugins/duplicate_detection_algorithms/distance_matrix

# Run with coverage
pytest --cov

📝 Citation

If you use deduplicate_lib in your research, please cite:

@software{deduplicate2026,
  title={deduplicate_lib: Auto Tolerance Finding Deduplication Algorithms in Python},
  author={Julian Holland},
  year={2026},
  url={https://github.com/julianholland/deduplicate},
  version={0.0.2dev}
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• The Fritz Haber Institute
• Juan Manuel Lombardi <3
• Maximillion Ach
• Chiara Panosetti

Project Links

• GitHub Repository

Project To-Do

• Add example.ipynb
• Create general Pre-allocation protocal
• Add benchmarks for time and robustness
• Add Locality-Sensitive Hashing as an option
• Speedup slow tasks with Numba
• Set up Read the Docs
• Create general deduplicate function
• Speed up NTPP