Key Features

• Easy to use deduplication algorithms for any vector array
• Suite of tolerance tuning algorithms to help you find the right tolerance value for your system
• Suite of benchmarking tools to ensure rigor, accuracy, and speed (not yet implemented)
• Factory Plugin architecture, for easy extensibility and modification

Implemented Algorithms

• Distance Matrix (Simple, accurate, expensive): Computes the distance matrix for all vectors and determines duplicates by finding those that fall below a given distance
• Multi Hashing (Fast): Smears and rounds the vectors using a normal distribution and computes the hashes for each which are then used to determine duplicates by proportion of hash clashes.

Quick Start

pip install deduplicate_lib

from deduplicate_lib.plugins.duplicate_detection_algorithms.multi_hashing import MultiHashing
import numpy as np

dataset = np.array([[1.0, 2.0], [1.01, 2.01], [5.0, 6.0]])
dda = MultiHashing(tolerance=0.1, dataset_array=dataset)

# return unique vectors
print(dda.deduplicate())

# check a single vector against the dataset
dda.input_vector = np.array([1.0, 2.0])
print(dda.duplicate_check())  # True

See the full documentation for API reference, tolerance tuning, and more examples.

Dependencies

• Python 3.10+
• numpy
• numba
• scipy

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

# Clone the repository
git clone https://github.com/julianholland/deduplicate.git
cd deduplicate

# Create a virtual environment with the package and dev dependencies (needs uv)
uv sync

# Run tests
uv run pytest

# Run linting
uv run ruff check .
uv run ruff format .

Running Tests

# Run all tests
pytest

# Run specific test categories
pytest tests/core/
pytest tests/plugins/
pytest tests/plugins/duplicate_detection_algorithms/distance_matrix

# Run with coverage
pytest --cov

📝 Citation

If you use deduplicate_lib in your research, please cite:

@software{deduplicate2026,
  title={deduplicate_lib: Auto Tolerance Finding Deduplication Algorithms in Python},
  author={Julian Holland},
  year={2026},
  url={https://github.com/julianholland/deduplicate},
  version={0.0.5}
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• The Fritz Haber Institute
• Juan Manuel Lombardi <3
• Maximillion Ach
• Chiara Panosetti

Project Links

• GitHub Repository
• Documentation

Project To-Do

• Add example.ipynb
• Create general Pre-allocation protocal
• Add benchmarks for time and robustness
• Add Locality-Sensitive Hashing as an option
• Speedup slow tasks with Numba
• Set up Read the Docs
• Create general deduplicate function
• Speed up NTPP