deduplicate-lib 0.0.1

A Python package for deduplicating data.

deduplicate_lib

deduplication algorithms in python

---

Key Features

• Easy to use deduplication algorithms for any vector array
• Suite of tolerance tuning algorithms to help you find the right tolerance value for your system
• Suite of benchmarking tools to ensure rigor, accuracy, and speed (not yet implemented)
• Factory Plugin architecture, for easy extensibility and modification

---

Implemented Algorithms

• Distance Matrix (Simple, accurate, expensive): Computes the distance matrix for all vectors and determines duplicates by finding those that fall below a given distance
• Multi Hashing (Fast): Smears and rounds the vectors using a normal distribution and computes the hashes for each which are then used to determine duplicates by proportion of hash clashes.

Quick Start

install using pip

pip install deduplicate_lib

load your data into python

from deduplicate_lib.plugins.deduplication_algorithms.multi_hash import MultiHash

# define your paramerters in the MultiHash object
dda=MultiHash(
      tolerance=0.01,
      dataset_array: your_data_array,
      perturbations: int = 200, 
    )

print(dda.get_dataset_unique_structures())

A more detailed example can be seen in the examples directory