dtbag 0.1.0

Data Tool Bag (dtbag) - A Python library for text processing, data cleaning, and similarity-based clustering of textual data

dtbag is a comprehensive Python library designed for efficient text processing and data cleaning. It provides robust tools for identifying and unifying similar text entries, making it ideal for preprocessing textual data in data science and machine learning pipelines.

Core Features

1. Text Similarity & Clustering

• Advanced Levenshtein distance implementation for accurate string matching
• Automatic clustering of similar text entries based on configurable thresholds
• Group-based text unification for consistent data representation

2. Data Cleaning & Normalization

• Intelligent duplicate detection and removal
• Text unification based on frequency analysis
• Support for multilingual text processing

3. Production-Ready Tools

• Scikit-learn compatible API design
• Memory-efficient algorithms for large datasets
• Easy integration with existing data pipelines

#What's Inside:

*CatLists Identifies clusters of similar text entries and returns grouped lists with their most frequent representatives.

*CatUnifier
Transforms lists by replacing similar items with their most common representative, maintaining original list structure.

#Quick Start

from dtbag import CatUnifier

"Clean inconsistent data entries"
data = ["Yassine", "Parrise", "Yasin", "Pris", "PParis" "Yasine", "yasyne", "Paris"]
unifier = CatUnifier()
clean_data = unifier.fit_transform(data, threshold=0.7)
*Result: ["Yassine", "Paris"]



#Installation

```bash
pip install dtbag