Skip to main content

✨ Welcome to VeriData ✨

An intuitive, high-level toolkit for cleaning and validating pandas DataFrames that accelerates data science and machine learning workflows.

PyPI version License Python versions Build Status

🤔 Why VeriData?

Data scientists spend approximately 80% of their time on data cleaning and preprocessing tasks. This is time that could be better spent on analysis, modeling, and generating insights. DataCleanX solves this problem by automating the most common data cleaning workflows with simple, intuitive functions.

Key Benefits:

  • ⏰ Save Time: Transform hours of repetitive cleaning code into a single function call
  • 🎯 Enforce Best Practices: Built-in validation and robust statistical methods
  • 📖 Improve Readability: Replace complex pandas chains with clear, expressive functions
  • 🔧 Stay Flexible: Use the all-in-one clean() function or individual modules as needed

🚀 Example in Action

See the power of DataCleanX in transforming messy, real-world data into analysis-ready datasets.

Before VeriData:

import pandas as pd
import numpy as np

# A sample of messy, real-world data
data = {
    'age': [25, 34, 999, np.nan, 28],
    'salary': [50000, 80000, 5000000, 75000, np.nan],
    'department': ['  Engineering!!  ', 'Sales', 'MARKETING', 'sales', 'Engineering'],
    'review': ['Excellent work!', 'N/A', 'Great performance.', '', 'Outstanding!!!']
}
df_messy = pd.DataFrame(data)
print(df_messy)
   age     salary      department           review
0 25.0    50000.0   Engineering!!    Excellent work!
1 34.0    80000.0           Sales                N/A
2 999.0  5000000.0       MARKETING  Great performance.
3  NaN    75000.0           sales                  
4 28.0        NaN    Engineering    Outstanding!!!

After VeriData:

import veridata as vd

# Clean the entire DataFrame with one powerful function
df_clean = vd.clean(df_messy)
print(df_clean)
   age   salary   department         review
0 25.0  50000.0  Engineering  excellent work
1 34.0  80000.0        Sales  great performance
3 29.0  75000.0        Sales               
4 28.0  68333.3  Engineering    outstanding

Notice how DataCleanX automatically handled the outliers, missing values, text inconsistencies, and rare categories - all in one line of code!

💡 The Philosophy: Engine vs. Dashboard

Think of pandas as a powerful car engine - it gives you incredible control and performance, but requires mechanical expertise to operate effectively. VeriData is like having a smart car dashboard - it uses that same powerful engine under the hood, but provides an intuitive, user-friendly interface that lets you focus on where you're going rather than how the engine works.

VeriData doesn't replace pandas; it makes pandas more accessible and efficient for everyday data cleaning tasks.

🎯 Key Features

  • 🧹 Comprehensive Cleaning: Powerful one-liner clean() function for complete data preprocessing workflows
  • 📊 Missing Value Imputation: Handle missing data with strategies including mean, median, mode, and advanced KNN imputation
  • 🎯 Robust Outlier Detection: Detect and remove outliers using statistical methods (IQR, MAD) and isolation techniques
  • 📝 Advanced Text Cleaning: Automatically lowercase, remove punctuation, digits, whitespace, and common stopwords
  • 🏷️ Smart Categorical Handling: Group rare categories and encode features for machine learning (one-hot, label encoding)
  • ✅ Schema Validation: Enforce data types and automatically detect schemas to prevent downstream errors
  • 📈 Data Quality Visualizations: Generate comprehensive reports with heatmaps, distribution plots, and correlation matrices
  • 🔧 Duplicate Detection: Intelligent identification and removal of duplicate records
  • 🎨 Flexible API: Use individual functions for granular control or the all-in-one solution for speed

🚀 Getting Started

Installation

Install VeriData directly from PyPI:

pip install veridata

Quick Usage

import veridata as vd
import pandas as pd

# Load your messy data
df = pd.read_csv('your_messy_data.csv')

# Option 1: Clean everything at once
df_clean = vd.clean(df)

# Option 2: Use individual functions for more control
df_filled = vd.handle_missing(df, strategy='median')
df_no_outliers = vd.remove_outliers(df_filled, method='iqr')
df_final = vd.clean_text(df_no_outliers, column='text_column')

# Generate a data quality report
report = vd.generate_report(df)
print(report)

📚 Documentation

For comprehensive guides, tutorials, and detailed API documentation, explore our documentation:

🤝 Contributing

We welcome contributions from the community! Whether you're fixing bugs, adding new features, improving documentation, or sharing feedback, your involvement helps make VeriData better for everyone.

Ready to contribute? Check out our Contributing Guide to get started. We've made the process as smooth as possible and provide guidance for contributors of all experience levels.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Metadata

Release files for VeriData 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for VeriData 0.1.0
File Size Uploaded Provenance Actions
veridata-0.1.0.tar.gz 12.4 kB Without provenance Details for veridata-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for VeriData 0.1.0
File Interpreter ABI Platform Actions
veridata-0.1.0-py3-none-any.whl Python 3 none any Details for veridata-0.1.0-py3-none-any.whl

Total release size: 24.2 kB

Release files / veridata-0.1.0.tar.gz

Download URL veridata-0.1.0.tar.gz
Size 12.4 kB
Tags Source
SHA-256 checksum
How to use checksums
671e603aa51333f798adf466d1cbbba425f789534ff35ddeb2570cf47af1aa78
BLAKE2b-256 checksum
How to use checksums
b7968fd8ff56260f220efcec78669dc50fa8a8b7cea3c60e515a93225f6262bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.7

Release files / veridata-0.1.0-py3-none-any.whl

Download URL veridata-0.1.0-py3-none-any.whl
Size 11.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9e172b4afc9889c29d35bf88519d4035c1c114fd9718a9e9f2252a89793b57c2
BLAKE2b-256 checksum
How to use checksums
f260cc9b0e6e9be1f49f188cb4eeea9916a0bd127eaf6d0da14259694df663fc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.7

Security for VeriData 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page