splurge-data-profiler 2025.2.0

A data profiling tool for delimited and database sources.

Splurge Data Profiler

A powerful data profiling tool for delimited and database sources that automatically infers data types and creates optimized data lakes (SQLite database).

Features

• DSV File Support: Profile CSV, TSV, and other delimiter-separated value files
• Automatic Type Inference: Intelligently detect data types using adaptive sampling
• Data Lake Creation: Generate SQLite databases with optimized schemas
• Inferred Tables: Create tables with both original and type-cast columns
• Flexible Configuration: JSON-based configuration for customization
• Command Line Interface: Easy-to-use CLI for batch processing
• Comprehensive Testing: Extensive test coverage ensuring reliability and robustness
• Production Ready: Enterprise-grade error handling and performance optimization

Installation

pip install splurge-data-profiler

Quick Start

1. Create a configuration file:

python -m splurge_data_profiler create-config examples/example_config.json

2. Profile your data:

python -m splurge_data_profiler profile examples/example_data.csv examples/example_config.json

CLI Usage

Profile Command

Profile a DSV file and create a data lake:

python -m splurge_data_profiler profile <dsv_file> <config_file> [options]

Options:

• --verbose: Enable verbose output

Create Config Command

Generate a sample configuration file:

python -m splurge_data_profiler create-config <output_file>

Configuration

The configuration file is a JSON file that specifies how to process your DSV file:

{
  "data_lake_path": "./data_lake",
  "dsv": {
    "delimiter": ",",
    "encoding": "utf-8"
  }
}

Documentation

For detailed documentation, examples, and API reference, see:

• Detailed Documentation
• Changelog

Requirements

• Python 3.10+
• SQLAlchemy >= 2.0.37

Quality Assurance

This project maintains high code quality through comprehensive testing:

• Unit Tests: Core component testing with 100% coverage of critical paths
• Integration Tests: End-to-end workflow validation
• Edge Case Tests: Error handling and boundary condition testing
• E2E Tests: Complete user scenario validation
• Performance Tests: Large dataset processing validation

Run tests with:

pytest

License

MIT License