pyspark-analyzer 5.0.2

A comprehensive PySpark DataFrame profiler for generating detailed statistics and data quality reports

🔍 PySpark DataFrame Profiler

A comprehensive PySpark DataFrame profiler for generating detailed statistics and data quality reports with intelligent sampling capabilities for large-scale datasets.

📚 Documentation | 🐛 Issues | 💡 Examples

✨ Features

• 🚀 Intelligent Sampling: Automatic sampling for datasets >10M rows with quality estimation
• 📊 Comprehensive Statistics: Null counts, distinct values, min/max, mean, std, median, quartiles
• 🎯 Type-Aware Analysis: Specialized statistics for numeric, string, and temporal columns
• ⚡ Performance Optimized: Single-pass aggregations, batch processing, approximate functions
• 🔍 Quality Monitoring: Statistical quality scores and confidence reporting
• 🎨 Flexible Output: Dictionary, JSON, and human-readable summary formats
• 📈 Large Dataset Support: Intelligent caching, partitioning, and sampling options
• 🎯 Adaptive Partitioning: Smart partition optimization considering AQE, data size, and cluster configuration
• 📊 Progress Tracking: Visual progress indicators for long-running operations

🚀 Quick Start

Prerequisites

• Python >=3.8
• PySpark >=3.0.0 (native median function available in 3.4.0+, fallback for older versions)
• Java 17+ (required for PySpark)
  • macOS: brew install openjdk@17
  • Ubuntu/Debian: sudo apt-get install openjdk-17-jdk
  • Windows: Download from Oracle or OpenJDK

📦 Installation

From PyPI (Recommended)

pip install pyspark-analyzer

From Source

git clone https://github.com/bjornvandijkman1993/pyspark-analyzer.git
cd pyspark-analyzer
pip install -e .

Using uv (for development)

git clone https://github.com/bjornvandijkman1993/pyspark-analyzer.git
cd pyspark-analyzer
uv sync

Basic Usage

from pyspark.sql import SparkSession
from pyspark_analyzer import analyze

# Create Spark session
spark = SparkSession.builder.appName("DataProfiling").getOrCreate()

# Load your DataFrame
df = spark.read.parquet("your_data.parquet")

# Profile the DataFrame (returns pandas DataFrame by default)
profile_df = analyze(df)

# View column statistics
print(profile_df)

# Get dictionary format for programmatic access
profile_dict = analyze(df, output_format="dict")
print(f"Total Rows: {profile_dict['overview']['total_rows']:,}")
print(f"Total Columns: {profile_dict['overview']['total_columns']}")

# Get human-readable summary
summary = analyze(df, output_format="summary")
print(summary)

Sampling Options

from pyspark_analyzer import analyze

# Option 1: Disable sampling completely
profile = analyze(df, sampling=False)

# Option 2: Sample to specific number of rows
profile = analyze(df, target_rows=100_000, seed=42)

# Option 3: Sample by fraction
profile = analyze(df, fraction=0.1, seed=42)  # 10% sample

# Option 4: Auto-sampling (default behavior)
# Automatically samples large datasets (>10M rows)
profile = analyze(df)

# Check sampling information (with dict output)
profile_dict = analyze(df, output_format="dict")
sampling_info = profile_dict['sampling']
if sampling_info['is_sampled']:
    print(f"Sample size: {sampling_info['sample_size']:,} rows")
    print(f"Speedup: {sampling_info['estimated_speedup']:.1f}x")

Advanced Features

from pyspark_analyzer import analyze

# Include advanced statistics (skewness, kurtosis, percentiles)
profile = analyze(df, include_advanced=True)

# Include data quality analysis
profile = analyze(df, include_quality=True)

# Profile specific columns only
profile = analyze(df, columns=["age", "salary", "department"])

# Different output formats
profile_df = analyze(df)                    # Default: pandas DataFrame
profile_dict = analyze(df, output_format="dict")    # Dictionary format
summary = analyze(df, output_format="summary")      # Human-readable summary

# Progress tracking for long-running operations
profile = analyze(df, show_progress=True)    # Show progress bar
profile = analyze(df, show_progress=False)   # Disable progress

# Control via environment variable
# export PYSPARK_ANALYZER_PROGRESS=always  # Always show progress
# export PYSPARK_ANALYZER_PROGRESS=never   # Never show progress
# export PYSPARK_ANALYZER_PROGRESS=auto    # Auto-detect (default)

# Save pandas output to various formats
profile_df = analyze(df)
profile_df.to_csv("profile.csv")
profile_df.to_parquet("profile.parquet")
profile_df.to_html("profile.html")

📊 Example Output

Default: Pandas DataFrame

profile_df = analyze(df)
print(profile_df)

# Output:
#                    column_name     data_type  null_count  null_percentage  distinct_count  ...
# 0                     user_id   IntegerType           0              0.0           50000  ...
# 1                         age   IntegerType         100              0.1             120  ...
# 2                       email    StringType           0              0.0           49950  ...

# Access metadata
print(profile_df.attrs['overview'])
# {'total_rows': 1000000, 'total_columns': 5, ...}

Dictionary Format

profile_dict = analyze(df, output_format="dict")

# Output:
{
    'overview': {
        'total_rows': 1000000,
        'total_columns': 5,
        'column_types': {...}
    },
    'sampling': {
        'is_sampled': True,
        'sample_size': 100000,
        'quality_score': 0.95,
        'estimated_speedup': 10.0
    },
    'columns': {
        'user_id': {
            'data_type': 'IntegerType()',
            'null_count': 0,
            'distinct_count': 50000,
            'min': 1,
            'max': 999999,
            'mean': 500000.0,
            ...
        }
    }
}

🏗️ Architecture

Core Components

• analyze(): Simple, unified API for all profiling operations
• StatisticsComputer: Handles individual column statistics computation
• SamplingConfig: Simple, clear configuration for sampling behavior
• BatchStatisticsComputer: Optimized batch processing for large datasets

Performance Optimizations

• Intelligent Sampling: Automatic sampling for datasets >10M rows
• Quality Estimation: Statistical quality scores for sampling accuracy
• Batch Aggregations: Minimize data scans with combined operations
• Approximate Functions: Fast distinct counts and percentile computations
• Smart Caching: Intelligent caching for multiple operations
• Adaptive Partitioning: Dynamic partition optimization based on:
  • Data size and row count estimation
  • Cluster configuration (parallelism, shuffle partitions)
  • Spark's Adaptive Query Execution (AQE) status
  • Avoids repartitioning overhead when not beneficial

📚 Examples

Check out the examples directory for comprehensive usage examples:

• installation_verification.py - Verify your installation
• simple_api_example.py - Quick start with the new analyze() API
• basic_usage.py - Traditional class-based usage demonstration
• sampling_example.py - All sampling configuration options
• pandas_output_example.py - Working with pandas DataFrame output
• advanced_statistics_demo.py - Advanced features and data quality analysis

🧪 Development

Setup with uv (Recommended)

# Install uv (ultra-fast Python package installer)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/bjornvandijkman1993/pyspark-analyzer.git
cd pyspark-analyzer

# Create virtual environment and install dependencies
uv sync --all-extras

# Verify installation
uv run python examples/installation_verification.py

Traditional Setup

# Clone the repository
git clone https://github.com/bjornvandijkman1993/pyspark-analyzer.git
cd pyspark-analyzer

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode
pip install -e ".[dev]"

# Verify installation
python examples/installation_verification.py

Testing

# Using Makefile (handles Java setup automatically)
make test                # Run all tests
make test-cov           # Run tests with coverage
make test-quick         # Run tests (stop on first failure)

# Using uv directly (requires .env file)
source .env && uv run pytest
source .env && uv run pytest --cov=pyspark_analyzer

# Traditional pytest
pytest
pytest --cov=pyspark_analyzer
pytest tests/test_sampling.py -v

Code Quality

# Format code
uv run black pyspark_analyzer/ tests/ examples/
# or: black pyspark_analyzer/ tests/ examples/

# Lint code (using ruff)
uv run ruff check pyspark_analyzer/
# or: ruff check pyspark_analyzer/

# Type checking
uv run mypy pyspark_analyzer/
# or: mypy pyspark_analyzer/

# Security scanning
uv run bandit -c .bandit -r pyspark_analyzer/
# or: bandit -c .bandit -r pyspark_analyzer/

# Run all pre-commit hooks
pre-commit run --all-files

📋 Requirements

• Python: 3.8+
• PySpark: 3.0.0+ (native median function available in 3.4.0+, fallback for older versions)
• Java: 17+ (required by PySpark)

📚 Documentation

Comprehensive documentation is available at pyspark-analyzer.readthedocs.io

• Installation Guide - Detailed setup instructions
• Quick Start - Get up and running quickly
• User Guide - Advanced usage and best practices
• API Reference - Complete API documentation
• Examples - Real-world usage examples

🔒 Security

This project takes security seriously. We employ multiple layers of security scanning:

• Dependency Scanning: Safety and pip-audit check for known vulnerabilities
• Static Analysis: Bandit scans for security issues in code
• Secret Detection: detect-secrets prevents secrets from being committed
• CodeQL: GitHub's semantic code analysis for security vulnerabilities
• Pre-commit Hooks: Security checks before code is committed
• Automated Updates: Dependabot keeps dependencies up-to-date

For more information, see our Security Policy.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

All pull requests are automatically scanned for security issues.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with PySpark for distributed data processing
• Inspired by pandas-profiling for comprehensive data analysis
• Uses statistical sampling techniques for performance optimization

---

⭐ Star this repo if you find it useful!