edasuite 0.0.4

A Python library for exploratory data analysis with advanced statistical features

EDASuite

A comprehensive Python library for exploratory data analysis with advanced features for data profiling, quality assessment, and stability monitoring.

Features

Core Analysis

• Automated Feature Analysis
  • Continuous features: mean, median, std, quartiles, skewness, kurtosis, outliers
  • Categorical features: mode, value counts, cardinality, entropy
  • Automatic type inference with schema override support
  • Missing value detection and sentinel value replacement

Advanced Statistics

• Target Relationship Analysis

  • Information Value (IV) and Weight of Evidence (WoE)
  • Optimal binning for continuous features
  • Predictive power classification
  • Statistical significance testing

• Correlation Analysis

  • Pearson and Spearman correlations with p-values
  • Configurable correlation thresholds
  • Top-N correlation tracking per feature
  • Smart feature selection for large datasets

Data Quality

• Quality Assessment System

  • Automated quality scoring (0-10 scale)
  • Per-feature quality flags (high_missing, low_variance, constant, outliers)
  • Overall dataset quality metrics
  • Actionable recommendations

• Sentinel Value Handling

  • Automatic detection and replacement of no-hit values
  • Provider-specific default value handling
  • Configurable via DatasetSchema

Stability Monitoring

• Cohort-Based Stability

  • PSI (Population Stability Index) for categorical features
  • KS (Kolmogorov-Smirnov) test for continuous features
  • Train/test drift detection
  • Feature-level stability metrics

• Time-Based Stability

  • Multiple time window strategies (monthly, weekly, quartile, custom)
  • Temporal trend analysis (increasing, decreasing, volatile)
  • Auto-detection of optimal time periods
  • Minimum sample size enforcement

Provider Analytics

• Provider Match Rates
  • Automatic detection via <provider>_record_not_found columns
  • Data coverage statistics by provider (% of records with data)
  • Feature-level availability tracking
  • Not-found record counts per provider
  • Supports both column-based and schema-based detection

Performance

• Large Dataset Support
  • Multiple file format support (CSV, Parquet)
  • Chunked CSV reading for files >100MB
  • Configurable sampling for faster analysis
  • Memory-efficient correlation computation
  • Tested with 100K+ rows, 400+ features

Installation

pip install edasuite

Quick Start

Basic Usage

from edasuite import EDARunner, DataLoader
import pandas as pd

# Option 1: Load from file using DataLoader
df = DataLoader.load_csv("data.csv")

# Option 2: Use existing DataFrame
df = pd.read_csv("data.csv")  # or from database, etc.

# Initialize runner
runner = EDARunner(
    max_categories=50,
    top_correlations=10
)

# Run analysis
results = runner.run(
    data=df,
    output_path="eda_results.json"
)

Loading Data

EDASuite provides DataLoader utilities for loading data:

from edasuite import DataLoader

# Load CSV
df = DataLoader.load_csv("data.csv")

# Load Parquet (faster for large files)
df = DataLoader.load_parquet("data.parquet")

# Load with sampling
df = DataLoader.load_csv("large_file.csv", sample_size=10000)

With DatasetSchema

from edasuite import (
    EDARunner, DataLoader,
    ColumnConfig, ColumnType, ColumnRole, Sentinels, DatasetSchema,
)

# Load data and schema
df = DataLoader.load_csv("data.csv")
schema = DataLoader.load_schema("schema.json")

# Or create schema programmatically
schema = DatasetSchema([
    ColumnConfig('age', ColumnType.CONTINUOUS, ColumnRole.FEATURE,
                 provider='demographics', description='User age',
                 sentinels=Sentinels(not_found='-1')),
    ColumnConfig('zip_code', ColumnType.CATEGORICAL, ColumnRole.FEATURE,
                 provider='address', description='ZIP code',
                 sentinels=Sentinels(not_found='', missing='00000')),
    ColumnConfig('target', ColumnType.BINARY, ColumnRole.TARGET),
])

# Run with schema
runner = EDARunner()
results = runner.run(
    data=df,
    schema=schema,
    target_variable="target",
    output_path="eda_results.json"
)

Schema JSON format (schema.json):

{
  "columns": [
    {
      "name": "age",
      "type": "continuous",
      "role": "feature",
      "provider": "demographics",
      "description": "User age",
      "sentinels": {
        "not_found": "-1",
        "missing": null
      }
    }
  ]
}

Working with DataFrames

EDARunner works with pandas DataFrames, making it easy to integrate into existing data pipelines:

import pandas as pd
from edasuite import EDARunner

# From database
df = pd.read_sql("SELECT * FROM users", connection)

# From API
import requests
data = requests.get("https://api.example.com/data").json()
df = pd.DataFrame(data)

# In-memory transformations
df['age_group'] = pd.cut(df['age'], bins=[0, 30, 50, 100])

# Run EDA
runner = EDARunner()
results = runner.run(data=df, target_variable='target')

This is particularly useful for:

• Working in Jupyter notebooks
• Data loaded from databases (via pd.read_sql())
• In-memory transformations without saving to disk
• Integration with existing data pipelines

See examples/example_12_dataframe_input.py for more examples.

Stability Analysis

Cohort-Based (Train/Test)

from edasuite import EDARunner, DataLoader

# Load data and schema
df = DataLoader.load_parquet("data.parquet")
schema = DataLoader.load_schema("schema.json")

# Configure for stability analysis
runner = EDARunner(
    calculate_stability=True,
    cohort_column='dataTag',
    baseline_cohort='training',
    comparison_cohort='test'
)

results = runner.run(
    data=df,
    schema=schema
)

Time-Based

from edasuite import EDARunner, DataLoader

# Load data and schema
df = DataLoader.load_parquet("data.parquet")
schema = DataLoader.load_schema("schema.json")

# Configure for time-based stability
runner = EDARunner(
    time_based_stability=True,
    time_column='onboarding_time',
    time_window_strategy='monthly',  # or 'weekly', 'quartiles', 'custom'
    baseline_period='first',
    comparison_periods='all',
    min_samples_per_period=100
)

results = runner.run(
    data=df,
    schema=schema
)

DatasetSchema

DatasetSchema enables advanced functionality by defining column types, roles, providers, and sentinel values:

Variable Type Override

Override automatic type inference:

{
  "name": "customer_id",
  "type": "categorical",
  "role": "feature"
}

Sentinel Values

Define values that should be treated as missing:

{
  "name": "income",
  "type": "continuous",
  "role": "feature",
  "sentinels": {
    "not_found": "-1",
    "missing": "0"
  }
}

Provider Tracking

Track data sources:

{
  "name": "credit_score",
  "type": "continuous",
  "role": "feature",
  "provider": "bureau_provider",
  "description": "FICO credit score"
}

Output Format

EDASuite produces structured JSON output with three top-level sections:

metadata

• Timestamp, execution time, version
• Configuration (target variable, sampling, correlations)
• Schema availability indicator

summary

• Feature type distribution and counts
• Data quality score with recommendations
• Dataset info (rows, columns, memory, missing, duplicates)
• Provider match rates (if schema with providers is used)
• Feature counts (high correlation, redundant, high IV, high stability)
• Top features by statistical score

features

List of per-feature analysis, each including:

• Statistics (mean, median, mode, quartiles, etc.)
• Distribution (histogram or value counts)
• Missing values
• Quality assessment
• Correlations (with target and other features)
• Target relationship (IV, WoE if target specified)
• Stability (PSI/KS if enabled)

Provider Match Rates / Hit Rates

EDASuite automatically computes provider match rates (also called "hit rates") to help you understand data coverage from different third-party data providers.

Automatic Detection

Provider match rates are computed automatically during EDA using one of two methods:

Method 1: Using <provider>_record_not_found columns (Preferred)

If your dataset includes columns like payu_record_not_found, truecaller_record_not_found, etc., EDASuite will automatically detect and use them:

runner = EDARunner()
df = DataLoader.load_csv("data.csv")
results = runner.run(data=df)

# Access provider stats
provider_stats = results['summary']['provider_match_rates']

Method 2: Using DatasetSchema (Fallback)

If no record_not_found columns exist, you can use a schema to group features by provider:

df = DataLoader.load_csv("data.csv")
schema = DataLoader.load_schema("schema.json")

runner = EDARunner()
results = runner.run(data=df, schema=schema)

# Provider stats show match rates based on feature null analysis
provider_stats = results['summary']['provider_match_rates']

Example

See examples/example_10_provider_match_rates.py for a complete working example.

Feature Counts

EDASuite automatically computes feature counts across 4 key categories - perfect for building dashboard UIs and feature selection workflows.

Automatic Computation

Feature counts are computed automatically during EDA and included in the results:

from edasuite import EDARunner, DataLoader

runner = EDARunner()
df = DataLoader.load_csv("data.csv")
results = runner.run(
    data=df,
    target_variable="target"  # Required for correlation and IV
)

# Access feature counts
feature_counts = results['summary']['feature_counts']

print(f"High Correlation: {feature_counts['high_correlation']['count']}")
print(f"Redundant Features: {feature_counts['redundant_features']['count']}")
print(f"High IV: {feature_counts['high_iv']['count']}")
print(f"High Stability: {feature_counts['high_stability']['count']}")

Categories

Category	Threshold	Description
High Correlation	|correlation| > 0.1	Features correlated with target variable
Redundant Features	correlation > 0.7	Features highly correlated with another feature
High IV	IV > 0.1	Features with strong predictive power
High Stability	PSI/KS < 0.5	Features stable across cohorts (low drift)

Example

See examples/example_11_feature_counts.py for a complete working example with UI formatting.

Advanced Configuration

Correlation Settings

runner = EDARunner(
    top_correlations=10,           # Top N correlations per feature
    max_correlation_features=500   # Limit features in correlation matrix
)

Sampling for Large Datasets

runner = EDARunner(
    sample_size=10000  # Analyze sample of 10K rows
)

Custom Column Selection

df = DataLoader.load_csv("data.csv")
results = runner.run(
    data=df,
    columns=['age', 'income', 'zip_code']  # Analyze specific columns
)

Compact JSON Output

df = DataLoader.load_csv("data.csv")
results = runner.run(
    data=df,
    output_path="results.json",
    compact_json=True  # Minimize JSON size
)

Parquet File Benefits

Parquet format offers significant advantages:

• Faster loading: Columnar format with efficient compression
• Smaller file size: Typically 50-80% smaller than CSV
• Type preservation: Maintains data types (no type inference needed)
• Column selection: Read only needed columns (reduces memory usage)

# Convert CSV to Parquet (one-time operation)
import pandas as pd
df = pd.read_csv("data.csv")
df.to_parquet("data.parquet", index=False)

# Then use Parquet for faster analysis
df = DataLoader.load_parquet("data.parquet")
runner = EDARunner()
results = runner.run(data=df)

Development

pip install -e .           # Install for development
python -m build            # Build package
python -m pytest tests/    # Run tests

Documentation

• Architecture — internals, module structure, data flow
• Decision Records — key design decisions and rationale
• Examples — usage examples and demos

Requirements

• Python 3.9+
• pandas >= 2.0.0
• numpy >= 1.24.0
• scipy >= 1.10.0
• pyarrow >= 10.0.0 (for Parquet support)

License

MIT License - see LICENSE file for details.

Contact

For questions or suggestions:

• Email: [e-mail]
• GitHub: https://github.com/lattiq/edasuite

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.