pysuricata 0.0.16

A lightweight EDA tool inspired by the curious nature of suricates. Built just for fun 🔬.

PySuricata

Exploratory Data Analysis for Python, Built on Streaming Algorithms

Quick Start • Documentation • Examples

---

What It Does

PySuricata generates self-contained HTML reports from pandas or polars DataFrames. Reports include per-column statistics, histograms, correlation chips, missing value analysis, and outlier detection.

Data is processed in chunks using streaming algorithms, so memory usage stays bounded regardless of dataset size.

Quick Start

Installation

pip install pysuricata

With polars support:

pip install pysuricata[polars]

Generate a Report

import pandas as pd
from pysuricata import profile

url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
df = pd.read_csv(url)

report = profile(df)
report.save_html("titanic_report.html")

▶ See a live example report →

Features

• Streaming architecture — Data is processed in configurable chunks, keeping memory bounded. Useful for datasets that don't fit in RAM.
• Pandas and Polars — Works natively with pandas.DataFrame, polars.DataFrame, and polars.LazyFrame.
• Self-contained HTML — Single file with inline CSS, JS, and SVG charts. No external assets needed.
• Configurable — Control chunk sizes, sample sizes, sketch parameters, and correlation thresholds via ReportConfig.
• Reproducible — Seeded random sampling produces deterministic results across runs.
• CLI tool — Profile datasets from the command line.

How It Works

PySuricata uses well-known streaming algorithms from the academic literature:

Algorithm	Purpose	Time	Space
Welford/Pébay	Exact mean, variance, skewness, kurtosis	O(1) per value	O(1)
KMV sketch	Distinct count estimation (~2.2% error)	O(log k) per value	O(k)
Misra-Gries	Top-k frequent values	O(1) amortized	O(k)
Reservoir sampling	Uniform random sample for quantiles	O(1) per value	O(s)

k = sketch size (default 1024), s = sample size (default 10 000)

All statistics are computed in a single pass over the data.

What's in a Report

Each column is analyzed based on its type:

• Numeric — Mean, variance, skewness, kurtosis, quantiles, histogram, outlier detection (IQR, MAD, z-score), correlations
• Categorical — Top values, distinct count, entropy, Gini impurity, string length statistics
• DateTime — Temporal range, hour/day/month distributions, monotonicity detection
• Boolean — True/false ratios, entropy, balance score

Plus dataset-level metrics: row/column counts, memory usage, missing value percentages, and duplicate row estimates.

Streaming Large Datasets

Process datasets larger than RAM by passing a generator:

import pandas as pd
from pysuricata import profile

def read_in_chunks():
    for i in range(100):
        yield pd.read_parquet(f"data/part-{i}.parquet")

report = profile(read_in_chunks())
report.save_html("large_report.html")

Statistics Only (No HTML)

Use summarize() for CI/CD quality checks:

from pysuricata import summarize

stats = summarize(df)

assert stats["dataset"]["missing_cells_pct"] < 5.0
assert stats["dataset"]["duplicate_rows_pct_est"] < 1.0

print(f"Mean age: {stats['columns']['age']['mean']:.1f}")

Configuration

from pysuricata import profile, ReportConfig

config = ReportConfig()
config.compute.chunk_size = 250_000
config.compute.random_seed = 42
config.compute.compute_correlations = True
config.compute.corr_threshold = 0.5
config.render.title = "My Analysis"

report = profile(df, config=config)

See the Configuration Guide for all options.

CLI

# Generate HTML report
pysuricata profile data.csv --output report.html

# Get JSON statistics
pysuricata summarize data.csv

Documentation

• Quick Start
• User Guide
• Configuration
• API Reference
• Statistical Methods
• Examples

Contributing

Contributions are welcome. See the Contributing Guide.

git clone https://github.com/alvarodiez20/pysuricata.git
cd pysuricata
uv sync --dev
uv run pytest

License

MIT License. See LICENSE for details.

Acknowledgments

Built using algorithms from:

• Welford, B.P. (1962) — Streaming moments
• Pébay, P. (2008) — Parallel merging of moments
• Bar-Yossef, Z. et al. (2002) — KMV distinct count estimation
• Misra, J. & Gries, D. (1982) — Streaming heavy hitters

Named after suricatas (meerkats) — small, vigilant animals that work cooperatively and thrive in harsh environments with limited resources.