meti_profil

A modern, Rust-powered data profiling library with Python bindings. It reads CSV, Parquet, and Excel files (or pandas / polars DataFrames) and generates a hybrid Markdown report that is readable by humans and structured for consumption by code agents.

Installation

pip install meti-profil

The import name is meti_profil (import meti_profil as mp). PyPI normalizes the distribution name, so pip install meti_profil also works.

Quick start

import meti_profil as mp

# From a file
report = mp.ProfileReport("data.csv", title="My dataset")

# Interactive HTML report (self-contained, works offline)
report.to_html("profile.html")

# Markdown report (great for diffs and code agents)
report.to_file("profile.md")

# From a pandas DataFrame
import pandas as pd
df = pd.read_csv("data.csv")
report = mp.ProfileReport(df)

# Programmatic access
print(report.get_summary())          # dataset-level metrics
print(report.get_column_info("age")) # per-column schema info
markdown = report.to_markdown()
html = report.to_html()              # returns the HTML as a string

In a notebook

In Jupyter / VSCode, just display the report — it renders inline as an interactive dashboard (sandboxed, no external resources):

report = mp.ProfileReport(df)
report  # interactive histograms, bar charts, correlation heatmap, ...

ProfileReport parameters

Report format

The Markdown report starts with a quoted YAML frontmatter block (rows, columns, missing cells, duplicates, version), then an ## Agent Summary JSON block for code agents, followed by normalized ## sections: Overview, Schema, Numeric Columns, Categorical Columns, Missing Values, Duplicate Rows, and Correlations when high-correlation pairs are present.

Because the agent summary is compact and stable JSON, coding agents can inspect key dataset diagnostics without reading the full report or parsing long Markdown tables. This can reduce context usage and make downstream automation more reliable, especially when reports are generated repeatedly in data workflows.

Features

• Fast Rust engine backed by Apache Arrow.
• Reads CSV, Parquet (snappy/zstd/lz4/brotli/gzip), and Excel files.
• Accepts pandas and polars DataFrames.
• Schema/type detection, descriptive numeric statistics across Arrow integer and float types, categorical frequencies, missing-value and duplicate-row analysis, and Pearson correlations. Null, NaN and infinite numeric values are treated as unavailable for numeric statistics/correlations.
• Interactive HTML report: a single self-contained file (embedded CSS/JS, no CDN) with section navigation, responsive tables, histograms, categorical bar charts, a missing-value overview and a correlation heatmap with safe SVG tooltips.
• Native notebook rendering in Jupyter / VSCode via _repr_html_.
• Clean Markdown reports optimized for both humans and code agents, including escaped table cells and a stable JSON summary block.

Output formats

Development

Requires a Rust toolchain (1.78+) and Python 3.10+.

python3 -m venv .venv
source .venv/bin/activate
pip install maturin pytest pandas polars pyarrow

# Build the extension in-place
maturin develop

# Run the test suites
cargo test --workspace
pytest tests/python -v

License

MIT