drishtipy

Lightweight Data Profiling & Data Quality for pandas

Understand your DataFrame in seconds.

drishtipy is a lightweight, dependency-minimal pandas profiling toolkit for data quality, statistics, ML readiness, ETL analysis, PII detection, correlations, dataset comparison, and HTML reporting.

The goal is simple:

Load your DataFrame → profile it → find problems → understand the data → improve it.

---

⚡ See It in Action

import pandas as pd
import drishtipy
or
from drishtipy import profile_of  # ✅ full autocomplete + type checking

df = pd.read_csv("data.csv")

# Complete profile
df.profile.info() or profile_of(df).info()

# Data structure
df.profile.schema()

# Statistical analysis
df.profile.statistics()

# Data quality analysis
df.profile.quality()

# ML readiness & suggestions
df.profile.ml()

# ETL readiness & cleaning issues
df.profile.etl()

# Overall data quality score
df.profile.quality_score()

# Data quality alerts
df.profile.alerts()

# Potential PII detection
df.profile.pii()

# Correlation analysis
df.profile.correlations(by="pairs")

# Generate a shareable HTML report
df.profile.html("report.html")

# Compares two DataFrames and generates an HTML before/after report
df.profile.compare_html(df2, path="compare.html")

# Auto-discover relationships across every column pair
df.profile.relationships()
or 
df.profile.relationships().to_html("relationships.html")  
or
df.profile.relationships().to_html("relationships.html", style="table")

That's it. No complex configuration. No wrapper object required.

Just import drishtipy and use:

df.profile

---

🎯 What Can drishtipy Tell You?

Give drishtipy a pandas DataFrame and quickly answer questions like:

• What columns does my dataset contain?
• Which columns have missing values?
• Where are the duplicates?
• Which numeric columns contain outliers?
• How good is the overall data quality?
• Which columns may contain PII?
• Is this dataset ready for machine learning?
• What problems could affect an ETL pipeline?
• Which numeric features are strongly correlated?
• What changed after cleaning my data?
• Can I generate a report to share with someone else?

---

🚀 One DataFrame. Multiple Insights.

                         pandas DataFrame
                                │
                                ▼
                         ┌─────────────┐
                         │  drishtipy  │
                         └─────────────┘
                                │
          ┌─────────────┬───────┼────────┬──────────────┐
          ▼             ▼       ▼        ▼              ▼
       Schema       Statistics Quality   ML             ETL
          │             │       │        │              │
          │             │       ▼        ▼              ▼
          │             │   Quality    Readiness     Cleaning
          │             │    Score                   Issues
          │             │       │
          │             ▼       ▼
          │        Correlations Alerts
          │
          ├──────────────► PII Detection
          │
          ├──────────────► Before / After Comparison
          │
          └──────────────► HTML Report

---

⭐ Core Features

Feature	Purpose
Schema	Understand columns, dtypes, missing values, uniqueness and memory
Statistics	Descriptive statistics, quantiles, skewness, kurtosis and more
Quality	Missing values, duplicates, zeros and outliers
Quality Score	Overall and per-column data quality scoring
Alerts	High, medium and low severity data-quality issues
PII Detection	Detect potentially sensitive and identifiable data
Correlations	Find relationships and multicollinearity candidates
ML Readiness	Analyze feature suitability for machine learning
ETL Readiness	Identify common cleaning and transformation problems
Comparison	Compare DataFrames before and after transformations
HTML Reports	Generate standalone, shareable reports
Large CSV	Profile large CSV files using chunking and sampling
Relationship Discovery	Auto-detect, score, and rank relationships between every pair of columns

---

📦 Installation

pip install drishtipy

Then:

import pandas as pd
import drishtipy

df = pd.read_csv("data.csv")

df.profile.quality()

Development installation

pip install -e .

For development dependencies:

pip install -e ".[dev]"

---

🐼 Designed for pandas

drishtipy extends the familiar pandas workflow with a .profile accessor.

Instead of:

DataProfiler(df).info_dataframe(
    section="quality"
)

you can simply write:

df.profile.quality()

The profiling results remain pandas-friendly, so you can continue to use normal pandas operations:

report = df.profile.quality()

report.sort_values(
    "Missing Count",
    ascending=False
)

Filter them:

report[
    report["Missing Count"] > 0
]

Or export them:

report.to_csv(
    "quality_report.csv",
    index=False
)

---

📋 Schema Profiling

Inspect the structure of your DataFrame:

df.profile.schema()

The schema report includes one row per column with:

• Column name and pandas dtype
• Non-null count, missing count, missing %
• Unique count, unique %
• Memory usage (bytes)

---

📊 Statistical Profiling

Generate descriptive statistics:

df.profile.statistics()

One row per column, with (for numeric columns) count, mean, median, std, min/Q1/Q2/Q3/95th/99th percentile/max, range, IQR, skewness, and kurtosis — plus mode and mode frequency for every column regardless of dtype.

---

🔍 Data Quality Profiling

Analyze common data-quality problems:

df.profile.quality()

The quality report includes:

• Duplicate value count (per column)
• Missing value count
• Zero count (numeric columns)
• IQR-based outlier count and outlier % (numeric columns)

Example:

Column    Duplicate Count    Missing Count    Zero Count    Outlier Count    Outlier %
------------------------------------------------------------------------------------
age                     0                1             0                0        0.00
salary                  0                0             0                1       25.00
city                    1                0          None             None        None

---

⭐ Quality Score

Get a quick, weighted assessment of your dataset across five dimensions — Missing Values, Duplicates, Outliers, Data Types, and Invalid Values:

df.profile.quality_score()

Example output (a Metric / Score DataFrame):

                 Metric  Score
 Overall Quality Score   88.4
         Missing Values  92.0
             Duplicates 100.0
               Outliers  71.0
             Data Types  95.0
         Invalid Values  84.0

For a column-level breakdown (one row per column, plus a per-column Quality Score):

df.profile.quality_score(by="column")

Filter by column type:

df.profile.quality_score(column_type="numeric")

Custom scoring weights (unspecified dimensions keep their default weight of 20; values are normalized automatically, so they don't need to sum to 100):

df.profile.quality_score(
    weights={
        "missing": 40,
        "outliers": 30,
    }
)

Note: Duplicates at the overall level measures fully duplicated rows. At the column level (by="column"), it measures repeated values within that column instead — the two aren't the same metric.

---

🚨 Data Quality Alerts

Find important data-quality problems automatically, ranked by severity, in one table instead of reading every section separately:

df.profile.alerts()

Returns a DataFrame with columns Severity, Column, Alert, and Details (table-wide issues like duplicate rows or correlated pairs use "(table)" as the column value). Detected alert types include: Missing Values, Duplicate Rows, Outliers, Constant Column, Near-Constant Column, Skewed Distribution, High Cardinality, Possible ID Column, Wrong Data Type, Empty Strings, Extra Whitespace, Zero-Heavy Column, Highly Correlated Pair, and Potential PII.

Filter for critical issues only:

df.profile.alerts(min_severity="high")

min_severity accepts "high" (High only), "medium" (High + Medium), or "low" (everything — the default).

---

🔐 PII Detection

Detect potentially personally identifiable information:

df.profile.pii()

Returns a DataFrame — Column, PII Type, Confidence % — sorted by confidence, descending. Detected types:

Type	Meaning
EMAIL	Email addresses
PHONE	Phone numbers (Indian mobile pattern + general)
POSSIBLE_ID	Aadhaar-shaped 12-digit numbers
POSSIBLE_PAN	Indian PAN format (ABCDE1234F)
IP	IPv4 addresses
POSSIBLE_NAME	Free-text columns that look like personal names
POSSIBLE_ADDRESS	Free-text columns that look like postal addresses

The POSSIBLE_ prefix is a reminder that ID/PAN/Name/Address detections are pattern/heuristic guesses, not verified PII — always review before acting on them (e.g. before dropping or publishing columns). EMAIL, PHONE, and IP use stricter pattern matching.

Filter by confidence:

df.profile.pii(min_confidence=80)

For large DataFrames, only the first sample non-null values per column are scanned by default (for speed):

df.profile.pii(sample=2000)   # default
df.profile.pii(sample=None)   # scan every value instead

Generate a masked copy instead of a report:

masked_df = df.profile.pii(mask=True)

mask=True returns a masked copy of the DataFrame — the original is never modified. Masking is type-aware (emails keep the domain, phone/ID numbers keep the last few digits, addresses are fully redacted, etc.).

---

🧮 Correlation Analysis

Analyze relationships between numerical columns (Pearson by default):

df.profile.correlations()

Returns the full column-by-column correlation matrix. Get a tidy, one-row-per-pair view instead, sorted by strength:

df.profile.correlations(by="pairs")

Find only strongly correlated pairs — useful for spotting multicollinearity candidates before modeling:

df.profile.correlations(by="pairs", threshold=0.8)

Supported methods:

df.profile.correlations(method="pearson")
df.profile.correlations(method="spearman")
df.profile.correlations(method="kendall")

Raises ValueError if fewer than two numeric columns are available.

---

🤖 ML Readiness

Analyze whether your DataFrame is suitable for machine-learning workflows:

df.profile.ml()

One row per column, including: feature type, unique count/%, missing %, variance, skewness, outlier %, cardinality, encoding suggestion, scaling suggestion, recommended transformation, and an overall feature status (e.g. "Good") flagging columns that may need attention before modeling.

---

🔄 ETL Readiness

Analyze your DataFrame for common ETL and cleaning problems:

df.profile.etl()

One row per column, including: missing/duplicate/unique counts and %, zero count, negative count, empty-string count, whitespace-issue count, special-character count, outlier count/%, an Issue/Issue Count summary, an ETL Status (e.g. "Ready"), and a Recommended Action.

---

🔬 Before / After DataFrame Comparison

drishtipy includes DataComparator for comparing DataFrames before and after transformations.

Useful for:

• ETL pipelines
• Data cleaning
• Data transformation
• Feature engineering
• Data validation
• Preprocessing workflows

from drishtipy import DataComparator

comparator = DataComparator()

comparison = comparator.compare_dataframe(
    before_df,
    after_df
)

Or through the pandas accessor:

df.profile.compare(df2)

Comparison levels

Supported levels:

All
Dataset
Schema
Quality
ML

("quality" and "ml" currently produce the same column-level report.)

Example:

comparator.compare_dataframe(
    before_df,
    after_df,
    level="quality"
)

The result is a long-format DataFrame with columns Section, Metric, Before, After, Change, and Status (not every row populates every column). Raises TypeError if either argument isn't a DataFrame, and ValueError for an unrecognized level.

Changes only

compare_dataframe can return dozens of rows on wide DataFrames. Keep only the rows where something actually changed:

comparator.compare_dataframe(
    before_df,
    after_df,
    changes_only=True
)

Compact summary

A one-row-per-column view instead of the long-format table:

comparator.summary(
    before_df,
    after_df
)

   Column     Status                          Summary
0     age    Changed  missing 1->0; outliers 1->0; mean 122.5->35.17
1    city  Unchanged                         No change
2 new_col      Added                     Column added
3 old_col    Removed                   Column removed

Specific columns

comparator.compare_dataframe(
    before_df,
    after_df,
    level="quality",
    columns=["age", "salary"]
)

A column missing from one side is still reported as added/removed. Raises ValueError if a name in columns isn't present in either DataFrame. Note: level="dataset" metrics (row/column counts, memory, etc.) always reflect the full before/after DataFrames, not the selection.

---

🌐 HTML Reports

Generate a standalone, dark-themed HTML report instead of raw DataFrames — handy for sharing a report without a notebook:

df.profile.html(
    "profile_report.html"
)

Comparison reports:

df.profile.compare_html(
    df2,
    path="comparison_report.html"
)

Explicit classes also support HTML output:

DataProfiler(df).to_html(
    path="profile_report.html"
)

DataComparator().to_html(
    before_df,
    after_df,
    path="comparison_report.html"
)

Both accept the same filtering arguments as their DataFrame-returning counterparts (section/column_type for DataProfiler, level/ changes_only/columns for DataComparator), plus title for a custom page heading. Omitting path returns the HTML as a string instead of writing a file.

---

🕸️ Relationship Discovery

For a 50-column dataset, there are 50×49/2 = 1,225 possible column pairs — nobody is manually checking all of them. relationships() scans every pair automatically, picks a statistically appropriate method based on each column's semantic type (not just its pandas dtype), filters out pairs that can't be meaningfully compared, and ranks what's left:

result = df.profile.relationships()
result          # DRISHTIPY — RELATIONSHIP DISCOVERY dashboard, like info()

DRISHTIPY — RELATIONSHIP DISCOVERY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Columns analyzed:       15
Possible pairs:         105
Pairs analyzed:         91
Ignored pairs:          20

Meaningful relationships: 27
Strong:                    8
Moderate:                  7
Weak:                      12

TOP RELATIONSHIPS

  Region             ↔ City               1.00 🔥
  Category           ↔ Product            1.00 🔥
  Sales_Amount       ↔ Cost               1.00 🔥
  Sales_Amount       ↔ Profit             0.98 🔥
  Cost               ↔ Profit             0.96 🔥

result.top(20) / .matrix() / .graph() / .insights() / .dependencies()

What gets compared, and how

Each pair is routed to a method based on semantic type (drishtipy.semantic.detect_semantic_type — id, numeric, categorical, boolean, date/datetime, email, phone, currency, percentage, text, constant), not raw pandas dtype:

Pair type	Method	Notes
numeric ↔ numeric	Pearson and Spearman	The stronger of the two wins — a monotonic non-linear relationship that Pearson underrates still gets caught
categorical ↔ numeric	ANOVA effect size (η²)	Also reports a p-value from an F-test when scipy is installed
categorical ↔ categorical	Cramér's V + normalized Mutual Information	Built from a contingency table with pandas/numpy only — no scipy required for the effect size itself
date ↔ numeric	Spearman trend	Direction reported as Increasing / Decreasing / Flat

Pairs that can't be meaningfully compared are filtered out before any statistics run: ID ↔ anything, constant columns, free text, and email/phone columns are excluded by default (RelationshipConfig(include_id_pairs=True) opts back in for advanced use).

Configuring thresholds

from drishtipy import RelationshipConfig

result = df.profile.relationships(
    RelationshipConfig(
        strong_threshold=0.80,
        moderate_threshold=0.50,
        min_sample_size=30,
        max_categories=50,     # categorical columns above this are skipped
        max_pairs=5_000,       # safety cap for very wide DataFrames
        sample_size=100_000,   # rows; None disables sampling
    )
)

If a pair doesn't have enough valid (non-null, overlapping) observations, it's reported as "Not enough valid observations" rather than a misleading score.

Views on the result

result.top(20)               # ranked DataFrame of the strongest relationships
result.matrix()               # symmetric strength matrix (method varies per cell — see below)
result.graph(top=20)          # RelationshipGraph: nodes + weighted edges
result.graph(threshold=0.70)  # only edges at/above a strength cutoff
result.insights()             # auto-generated, human-readable observations
result.dependencies()         # potential functional dependencies (farmer_id -> farmer_name)
result.redundant_columns()    # column groups that look like duplicates/derived copies
result.to_dataframe()         # the full, unfiltered results table
result.to_json()
result.to_html("relationships.html")

result.to_html() renders a standalone KPI-card dashboard by default — top relationships, an inline-SVG relationship graph, color-coded insight cards, and a dependencies list, all computed from the actual result (not a static mockup):

result.to_html("relationships.html")                    # dashboard style (default)
result.to_html("relationships.html", style="table")      # plain multi-section tables instead
result.to_html("relationships.html", top_n=10, graph_top=15)

The graph is real SVG with a computed circular layout (works for any number of nodes), not a fixed set of hand-placed positions — the most-connected column gets a highlighted "hub" style, matching how it's usually the one worth investigating first.

result.matrix() is not a plain correlation matrix — each cell's value comes from whichever method fit that pair's semantic types, so don't treat the whole matrix as linear-correlation strength.

result.graph() returns nodes/edges and a readable adjacency-list view — not a rendered image. Building an actual force-directed layout needs a graph/plotting stack (networkx, matplotlib) this library intentionally doesn't depend on; result.graph().edges is meant to be handed to one of those directly if you want a visual layout.

Functional dependencies and redundant columns

Beyond correlation, relationships() checks whether one column's values consistently determine another's (e.g. farmer_id -> farmer_name), and flags column groups that appear to store the same information twice (exact duplicates, or near-perfectly correlated numeric/categorical pairs) — language is deliberately hedged ("Potential functional dependency", consistency %) rather than claiming certainty the data may not fully support.

Multiple testing and significance

With hundreds of pairs analyzed at once, some correlations will look "significant" by chance alone. When scipy is installed (pip install drishtipy[relationships]), p-values are computed per pair and corrected for multiple testing via Benjamini-Hochberg FDR by default (RelationshipConfig(fdr_correction=False) to disable). Without scipy, effect sizes (Pearson/Spearman/η²/Cramér's V) are still computed — Significance just reports "Unknown (scipy not installed)" instead of guessing.

Integration

df.profile.relationships()          # via the accessor
df.profile.info().relationships()   # chained from the full profile report
DataProfiler(df).relationships()    # via the explicit class API

---

📁 Large CSV Profiling

Profile large CSV files without loading the entire source into memory:

from drishtipy import DataProfiler

profiler = DataProfiler.from_csv(
    "huge_dataset.csv",
    sample_size=100_000,
    chunksize=50_000,
    random_state=42
)

Then:

profiler.info_dataframe()

Check whether sampling was used:

profiler.is_sampled

Get the exact source row count:

profiler.total_rows_in_source

How sampling works

CSV file
   │
   ├── Pass 1
   │     └── Count total rows
   │
   ├── Pass 2
   │     └── Select a random sample
   │
   └── Profile the sample

The file is streamed in chunks twice — once to count exact total rows, once to keep each row with probability sample_size / total_rows (a random, roughly-uniform sample). If the file has fewer rows than sample_size, it's loaded in full and is_sampled stays False.

When sampling is used, the source row count (total_rows_in_source) is exact, while statistics/ML sections computed from the sample are estimates of the complete dataset. to_html() on a sampled profiler automatically notes the sample size vs. total rows in the report.

Additional keyword arguments can be forwarded to pandas.read_csv():

DataProfiler.from_csv(
    "data.csv",
    usecols=["age", "salary"],
    dtype={"age": "float64"},
    sep=",",
    parse_dates=["date"]
)

---

🧱 Explicit Class API

The pandas accessor is recommended, but explicit classes are also available — useful if you want to keep the profiler object around for repeated calls, or prefer not to rely on accessor registration:

import pandas as pd

from drishtipy import DataProfiler

df = pd.DataFrame({
    "age": [25, 32, None, 47],
    "city": ["Delhi", "Mumbai", "Delhi", "Pune"]
})

profiler = DataProfiler(df)

profiler.info_dataframe(
    section="schema"
)

profiler.quality_score()
profiler.alerts()
profiler.pii()
profiler.correlations()

Raises TypeError if df isn't a pandas.DataFrame.

---

⚙️ info_dataframe()

The underlying profiler API provides:

profiler.info_dataframe(
    section="All",
    column_type="All"
)

section

Supported values (case-insensitive):

All
Schema
Statistics
Quality
ML
ETL

"All" returns a dict mapping section name -> DataFrame:

{
    "Schema": ...,
    "Statistics": ...,
    "Quality": ...,
    "ML": ...,
    "ETL": ...
}

Any other value returns just that section's DataFrame.

column_type

Supported values (case-insensitive):

All
Numeric
Categorical

Example:

profiler.info_dataframe(
    section="statistics",
    column_type="numeric"
)

Raises ValueError if section or column_type isn't recognized, or if column_type filters out every column.

---

🧠 Design Philosophy

pandas-first

drishtipy extends pandas rather than replacing it.

Lightweight

The core package intentionally keeps dependencies minimal — pandas is the only required dependency.

Non-destructive

Profiling operations analyze the DataFrame without modifying it. The one exception is explicit: df.profile.pii(mask=True) returns a masked copy, never mutating the original.

Composable

Reports remain compatible with normal pandas operations — sort, filter, export, or display them however you like.

Human-readable

HTML reports and summaries are designed to make data problems easy to understand at a glance.

---

🧪 Development

Install development dependencies:

pip install -e ".[dev]"

Run tests:

pytest

Recommended workflow:

git clone <repository>
cd drishtipy

pip install -e ".[dev]"

pytest

---

🗺️ Roadmap

Potential future capabilities include:

• Advanced validation rules
• Automatic cleaning suggestions
• Smart semantic data-type/data-dictionary detection
• Time-series profiling
• Advanced ML-readiness checks
• Data drift detection
• Statistical distribution analysis
• Excel / JSON / Parquet / SQL database profiling
• Expanded interactive HTML dashboards
• Additional data-quality checks

---

📦 Package Information

Property	Value
Package	drishtipy
Import	drishtipy
Current Version	0.6.0
Python	>=3.8
License	MIT
Primary Dependency	pandas (>=1.3)

Install:

pip install drishtipy

Import:

import drishtipy

Use:

df.profile.quality()

---

📄 License

drishtipy is released under the MIT License.