drishtipy 0.6.1

drishtipy

Lightweight Data Profiling & Data Quality for pandas (Fast Pandas Profiling Alternative)

Understand your DataFrame in seconds.

drishtipy is a lightweight, dependency-minimal pandas profiling toolkit for data quality, statistics, ML readiness, ETL analysis, PII detection, correlations, dataset comparison, and HTML reporting. It extends the native pandas workflow to catch schema errors, trace data quality metrics, and automate privacy checks in a single line of code.

The goal is simple:

Load your DataFrame → profile it → find problems → understand the data → improve it.

---

⚡ See It in Action

import pandas as pd
import drishtipy
or
from drishtipy import profile_of  # ✅ full autocomplete + type checking

df = pd.read_csv("data.csv")

# Complete profile
df.profile.info() or profile_of(df).info()

# Data structure
df.profile.schema()

# Statistical analysis
df.profile.statistics()

# Data quality analysis
df.profile.quality()

# ML readiness & suggestions
df.profile.ml()

# ETL readiness & cleaning issues
df.profile.etl()

# Overall data quality score
df.profile.quality_score()

# Data quality alerts
df.profile.alerts()

# Potential PII detection
df.profile.pii()

# Correlation analysis
df.profile.correlations(by="pairs")

# Generate a shareable HTML report
df.profile.html("report.html")

# Compares two DataFrames and generates an HTML before/after report
df.profile.compare_html(df2, path="compare.html")

✨ **NEW:** This feature has been added recently.
# Auto-discover relationships across every column pair
df.profile.relationships()
or 
df.profile.relationships().to_html("relationships.html")  
or
df.profile.relationships().to_html("relationships.html", style="table")

✨ **NEW:** This feature has been added recently.
#utoML: benchmark regression models, no sklearn import needed
result = df.ml.auto_predict(target="Sales")
✨ **NEW:** This feature has been added recently.
# Lazy, composable ETL pipeline
clean = (       
    drishtipy.ETL()
    .extract("data.csv")
    .text("name").strip().title()
    .fill_missing("age", strategy="median")
    .load("clean.csv", report="etl_report.html")

That's it. No complex configuration. No wrapper object required.

Just import drishtipy and use:

df.profile

---

🎯 What Can drishtipy Tell You?

Give drishtipy a pandas DataFrame and quickly answer questions like:

• What columns does my dataset contain?
• Which columns have missing values?
• Where are the duplicates?
• Which numeric columns contain outliers?
• How good is the overall data quality?
• Which columns may contain PII?
• Is this dataset ready for machine learning?
• What problems could affect an ETL pipeline?
• Which numeric features are strongly correlated?
• What changed after cleaning my data?
• Can I generate a report to share with someone else?

---

🚀 One DataFrame. Multiple Insights.

                         pandas DataFrame
                                │
                                ▼
                         ┌─────────────┐
                         │  drishtipy  │
                         └─────────────┘
                                │
          ┌─────────────┬───────┼────────┬──────────────┐
          ▼             ▼       ▼        ▼              ▼
       Schema       Statistics Quality   ML             ETL
          │             │       │        │              │
          │             │       ▼        ▼              ▼
          │             │   Quality    Readiness     Cleaning
          │             │    Score                   Issues
          │             │       │
          │             ▼       ▼
          │        Correlations Alerts
          │
          ├──────────────► PII Detection
          │
          ├──────────────► Before / After Comparison
          │
          └──────────────► HTML Report

---

⭐ Core Features

Feature	Purpose
Schema	Understand columns, dtypes, missing values, uniqueness and memory
Statistics	Descriptive statistics, quantiles, skewness, kurtosis and more
Quality	Missing values, duplicates, zeros and outliers
Quality Score	Overall and per-column data quality scoring
Alerts	High, medium and low severity data-quality issues
PII Detection	Detect potentially sensitive and identifiable data
Correlations	Find relationships and multicollinearity candidates
ML Readiness	Analyze feature suitability for machine learning
ETL Readiness	Identify common cleaning and transformation problems
Comparison	Compare DataFrames before and after transformations
HTML Reports	Generate standalone, shareable reports
Large CSV	Profile large CSV files using chunking and sampling
Relationship Discovery	Auto-detect, score, and rank relationships between every pair of columns
AutoML Regression	Benchmark multiple regression models automatically, no scikit-learn import required
ETL Pipeline	Lazy, composable, pandas-native pipeline — extract, clean, transform, validate, quality-score

---

📦 Installation

pip install drishtipy

Then:

import pandas as pd
import drishtipy

df = pd.read_csv("data.csv")

df.profile.quality()

Development installation

pip install -e .

For development dependencies:

pip install -e ".[dev]"

---

🐼 Designed for pandas

drishtipy extends the familiar pandas workflow with a .profile accessor.

Instead of:

DataProfiler(df).info_dataframe(
    section="quality"
)

you can simply write:

df.profile.quality()

The profiling results remain pandas-friendly, so you can continue to use normal pandas operations:

report = df.profile.quality()

report.sort_values(
    "Missing Count",
    ascending=False
)

Filter them:

report[
    report["Missing Count"] > 0
]

Or export them:

report.to_csv(
    "quality_report.csv",
    index=False
)

---

📋 Schema Profiling

Inspect the structure of your DataFrame:

df.profile.schema()

The schema report includes one row per column with:

• Column name and pandas dtype
• Non-null count, missing count, missing %
• Unique count, unique %
• Memory usage (bytes)

---

📊 Statistical Profiling

Generate descriptive statistics:

df.profile.statistics()

One row per column, with (for numeric columns) count, mean, median, std, min/Q1/Q2/Q3/95th/99th percentile/max, range, IQR, skewness, and kurtosis — plus mode and mode frequency for every column regardless of dtype.

---

🔍 Data Quality Profiling

Analyze common data-quality problems:

df.profile.quality()

The quality report includes:

• Duplicate value count (per column)
• Missing value count
• Zero count (numeric columns)
• IQR-based outlier count and outlier % (numeric columns)

Example:

Column    Duplicate Count    Missing Count    Zero Count    Outlier Count    Outlier %
------------------------------------------------------------------------------------
age                     0                1             0                0        0.00
salary                  0                0             0                1       25.00
city                    1                0          None             None        None

---

⭐ Quality Score

Get a quick, weighted assessment of your dataset across five dimensions — Missing Values, Duplicates, Outliers, Data Types, and Invalid Values:

df.profile.quality_score()

Example output (a Metric / Score DataFrame):

                 Metric  Score
 Overall Quality Score   88.4
         Missing Values  92.0
             Duplicates 100.0
               Outliers  71.0
             Data Types  95.0
         Invalid Values  84.0

For a column-level breakdown (one row per column, plus a per-column Quality Score):

df.profile.quality_score(by="column")

Filter by column type:

df.profile.quality_score(column_type="numeric")

Custom scoring weights (unspecified dimensions keep their default weight of 20; values are normalized automatically, so they don't need to sum to 100):

df.profile.quality_score(
    weights={
        "missing": 40,
        "outliers": 30,
    }
)

Note: Duplicates at the overall level measures fully duplicated rows. At the column level (by="column"), it measures repeated values within that column instead — the two aren't the same metric.

---

🚨 Data Quality Alerts

Find important data-quality problems automatically, ranked by severity, in one table instead of reading every section separately:

df.profile.alerts()

Returns a DataFrame with columns Severity, Column, Alert, and Details (table-wide issues like duplicate rows or correlated pairs use "(table)" as the column value). Detected alert types include: Missing Values, Duplicate Rows, Outliers, Constant Column, Near-Constant Column, Skewed Distribution, High Cardinality, Possible ID Column, Wrong Data Type, Empty Strings, Extra Whitespace, Zero-Heavy Column, Highly Correlated Pair, and Potential PII.

Filter for critical issues only:

df.profile.alerts(min_severity="high")

min_severity accepts "high" (High only), "medium" (High + Medium), or "low" (everything — the default).

---

🔐 PII Detection

Detect potentially personally identifiable information:

df.profile.pii()

Returns a DataFrame — Column, PII Type, Confidence % — sorted by confidence, descending. Detected types:

Type	Meaning
EMAIL	Email addresses
PHONE	Phone numbers (Indian mobile pattern + general)
POSSIBLE_ID	Aadhaar-shaped 12-digit numbers
POSSIBLE_PAN	Indian PAN format (ABCDE1234F)
IP	IPv4 addresses
POSSIBLE_NAME	Free-text columns that look like personal names
POSSIBLE_ADDRESS	Free-text columns that look like postal addresses

The POSSIBLE_ prefix is a reminder that ID/PAN/Name/Address detections are pattern/heuristic guesses, not verified PII — always review before acting on them (e.g. before dropping or publishing columns). EMAIL, PHONE, and IP use stricter pattern matching.

Filter by confidence:

df.profile.pii(min_confidence=80)

For large DataFrames, only the first sample non-null values per column are scanned by default (for speed):

df.profile.pii(sample=2000)   # default
df.profile.pii(sample=None)   # scan every value instead

Generate a masked copy instead of a report:

masked_df = df.profile.pii(mask=True)

mask=True returns a masked copy of the DataFrame — the original is never modified. Masking is type-aware (emails keep the domain, phone/ID numbers keep the last few digits, addresses are fully redacted, etc.).

🔐 PII & Sensitive Data

DrishtiPy includes local PII detection and analysis capabilities to help users identify potentially sensitive or personally identifiable information (PII) in datasets.

⚠️ Privacy & Legal Notice: DrishtiPy processes data locally on the user's device and does not transmit user data to external servers. The PII detection provided by DrishtiPy is an analytical aid and should not be considered a legal determination of whether data is personal, sensitive, regulated, or lawful to process.

Users are solely responsible for ensuring that their use, storage, processing, disclosure, and handling of personal, sensitive, or confidential data complies with all applicable laws, regulations, contractual obligations, and organizational policies in their jurisdiction.

Users should obtain appropriate authorization and apply suitable security and privacy controls before processing personal or sensitive data.

DrishtiPy does not provide legal advice and does not guarantee compliance with any specific law or regulation.

---

🧮 Correlation Analysis

Analyze relationships between numerical columns (Pearson by default):

df.profile.correlations()

Returns the full column-by-column correlation matrix. Get a tidy, one-row-per-pair view instead, sorted by strength:

df.profile.correlations(by="pairs")

Find only strongly correlated pairs — useful for spotting multicollinearity candidates before modeling:

df.profile.correlations(by="pairs", threshold=0.8)

Supported methods:

df.profile.correlations(method="pearson")
df.profile.correlations(method="spearman")
df.profile.correlations(method="kendall")

Raises ValueError if fewer than two numeric columns are available.

---

🤖 ML Readiness

Analyze whether your DataFrame is suitable for machine-learning workflows:

df.profile.ml()

One row per column, including: feature type, unique count/%, missing %, variance, skewness, outlier %, cardinality, encoding suggestion, scaling suggestion, recommended transformation, and an overall feature status (e.g. "Good") flagging columns that may need attention before modeling.

---

🔄 ETL Readiness

Analyze your DataFrame for common ETL and cleaning problems:

df.profile.etl()

One row per column, including: missing/duplicate/unique counts and %, zero count, negative count, empty-string count, whitespace-issue count, special-character count, outlier count/%, an Issue/Issue Count summary, an ETL Status (e.g. "Ready"), and a Recommended Action.

---

🔬 Before / After DataFrame Comparison

drishtipy includes DataComparator for comparing DataFrames before and after transformations.

Useful for:

• ETL pipelines
• Data cleaning
• Data transformation
• Feature engineering
• Data validation
• Preprocessing workflows

from drishtipy import DataComparator

comparator = DataComparator()

comparison = comparator.compare_dataframe(
    before_df,
    after_df
)

Or through the pandas accessor:

df.profile.compare(df2)

Comparison levels

Supported levels:

All
Dataset
Schema
Quality
ML

("quality" and "ml" currently produce the same column-level report.)

Example:

comparator.compare_dataframe(
    before_df,
    after_df,
    level="quality"
)

The result is a long-format DataFrame with columns Section, Metric, Before, After, Change, and Status (not every row populates every column). Raises TypeError if either argument isn't a DataFrame, and ValueError for an unrecognized level.

Changes only

compare_dataframe can return dozens of rows on wide DataFrames. Keep only the rows where something actually changed:

comparator.compare_dataframe(
    before_df,
    after_df,
    changes_only=True
)

Compact summary

A one-row-per-column view instead of the long-format table:

comparator.summary(
    before_df,
    after_df
)

   Column     Status                          Summary
0     age    Changed  missing 1->0; outliers 1->0; mean 122.5->35.17
1    city  Unchanged                         No change
2 new_col      Added                     Column added
3 old_col    Removed                   Column removed

Specific columns

comparator.compare_dataframe(
    before_df,
    after_df,
    level="quality",
    columns=["age", "salary"]
)

A column missing from one side is still reported as added/removed. Raises ValueError if a name in columns isn't present in either DataFrame. Note: level="dataset" metrics (row/column counts, memory, etc.) always reflect the full before/after DataFrames, not the selection.

---

🌐 HTML Reports

Generate a standalone, dark-themed HTML report instead of raw DataFrames — handy for sharing a report without a notebook:

df.profile.html(
    "profile_report.html"
)

Comparison reports:

df.profile.compare_html(
    df2,
    path="comparison_report.html"
)

Explicit classes also support HTML output:

DataProfiler(df).to_html(
    path="profile_report.html"
)

DataComparator().to_html(
    before_df,
    after_df,
    path="comparison_report.html"
)

Both accept the same filtering arguments as their DataFrame-returning counterparts (section/column_type for DataProfiler, level/ changes_only/columns for DataComparator), plus title for a custom page heading. Omitting path returns the HTML as a string instead of writing a file.

---

🕸️ Relationship Discovery

For a 50-column dataset, there are 50×49/2 = 1,225 possible column pairs — nobody is manually checking all of them. relationships() scans every pair automatically, picks a statistically appropriate method based on each column's semantic type (not just its pandas dtype), filters out pairs that can't be meaningfully compared, and ranks what's left:

result = df.profile.relationships()
result          # DRISHTIPY — RELATIONSHIP DISCOVERY dashboard, like info()

DRISHTIPY — RELATIONSHIP DISCOVERY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Columns analyzed:       15
Possible pairs:         105
Pairs analyzed:         91
Ignored pairs:          20

Meaningful relationships: 27
Strong:                    8
Moderate:                  7
Weak:                      12

TOP RELATIONSHIPS

  Region             ↔ City               1.00 🔥
  Category           ↔ Product            1.00 🔥
  Sales_Amount       ↔ Cost               1.00 🔥
  Sales_Amount       ↔ Profit             0.98 🔥
  Cost               ↔ Profit             0.96 🔥

result.top(20) / .matrix() / .graph() / .insights() / .dependencies()

What gets compared, and how

Each pair is routed to a method based on semantic type (drishtipy.semantic.detect_semantic_type — id, numeric, categorical, boolean, date/datetime, email, phone, currency, percentage, text, constant), not raw pandas dtype:

Pair type	Method	Notes
numeric ↔ numeric	Pearsonand Spearman	The stronger of the two wins — a monotonic non-linear relationship that Pearson underrates still gets caught
categorical ↔ numeric	ANOVA effect size (η²)	Also reports a p-value from an F-test when scipy is installed
categorical ↔ categorical	Cramér's V + normalized Mutual Information	Built from a contingency table with pandas/numpy only — no scipy required for the effect size itself
date ↔ numeric	Spearman trend	Direction reported as Increasing / Decreasing / Flat

Pairs that can't be meaningfully compared are filtered out before any statistics run: ID ↔ anything, constant columns, free text, and email/phone columns are excluded by default (RelationshipConfig(include_id_pairs=True) opts back in for advanced use).

Configuring thresholds

from drishtipy import RelationshipConfig

result = df.profile.relationships(
    RelationshipConfig(
        strong_threshold=0.80,
        moderate_threshold=0.50,
        min_sample_size=30,
        max_categories=50,     # categorical columns above this are skipped
        max_pairs=5_000,       # safety cap for very wide DataFrames
        sample_size=100_000,   # rows; None disables sampling
    )
)

If a pair doesn't have enough valid (non-null, overlapping) observations, it's reported as "Not enough valid observations" rather than a misleading score.

Views on the result

result.top(20)               # ranked DataFrame of the strongest relationships
result.matrix()               # symmetric strength matrix (method varies per cell — see below)
result.graph(top=20)          # RelationshipGraph: nodes + weighted edges
result.graph(threshold=0.70)  # only edges at/above a strength cutoff
result.insights()             # auto-generated, human-readable observations
result.dependencies()         # potential functional dependencies (farmer_id -> farmer_name)
result.redundant_columns()    # column groups that look like duplicates/derived copies
result.to_dataframe()         # the full, unfiltered results table
result.to_json()
result.to_html("relationships.html")

result.to_html() renders a standalone KPI-card dashboard by default — top relationships, an inline-SVG relationship graph, color-coded insight cards, and a dependencies list, all computed from the actual result (not a static mockup):

result.to_html("relationships.html")                    # dashboard style (default)
result.to_html("relationships.html", style="table")      # plain multi-section tables instead
result.to_html("relationships.html", top_n=10, graph_top=15)

The graph is real SVG with a computed circular layout (works for any number of nodes), not a fixed set of hand-placed positions — the most-connected column gets a highlighted "hub" style, matching how it's usually the one worth investigating first.

result.matrix() is not a plain correlation matrix — each cell's value comes from whichever method fit that pair's semantic types, so don't treat the whole matrix as linear-correlation strength.

result.graph() returns nodes/edges and a readable adjacency-list view — not a rendered image. Building an actual force-directed layout needs a graph/plotting stack (networkx, matplotlib) this library intentionally doesn't depend on; result.graph().edges is meant to be handed to one of those directly if you want a visual layout.

Functional dependencies and redundant columns

Beyond correlation, relationships() checks whether one column's values consistently determine another's (e.g. farmer_id -> farmer_name), and flags column groups that appear to store the same information twice (exact duplicates, or near-perfectly correlated numeric/categorical pairs) — language is deliberately hedged ("Potential functional dependency", consistency %) rather than claiming certainty the data may not fully support.

Multiple testing and significance

With hundreds of pairs analyzed at once, some correlations will look "significant" by chance alone. When scipy is installed (pip install drishtipy[relationships]), p-values are computed per pair and corrected for multiple testing via Benjamini-Hochberg FDR by default (RelationshipConfig(fdr_correction=False) to disable). Without scipy, effect sizes (Pearson/Spearman/η²/Cramér's V) are still computed — Significance just reports "Unknown (scipy not installed)" instead of guessing.

Integration

df.profile.relationships()          # via the accessor
df.profile.info().relationships()   # chained from the full profile report
DataProfiler(df).relationships()    # via the explicit class API

🤖 AutoML Regression

Train and benchmark multiple regression models with one call — no scikit-learn import required on your end. scikit-learn, numpy, and pandas are all installed automatically with pip install drishtipy.

result = df.ml.auto_predict(target="Sales")
result       # DrishtiPy AutoML Regression summary, printed like df.profile.info()

DrishtiPy AutoML Regression
──────────────────────────────────────────────

Target       : Sales
Models Tested: 6
CV Folds     : 5

Best Model   : Gradient Boosting

R²           : 0.9200
Adjusted R²  : 0.9100
RMSE         : 610.21
MAE          : 430.18
MAPE         : 5.10%
CV R²        : 0.9100
CV Std       : 0.0180
Overfit Gap  : 0.0200
Score        : 94.70/100

result.leaderboard() / .best_model() / .predict(new_data) / .report()

Not the same thing as df.profile.ml(). df.profile.ml() is the read-only ML-readiness report (encoding suggestions, no training). df.ml.auto_predict() actually trains and benchmarks models. Similar names, different jobs.

What it does automatically

• Detects numeric vs. categorical columns, imputes missing values (median for numeric, most-frequent for categorical), scales numeric features (matters a lot for Ridge/Lasso), and one-hot encodes categoricals — all inside a single scikit-learn Pipeline, fit only on the training split, so there's no leakage into the test set.
• Unseen categories at prediction time map to all-zero indicator columns instead of raising.
• Benchmarks 6 models by default — Linear Regression, Ridge, Lasso, Random Forest, Extra Trees, Gradient Boosting — plus XGBoost/ LightGBM/CatBoost automatically if they're installed (pip install drishtipy[advanced-ml]; skipped with a one-line notice otherwise, never an error).
• Computes 11 regression metrics per model (R², Adjusted R², RMSE, MAE, MSE, MAPE, sMAPE, Median AE, Max Error, Explained Variance, RMSLE — all with safe handling for zero targets, negative targets, and too-few-samples), train/test overfitting gap, and 5-fold cross-validation (mean + std of R²/RMSE/MAE).
• Ranks models by a weighted Composite Score (0-100) — R² 30%, RMSE 20%, MAE 20%, CV R² 15%, CV stability 5%, overfitting 5%, training speed 5% — not by R² alone, and not thrown off by models tying on a metric or a metric being unavailable for one model.

Configuring a run

result = df.ml.auto_predict(
    target="Sales",
    test_size=0.20,
    cv=5,
    random_state=42,
    include_optional=True,   # use XGBoost/LightGBM/CatBoost if installed
    n_jobs=-1,
)

Views on the result

result.summary()               # dict of headline numbers
result.leaderboard()           # every model tested, ranked by Composite Score
result.best_model()            # the fitted sklearn Pipeline (preprocessing + model)
result.predict(new_data)       # same preprocessing applied automatically
result.prediction_table()      # Actual / Predicted / Residual / Absolute Error (test split)
result.feature_importance()    # Feature / Importance, one-hot names mapped back, sorted
result.residual_analysis()     # mean/median/std/skew, outlier count, % within ±5/10/20%
result.overfitting_check()     # Train R² / Test R² / Gap / Status (Low/Moderate/High)
result.report("automl.html")   # standalone HTML dashboard

result.feature_importance() uses feature_importances_ for tree-based models and absolute coefficient magnitude for linear ones, with one-hot encoded categories mapped back to readable names (e.g. cat__city_Mumbai rather than an opaque column index).

🔄 ETL Pipeline

A lazy, composable, pandas-native ETL system — pandas + numpy only, no heavy dependencies. Every method below only registers an operation; nothing runs until .run(), .to_dataframe(), or .load().

import drishtipy as dp

pipeline = (
    dp.ETL()
    .extract("customers.xlsx")
    .text("name").strip().title()
    .numeric("age").to_numeric()
    .condition("age", "<", 0).replace(value=None)
)
# nothing has executed yet — no file read, no output written

result = pipeline.run()      # now it actually runs
df = result.data

Reusable, composable pipelines

Build small pipelines once, reuse them anywhere:

missing_pipeline = (
    dp.ETL()
    .fill_missing("age", strategy="median")
    .fill_missing("city", value="Unknown")
    .drop_missing(["email"])
)

text_pipeline = (
    dp.ETL()
    .text("name").strip().title()
    .text("email").strip().lower()
)

numeric_pipeline = (
    dp.ETL()
    .numeric("age").to_numeric()
    .condition("age", "<", 0).replace(value=None)
    .condition("age", ">", 120).replace(strategy="median")
)

date_pipeline = dp.ETL().date("signup_date").to_datetime()

Combine them with .include() (mutates the pipeline it's called on — built for assembling one master pipeline step by step):

master_pipeline = (
    dp.ETL()
    .extract("customers.xlsx")
    .include(text_pipeline)
    .include(numeric_pipeline)
    .include(missing_pipeline)
    .include(date_pipeline)
    .validate()
    .quality_report()
)

...or with + (returns a new pipeline, leaving both operands untouched — safe to combine the same reusable pipeline into several different masters):

pipeline_a = base_pipeline + numeric_pipeline
pipeline_b = base_pipeline + missing_pipeline
# base_pipeline, pipeline_a, and pipeline_b are all independent

Operator precedence note: dp.ETL().extract(x) + a + b.validate() calls .validate() on b alone, not on the combined pipeline — Python binds the method call to the nearest preceding expression. Wrap the + chain in parentheses before calling further methods: (dp.ETL().extract(x) + a + b).validate().

Text / numeric / date column API

.text("name").strip().lower().title()      # strip, lower, upper, title, capitalize
.numeric("age").to_numeric().round(2).abs()
.date("signup_date").to_datetime().year().month().day().day_name()

Date-part extraction (.year(), .month(), etc.) adds new columns (signup_date_year, ...) rather than overwriting the original.

Conditional replace

.condition("age", "<", 0).replace(value=None)
.condition("age", ">", 120).replace(strategy="median")

Operators: ==, !=, <, <=, >, >=, in, not_in, between, is_null, not_null. Strategies: median, mean, mode, min, max, zero. replace(value=None) explicitly nulls out matching values — it is not treated as "no value passed" (an internal sentinel distinguishes the two).

Missing values, duplicates, outliers

.fill_missing("age", strategy="median")     # median/mean/mode/min/max/zero/ffill/bfill
.fill_missing("city", value="Unknown")
.drop_missing(["email"])

.drop_duplicates(["customer_id"])           # or drop_duplicates() for all columns

.outliers("total_spent", method="iqr", action="flag")
# methods: iqr, zscore, modified_zscore, percentile
# actions: flag (adds total_spent_outlier, doesn't touch values), remove, replace, clip

Validation and quality scoring

.validate(required_columns=["customer_id"], ranges={"age": (0, 120)})
.quality_report()

quality_report() doesn't reimplement scoring — it calls the same DataProfiler.quality_score() used by df.profile.quality_score(), once on the data right after extract() and once on the final result, so "before ETL" vs. "after ETL" numbers are directly comparable to anything else in drishtipy.

validate() never turns a soft issue into a fatal error on its own — missing values, duplicates, and invalid coercions default to WARNING; only explicitly-configured required_columns/unique_columns checks can produce FAIL.

Execution and output

df = pipeline.to_dataframe()                     # run() and just take .data

result = pipeline.load("clean.csv")               # run + write output
result = pipeline.load("clean.xlsx", report="etl_report.html")  # + HTML report

.load() never touches the source file — every .run()/.load() call re-extracts from scratch, so running the same pipeline twice (or composing it into two different masters) always starts from the original data.

Inspecting a pipeline without running it

pipeline.describe()

DRISHTIPY ETL PIPELINE
----------------------

1. Extract customers.xlsx
2. Text: name → strip → title
3. Numeric: age → to_numeric
4. Condition: age < 0 → None

Status: CONFIGURED
Operations: 4

The result object

result.data                  # final pandas DataFrame
result.status                # PASS / WARNING / FAIL / COMPLETED
result.statistics             # dict: input/output records, missing fixed, duplicates, ...
result.warnings                # list of WARNING-level validation messages
result.execution_time
result.audit_log()            # DataFrame: one row per executed operation
result.validation_report()    # DataFrame: Check / Status / Details
result.execution_log()        # DataFrame: Timestamp / Operation / Duration / Status
result.to_html("report.html")
result.to_csv("out.csv")
result.to_excel("out.xlsx")

📁 Large CSV Profiling

Profile large CSV files without loading the entire source into memory:

from drishtipy import DataProfiler

profiler = DataProfiler.from_csv(
    "huge_dataset.csv",
    sample_size=100_000,
    chunksize=50_000,
    random_state=42
)

Then:

profiler.info_dataframe()

Check whether sampling was used:

profiler.is_sampled

Get the exact source row count:

profiler.total_rows_in_source

How sampling works

CSV file
   │
   ├── Pass 1
   │     └── Count total rows
   │
   ├── Pass 2
   │     └── Select a random sample
   │
   └── Profile the sample

The file is streamed in chunks twice — once to count exact total rows, once to keep each row with probability sample_size / total_rows (a random, roughly-uniform sample). If the file has fewer rows than sample_size, it's loaded in full and is_sampled stays False.

When sampling is used, the source row count (total_rows_in_source) is exact, while statistics/ML sections computed from the sample are estimates of the complete dataset. to_html() on a sampled profiler automatically notes the sample size vs. total rows in the report.

Additional keyword arguments can be forwarded to pandas.read_csv():

DataProfiler.from_csv(
    "data.csv",
    usecols=["age", "salary"],
    dtype={"age": "float64"},
    sep=",",
    parse_dates=["date"]
)

---

🧱 Explicit Class API

The pandas accessor is recommended, but explicit classes are also available — useful if you want to keep the profiler object around for repeated calls, or prefer not to rely on accessor registration:

import pandas as pd

from drishtipy import DataProfiler

df = pd.DataFrame({
    "age": [25, 32, None, 47],
    "city": ["Delhi", "Mumbai", "Delhi", "Pune"]
})

profiler = DataProfiler(df)

profiler.info_dataframe(
    section="schema"
)

profiler.quality_score()
profiler.alerts()
profiler.pii()
profiler.correlations()

Raises TypeError if df isn't a pandas.DataFrame.

---

⚙️ info_dataframe()

The underlying profiler API provides:

profiler.info_dataframe(
    section="All",
    column_type="All"
)

section

Supported values (case-insensitive):

All
Schema
Statistics
Quality
ML
ETL

"All" returns a dict mapping section name -> DataFrame:

{
    "Schema": ...,
    "Statistics": ...,
    "Quality": ...,
    "ML": ...,
    "ETL": ...
}

Any other value returns just that section's DataFrame.

column_type

Supported values (case-insensitive):

All
Numeric
Categorical

Example:

profiler.info_dataframe(
    section="statistics",
    column_type="numeric"
)

Raises ValueError if section or column_type isn't recognized, or if column_type filters out every column.

---

🧠 Design Philosophy

pandas-first

drishtipy extends pandas rather than replacing it.

Lightweight

The core package intentionally keeps dependencies minimal — pandas is the only required dependency.

Non-destructive

Profiling operations analyze the DataFrame without modifying it. The one exception is explicit: df.profile.pii(mask=True) returns a masked copy, never mutating the original.

Composable

Reports remain compatible with normal pandas operations — sort, filter, export, or display them however you like.

Human-readable

HTML reports and summaries are designed to make data problems easy to understand at a glance.

---

🧪 Development

Install development dependencies:

pip install -e ".[dev]"

Run tests:

pytest

Recommended workflow:

git clone <repository>
cd drishtipy

pip install -e ".[dev]"

pytest

---

🗺️ Roadmap

Potential future capabilities include:

• Advanced validation rules
• Automatic cleaning suggestions
• Smart semantic data-type/data-dictionary detection
• Time-series profiling
• Advanced ML-readiness checks
• Data drift detection
• Statistical distribution analysis
• Excel / JSON / Parquet / SQL database profiling
• Expanded interactive HTML dashboards
• Additional data-quality checks

---

📦 Package Information

Property	Value
Package	drishtipy
Import	drishtipy
Current Version	0.6.1
Python	>=3.8
License	MIT
Primary Dependency	pandas (>=1.3)

Install:

pip install drishtipy

Import:

import drishtipy

Use:

df.profile.quality()

---

📄 License

drishtipy is released under the MIT License.

⚠️ Privacy Notice: DrishtiPy processes data locally on the user's device and does not transmit user data to external servers. Users are responsible for ensuring that their use of personal, sensitive, or confidential data complies with applicable laws, regulations, and organizational policies.