datascope

PyPI: pip install datascope-dq

Data created upstream — by manufacturing teams entering UPCs, inventory staff assigning product codes, offshore developers choosing column types — silently breaks systems downstream. A product code with letters where EDI expects numbers. Fifteen "N/A" strings buried in 500 numeric rows that pandas silently drops, skewing every calculation by 3%.

datascope finds these problems, explains what's wrong in plain English, and tells you what to fix. It reads each cell's actual type (not what pandas infers), detects hidden quality issues, classifies their severity by downstream impact, and generates a professional diagnostic report.

What It Finds

Each finding is expressed as assumption vs. reality: what the data appears to be vs. what it actually contains. Every finding includes a downstream impact explanation, a fix recommendation, and a prevention rule.

Installation

pip install datascope-dq

For Parquet file support:

pip install datascope-dq[parquet]

Or install from source:

git clone https://github.com/MsShawnP/datascope.git
cd datascope
pip install -e .

Usage

# Analyze an Excel file
datascope data.xlsx

# Analyze a CSV
datascope sales_export.csv

# Analyze a Parquet file (requires pyarrow)
datascope data.parquet

# Specify a sheet and output directory
datascope data.xlsx --sheet Revenue --output-dir ./client_reports

Output Formats

# PDF report (default)
datascope data.xlsx

# Structured JSON for pipeline integration
datascope data.xlsx --format json

# Self-contained HTML report
datascope data.xlsx --format html

# Annotated Excel with highlighted problem cells
datascope data.xlsx --format annotated-excel

# PDF + JSON together
datascope data.xlsx --format both

CLI Flags

# Quiet mode — exit code only (0 = no critical, 1 = critical findings)
datascope data.xlsx --quiet

# Verbose mode — full tracebacks on analyzer failures
datascope data.xlsx --verbose

# Limit row count (default: warn at 500K cells, abort at 5M)
datascope huge_file.csv --max-rows 100000

Example Output

datascope: Analyzing sample_mixed_types.xlsx...
  200 rows x 4 columns

Found 2 findings:
  2 Critical  ########

Top critical findings:
  * revenue_mixed: However, 15 str values were found among 200 non-null values (the majority type covers 92.5%). Examples of unexpected values: 'N/A', 'N/A', 'N/A', and 2 more.
  * revenue_mixed: However, 7.5% of values (1 distinct sentinel string) are placeholder text rather than real data: 'N/A' (15 times).

Report saved: reports/sample_mixed_types_diagnostic.pdf

The Report

Reports are structured for non-technical readers — no jargon, no composite scores, no unexplained metrics.

Executive Summary — overall health assessment, finding counts by severity, top critical issues highlighted.

Findings by Severity — each finding presented as a card:

• Assumption: what the data appears to be
• Reality: what it actually contains
• Impact: what breaks downstream
• Recommended Fix: what to do now
• Prevention Rule: what right looks like going forward

Field Inventory — summary table of all columns with their detected issue types and severity.

Findings are color-coded (red/amber/blue) and grouped by severity so readers know what to fix first.

How It Works

Most tools let pandas (or the SQL driver, or Excel) decide column types. A column with 485 numbers and 15 strings becomes float64 — the strings become NaN, the type problem disappears, and every downstream calculation is quietly wrong.

datascope reads each cell's actual Python type via openpyxl (for Excel) or raw-string inference (for CSV). This cell-level type detection is always on — there's no flag to enable it because skipping it defeats the purpose.

The analysis pipeline:

1. Load — read with cell-level type preservation (no silent coercion)
2. Detect — seven analyzers scan for type inconsistencies, sentinels, missing values, format issues, and cardinality anomalies
3. Classify — severity assigned by downstream impact (critical = silent data loss, warning = likely misinterpretation, info = worth noting)
4. Compose — plain-English narrative generated for each finding
5. Report — output as PDF, HTML, JSON, or annotated Excel

Severity Model

Project Structure

datascope/
├── loaders/            # Excel, CSV, and Parquet with cell-level type tracking
│   ├── excel.py        # openpyxl-based, preserves per-cell Python types
│   ├── csv_loader.py   # Raw string inference with regex-accelerated datetime detection
│   ├── parquet.py      # Arrow schema → Python type mapping (optional pyarrow)
│   └── base.py         # Extension-based dispatch
├── analyzers/          # Seven detectors, each returns list[Finding]
│   ├── type_consistency.py
│   ├── sentinel.py
│   ├── format_check.py # Leading zeros + mixed dates
│   ├── cardinality.py  # Near-constant + duplicate IDs
│   └── missing_values.py
├── findings/           # Severity classifier + NL template engine
│   ├── severity.py     # Impact-based classification rules
│   ├── templates.py    # Plain-English templates per finding type
│   ├── composer.py     # Template dispatch
│   └── pipeline.py     # classify → compose → sort
├── reports/
│   ├── pdf.py          # Professional PDF with reportlab
│   ├── html.py         # Self-contained HTML with inline CSS
│   └── annotated_excel.py  # Highlighted cells + findings sheet
└── cli.py              # argparse CLI, pipeline orchestration

Requirements

• Python 3.10+
• pandas >= 2.0
• openpyxl >= 3.1
• reportlab >= 4.0
• pyarrow >= 12.0 (optional, for Parquet support)

License

MIT

Built by Lailara LLC — data hygiene and analytics consulting for specialty food brands scaling into national retail.