🧩 Dataset Quality Scoring Engine — System Framework (Markdown)

Local Development Split

The repository now separates responsibilities:

• dsqus/engine: core Python package code (no FastAPI).
• backend: FastAPI service that imports and calls the core package.
• ui: browser frontend that uploads files to the backend.

Run Upload Flow End-to-End

From repository root:

python -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r backend/requirements.txt
uvicorn backend.app:app --reload

From ui in another terminal:

npm install
npm run dev

Then open the Vite URL (typically http://localhost:5173) and upload a CSV/XLS/XLSX file.

#️⃣ 1. Overview

The Dataset Quality Scoring Engine (DQS) evaluates the quality of any dataset using automated, model-agnostic metrics. The system processes user-uploaded datasets, computes embeddings, analyzes statistical and semantic properties, and outputs a standardized quality score (0–100) along with detailed submetrics.

2. High-Level Workflow

User Upload → Preprocessing → Embedding → Metric Computation → Scoring → Report Generation → Cleanup

3. Input Specifications

The system accepts:

jsonl json txt csv folder of text/code files PDFs (extracted into text)

4. Preprocessing Pipeline

Validate file format Convert to normalized internal format (list[str or dict]) Clean text: remove control chars normalize whitespace optional: strip HTML/markup Segment long documents into meaningful chunks Remove empty or invalid samples

Output: clean, structured dataset

5. Embedding Generation

Two embedding flows:

5.1 Local Embeddings (Per Upload)

Used for:

redundancy coherence diversity factual contradictions clustering/domain analysis

These embeddings exist only for the request and are deleted afterward.

5.2 Global Reference Embeddings (Static)

Used only for novelty detection.

Pre-built FAISS/Vector DB containing ~1M representative samples:

Wikipedia Common Crawl samples C4 slices StackOverflow Books corpus Public domain corpora

This is static, never modified by user uploads.

6. Metric Computation

DQS computes 10 core quality metrics:

6.1 Redundancy Score compute embedding similarity within dataset clustering density = redundancy score = inverse redundancy 6.2 Malware / Toxicity Score run samples through pre-trained toxicity classifier aggregate severity 6.3 Diversity Score linguistic diversity (entropy, vocab richness) semantic diversity (embedding variance) 6.4 Readability Score Flesch–Kincaid sentence complexity coherency heuristics 6.5 Semantic Coherence embedding flow consistency perplexity using a small reference LLM 6.6 Novelty Score compare against global reference corpus nearest neighbor distance = novelty measure 6.7 Structure Quality

Applicable to:

JSON code SQL XML YAML

Checks:

syntax validity AST parsing success 6.8 Factual Conflict Score sample random pairs pass to NLI contradiction model aggregate contradictions 6.9 Domain Balance Score cluster dataset embeddings measure cluster distribution via entropy 6.10 Length Distribution Score detect outliers analyze token distribution

7. Composite Score Calculation

All metrics normalized 0–100.

Weighted aggregation formula:

overall_score = 0.15redundancy + 0.10toxicity + 0.10diversity + 0.10readability + 0.10coherence + 0.10novelty + 0.10structure + 0.10factual_conflict + 0.075domain_balance + 0.075length_distribution

8. Report Generation

Output includes:

8.1 JSON Report

Contains:

overall_score all sub-scores dataset metadata top detected issues summary of duplicates domain distribution histogram 8.2 Human-Readable Text Report simple explanations listed issues recommendations optional PDF

9. System Architecture

Components API Layer file upload async processing report delivery Compute Engine embeddings scoring logic batching concurrency optimized Reference Store FAISS/Qdrant global novelty index static Models Folder toxicity classifier contradiction/NLI model small LLM for perplexity

10. Execution Flow Diagram

[Upload] ↓ [Preprocess] ↓ [Generate Local Embeddings] ↓ [Compute All Self-Contained Metrics] ↓ [Compare with Global Reference Embeddings] ↓ [Aggregate Scores] ↓ [Generate JSON + Text Report] ↓ [Return to User] ↓ [Delete all temp embeddings + data]

11. Privacy Model

No dataset stored after processing No embeddings stored Only the report is saved (optional) Global reference embeddings NEVER contain user data Fully GDPR-safe

12. MVP Boundary (Important)

Not included in v1:

dataset cleaning dataset repair dataset marketplace collaborative annotation data augmentation agentic workflows

You stay laser-focused on: analysis → scoring → reporting.