gibsondedup 0.1.1

A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.

🧹 gibsondedup

High‑Performance Search Result Deduplication Engine

---

📌 The Problem

When aggregating search results from multiple providers (Google, Bing, DuckDuckGo), 40–60% of results are duplicates — same content, different URLs, different titles, different tracking parameters.

Source 1 (Google):
  url: https://www.python.org/docs?utm_source=google
  title: "Python Official Documentation"

Source 2 (Bing):
  url: http://python.org/docs/
  title: "Python Docs"

Source 3 (DuckDuckGo):
  url: https://python.org/docs?utm_medium=cpc
  title: "Learn Python - Official Docs"

These are the same resource. Without deduplication, users see noise instead of signal.

---

✨ The Solution

gibsondedup processes raw search results through a multi‑stage pipeline:

flowchart TD
    A[Raw Results] --> B[Parse & Validate]
    B --> C[URL Normalization]
    C --> D[Exact URL Grouping]
    D --> E[Semantic Merge]
    E --> F[Canonicalization]
    F --> G[Clean Output + Metadata]

Result: 100 noisy inputs → 60–70 clean, canonical results.

---

🚀 Installation

pip install gibsondedup

---

🏃 Quick Start

from gibsondedup import DeduplicationEngine

engine = DeduplicationEngine()

results = engine.process([
    {
        "title": "Python Official Documentation",
        "url": "https://www.python.org/docs?utm_source=google",
        "description": "Official Python docs",
        "source": "google"
    },
    {
        "title": "Python Docs",
        "url": "http://python.org/docs/",
        "description": "Python documentation",
        "source": "bing"
    },
    {
        "title": "Stack Overflow Python",
        "url": "https://stackoverflow.com/questions/tagged/python",
        "description": "Python questions",
        "source": "google"
    },
])

print(results)

Output:

{
  "results": [
    {
      "title": "Python Official Documentation",
      "url": "https://www.python.org/docs?utm_source=google",
      "canonical_url": "python.org/docs",
      "description": "Official Python docs",
      "sources": ["google", "bing"],
      "duplicates_removed": 1
    },
    {
      "title": "Stack Overflow Python",
      "url": "https://stackoverflow.com/questions/tagged/python",
      "canonical_url": "stackoverflow.com/questions/tagged/python",
      "description": "Python questions",
      "sources": ["google"],
      "duplicates_removed": 0
    }
  ],
  "meta": {
    "total_input": 3,
    "total_parsed": 3,
    "total_output": 2,
    "duplicates_removed": 1,
    "processing_time_ms": 0.09,
    "similarity_threshold": 0.8
  }
}

---

⚙️ Configuration

Custom Similarity Threshold

Control how aggressively similar‑titled results are merged:

# Conservative (default) — only merge highly similar titles
engine = DeduplicationEngine(similarity_threshold=0.8)

# Moderate — merge titles with moderate overlap
engine = DeduplicationEngine(similarity_threshold=0.5)

# Aggressive — merge titles with minimal overlap
engine = DeduplicationEngine(similarity_threshold=0.3)

Input Contract

Field	Type	Required	Description
title	string	✅	Result title
url	string	✅	Result URL
description	string	❌	Result description
source	string	❌	Source provider (google, bing, etc.)

Note: Malformed records (missing title or URL) are skipped gracefully — the pipeline continues processing valid records.

---

🧱 Architecture

Pipeline Stages

Stage	Description
1. Parser	Converts raw JSON payloads into structured internal contracts. Validates required fields. Skips malformed records without crashing.
2. URL Normalizer	Converts URLs to canonical form by removing protocol, www, tracking params (utm_*, fbclid, gclid), trailing slashes, default ports, and lowercasing everything.
3. Exact Grouping	Groups results by normalized canonical URL using a hash map. O(n) complexity – avoids O(n²) pairwise comparison.
4. Semantic Merger	Within groups sharing the same domain, compares titles using Jaccard similarity. Groups exceeding threshold are merged. Only compares within the same exact domain.
5. Canonicalizer	Selects the single best result per group using a combined title+description length heuristic. Collects source traceability from all merged results.

Data Contracts

# Input
RawSearchResult(title, url, description?, source?)

# Internal
NormalizedSearchResult(title, url, canonical_url, description, source)

# Output
CanonicalResult(title, url, canonical_url, description, sources[], duplicates_removed)

System Invariants

1. Same normalized URL → same duplicate group (always)
2. Canonical results preserve source traceability (always)
3. Pipeline stages do not mutate upstream data (always)
4. Normalization happens once per result (always)

---

🧠 Engineering Decisions

Question	Answer
Why hash‑based grouping (O(n)) over pairwise (O(n²))?	With 1000 results, O(n²) → 1M comparisons. O(n) → 1000. Milliseconds vs seconds.
Why exact domain matching for semantic merging?	python.org/docs and docs.python.org serve different content. Exact domain prevents false merges.
Why Jaccard similarity over Levenshtein distance?	Jaccard operates on word tokens – robust to word order changes & additions. Levenshtein is sensitive to trivial differences like "Docs" vs "Documentation".
Why keep normalization deterministic?	Same input → same output. Enables reproducible runs, safe caching, reliable tests.

---

📊 Performance

Input Size	Processing Time	Memory
100 results	~1ms	~1MB
1,000 results	~10ms	~5MB
10,000 results	~100ms	~50MB

Benchmarked on Ubuntu 22.04, Python 3.10, Intel i5

---

🧪 Testing

# Install dev dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=app --cov-report=term-missing

Test coverage: 59 tests across 5 modules

Module	Tests	Focus
test_normalizer.py	14	URL normalization
test_grouping.py	4	Exact grouping
test_canonicalizer.py	7	Best result selection
test_merger.py	11	Semantic merging
test_similarity.py	13	Jaccard similarity
test_pipeline.py	10	End‑to‑end pipeline

---

🗺️ Roadmap

Phase	Status	Features
Phase 1	✅ Complete	URL normalization, exact duplicate grouping, canonicalization with source traceability
Phase 2	✅ Complete	Jaccard similarity engine, domain‑based semantic merging, configurable threshold
Phase 3	🚧 Planned	Persistent caching (Redis), database storage (PostgreSQL), REST API, Rails wrapper
Phase 4	🔮 Planned	Semantic embeddings (vector similarity), distributed processing, production observability

---

📄 License

MIT License. See LICENSE file.

---

👤 Author

Aseda Gibson
Computer Engineering, University of Energy and Natural Resources (UENR), Ghana.
Backend systems, distributed architecture, infrastructure tooling.

---

_{Built with ❤️ and correctness‑oriented engineering.
Every architectural decision is documented. Every invariant is tested.}

```