gibsondedup 0.1.0

A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.

gibsondedup

A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.

---

The Problem

When aggregating search results from multiple providers (Google, Bing, DuckDuckGo), 40–60% of results are duplicates — same content, different URLs, different titles, different tracking parameters.

Source 1 (Google):
  url: https://www.python.org/docs?utm_source=google
  title: "Python Official Documentation"

Source 2 (Bing):
  url: http://python.org/docs/
  title: "Python Docs"

Source 3 (DuckDuckGo):
  url: https://python.org/docs?utm_medium=cpc
  title: "Learn Python - Official Docs"

These are the same resource. Without deduplication, users see noise instead of signal.

---

The Solution

gibsondedup processes raw search results through a multi-stage pipeline:

Raw Results
    ↓
Parse & Validate         (malformed records skipped gracefully)
    ↓
URL Normalization        (remove tracking params, www, trailing slashes)
    ↓
Exact URL Grouping       (hash-based, O(n) complexity)
    ↓
Semantic Merge           (Jaccard similarity, configurable threshold)
    ↓
Canonicalization         (select best result per group)
    ↓
Clean Output             (deduplicated results + metadata)

Result: 100 noisy inputs → 60–70 clean, canonical results.

---

Installation

pip install gibsondedup

---

Quick Start

from app.orchestration.pipeline import DeduplicationEngine

engine = DeduplicationEngine()

results = engine.process([
    {
        "title": "Python Official Documentation",
        "url": "https://www.python.org/docs?utm_source=google",
        "description": "Official Python docs",
        "source": "google"
    },
    {
        "title": "Python Docs",
        "url": "http://python.org/docs/",
        "description": "Python documentation",
        "source": "bing"
    },
    {
        "title": "Stack Overflow Python",
        "url": "https://stackoverflow.com/questions/tagged/python",
        "description": "Python questions",
        "source": "google"
    },
])

print(results)

Output:

{
  "results": [
    {
      "title": "Python Official Documentation",
      "url": "https://www.python.org/docs?utm_source=google",
      "canonical_url": "python.org/docs",
      "description": "Official Python docs",
      "sources": ["google", "bing"],
      "duplicates_removed": 1
    },
    {
      "title": "Stack Overflow Python",
      "url": "https://stackoverflow.com/questions/tagged/python",
      "canonical_url": "stackoverflow.com/questions/tagged/python",
      "description": "Python questions",
      "sources": ["google"],
      "duplicates_removed": 0
    }
  ],
  "meta": {
    "total_input": 3,
    "total_parsed": 3,
    "total_output": 2,
    "duplicates_removed": 1,
    "processing_time_ms": 0.09,
    "similarity_threshold": 0.8
  }
}

---

Configuration

Custom Similarity Threshold

Control how aggressively similar-titled results are merged:

# Conservative (default) — only merge highly similar titles
engine = DeduplicationEngine(similarity_threshold=0.8)

# Moderate — merge titles with moderate overlap
engine = DeduplicationEngine(similarity_threshold=0.5)

# Aggressive — merge titles with minimal overlap
engine = DeduplicationEngine(similarity_threshold=0.3)

Input Contract

Each result in the input array accepts:

Field	Type	Required	Description
title	string	✅	Result title
url	string	✅	Result URL
description	string	❌	Result description
source	string	❌	Source provider (google, bing, etc.)

Malformed records (missing title or URL) are skipped gracefully — the pipeline continues processing valid records.

---

Architecture

Pipeline Stages

Stage 1: Parser Converts raw JSON payloads into structured internal contracts. Validates required fields. Skips malformed records without crashing.

Stage 2: URL Normalizer Converts URLs to canonical form by:

• Removing protocol (http://, https://)
• Removing www prefix
• Removing tracking parameters (utm_*, fbclid, gclid)
• Removing trailing slashes
• Stripping default ports (80, 443)
• Preserving meaningful query parameters
• Lowercasing everything

Stage 3: Exact Grouping Groups results by normalized canonical URL using a hash map. Time complexity: O(n). Avoids the O(n²) pairwise comparison trap.

Stage 4: Semantic Merger Within groups sharing the same domain, compares representative titles using Jaccard similarity. Groups exceeding the similarity threshold are merged into one. Only compares within the same exact domain — python.org and docs.python.org are treated as separate domains.

Stage 5: Canonicalizer For each group, selects the single best result using a combined title + description length heuristic. Collects source traceability from all merged results.

Data Contracts

# Input
RawSearchResult(title, url, description?, source?)

# Internal
NormalizedSearchResult(title, url, canonical_url, description, source)

# Output
CanonicalResult(title, url, canonical_url, description, sources[], duplicates_removed)

System Invariants

1. Same normalized URL → same duplicate group (always)
2. Canonical results preserve source traceability (always)
3. Pipeline stages do not mutate upstream data (always)
4. Normalization happens once per result (always)

---

Engineering Decisions

Why hash-based grouping (O(n)) over pairwise comparison (O(n²))?

With 1000 results, O(n²) means 1,000,000 comparisons. O(n) means 1000. At scale this is the difference between milliseconds and seconds.

Why exact domain matching for semantic merging?

python.org/docs and docs.python.org serve genuinely different content despite sharing a base domain. Exact domain matching prevents false merges while still catching real duplicates like python.org/docs and python.org/reference.

Why Jaccard similarity over Levenshtein distance?

Jaccard operates on token sets (words), making it robust to word order changes and additions. Levenshtein operates on character sequences, making it sensitive to trivial differences like "Docs" vs "Documentation".

Why keep normalization deterministic?

Same input must always produce same output. This ensures:

• Results are reproducible across runs
• Caching normalized URLs is safe
• Tests are reliable and meaningful

---

Performance

Input Size	Processing Time	Memory
100 results	~1ms	~1MB
1,000 results	~10ms	~5MB
10,000 results	~100ms	~50MB

Benchmarked on Ubuntu 22.04, Python 3.10, Intel i5

---

Testing

# Install dev dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=app --cov-report=term-missing

Test coverage: 59 tests across 5 modules

tests/test_normalizer.py    14 tests  (URL normalization)
tests/test_grouping.py       4 tests  (exact grouping)
tests/test_canonicalizer.py  7 tests  (best result selection)
tests/test_merger.py        11 tests  (semantic merging)
tests/test_similarity.py    13 tests  (Jaccard similarity)
tests/test_pipeline.py      10 tests  (end-to-end pipeline)

---

Roadmap

Phase 1 (Complete)

• URL normalization
• Exact duplicate grouping
• Canonicalization with source traceability

Phase 2 (Complete)

• Jaccard similarity engine
• Domain-based semantic merging
• Configurable similarity threshold

Phase 3 (Planned)

• Persistent caching (Redis)
• Database storage (PostgreSQL)
• REST API exposure
• Rails wrapper (gem)

Phase 4 (Planned)

• Semantic embeddings (replace Jaccard with vector similarity)
• Distributed processing
• Production observability

---

License

MIT License. See LICENSE file.

---

Author

Aseda Gibson Computer Engineering, University of Energy and Natural Resources (UENR), Ghana. Backend systems, distributed architecture, infrastructure tooling.

GitHub: github.com/asedagibson

---

Built with correctness-oriented engineering. Every architectural decision is documented. Every invariant is tested.