piedomains: Classify website content using ML Models or LLMs

What's New in v0.10.0

• A taxonomy that asks answerable questions. Classes describing how a site is built and monetised (adv, tracker, spyware, redirector) are gone — a page does not state them, and they caused a third of all errors. recreation and hobby are split into their subcategories; porn/sex/models merge into adult.
• Accuracy 0.627 → 0.725 on the evaluation set across the last two releases, macro-F1 0.602 → 0.705.
• Confidence is a real probability: temperature-scaled softmax, rather than 39 per-class isotonic regressions applied elementwise and never renormalized.
• TensorFlow is gone, so Python 3.14 works.
• Bot walls are recovered from archive.org rather than classified as if the challenge page were the site.
• Failures are named. Every row carries a stable error_code; the run report aggregates by reason, stage and source.

Partly multilingual. The training corpus is overwhelmingly English, so non-English pages classify at 0.667 accuracy against 0.738 for English. Usable but not equal, and closing the gap needs multilingual training data rather than a multilingual encoder alone.

Breaking: the label set is now 47 classes, not 39, and some names changed (porn→adult, recreation→recreation/sports). See the changelog.

Breaking: image classification is unavailable while the screenshot model is retrained, so classify() is text-only and classify_by_images() raises. This changes no labels — the old "combined" path returned the text label every time and only averaged the confidences, so the image model could not affect an answer. See the changelog.

Installation

pip install piedomains

Requires Python 3.11+ (3.14 supported).

Basic Usage

from piedomains import DomainClassifier, DataCollector

classifier = DomainClassifier()
run = classifier.classify(["cnn.com", "amazon.com", "wikipedia.org"])

for result in run["results"]:
    print(f"{result['domain']}: {result['category']} ({result['confidence']:.3f})")

# Output:
# cnn.com: news (0.876)
# amazon.com: shopping (0.923)
# wikipedia.org: education (0.891)

Knowing What Failed

Every call returns both the per-domain rows and a run report, so a long URL list never fails silently. Each row carries a machine-readable status, the stage it reached, and a stable error_code:

run = classifier.classify(open("domains.txt").read().split())

print(run["report"])
# {'run_id': '8fe4cc80eeb5', 'total': 500, 'classified': 461, 'failed': 39,
#  'by_reason': {'dns_error': 12, 'timeout': 9, 'cannot_classify': 11, 'thin_content': 7},
#  'by_stage': {'fetch': 32, 'infer': 7},
#  'by_source': {'live': 448, 'archive': 13},
#  'missing': ['foo.com', 'bar.org', ...],
#  'started_at': ..., 'finished_at': ..., 'elapsed_ms': 184203}

# Retry only what is worth retrying
retry = [r["domain"] for r in run["results"] if r.get("retryable")]

error_code is a closed, stable set — safe to group on: invalid_domain, dns_error, connection_error, timeout, http_error, robots_blocked, content_type_rejected, content_too_large, no_archive_snapshot, archive_rate_limited, empty_text, missing_input_path, missing_screenshot, model_load_error, model_error, llm_error, bot_blocked, thin_content, cannot_classify, unknown.

cannot_classify is the umbrella terminal state: branch on it when you do not want to enumerate every cause.

Bot Walls

Roughly one domain in seven serves an anti-bot interstitial rather than a page. Changing the user-agent does not help — DataDome and Cloudflare fingerprint headless Chromium itself — so piedomains detects the interstitial and refetches the page from archive.org, which already has it. No evasion, and no challenge page classified as though it were the site.

run = classifier.classify(["etsy.com", "reuters.com", "indeed.com"])
for r in run["results"]:
    print(r["domain"], r["category"], r["source"], r["snapshot_timestamp"])
# etsy.com     shopping  archive  20260727020309
# reuters.com  news      archive  20260718201522
# indeed.com   jobsearch archive  20260724170521

A capture older than archive_max_age_days (default 365) is refused rather than passed off as the live page; those domains report cannot_classify. Set archive_fallback=False to turn this off and have blocked domains report bot_blocked instead.

From the command line:

classify_domains --file domains.txt --output json --report run-report.json
# stdout: {"results": [...], "report": {...}}
# stderr: 461/500 classified, 39 failed (run 8fe4cc80eeb5)
#           dns_error: 12
#           timeout: 9
# exit status is non-zero if any domain failed

Structured logs

Human-readable text is the default. For pipelines, opt into JSON lines — every record carries the run_id, plus domain/stage/error_code where relevant, so logs join against the report:

PIEDOMAINS_LOG_FORMAT=json classify_domains --file domains.txt
# {"ts":"...","level":"ERROR","run_id":"8fe4cc80eeb5","domain":"foo.com",
#  "stage":"fetch","error_code":"timeout","msg":"navigation timed out"}

Classification Methods

# Text-only. `classify` and `classify_by_text` are the same thing in this
# version -- see the note on image classification above.
run = classifier.classify(["github.com"])
run = classifier.classify_by_text(["news.google.com"])

# Batch processing with separated workflow
collector = DataCollector()
collection = collector.collect_batch(domains, batch_size=50)
results = classifier.classify_from_collection(collection, method="text")

Historical Analysis

# Analyze archived versions from archive.org
old_run = classifier.classify(["facebook.com"], archive_date="20100101")

# Batch processing with archive.org (respects rate limits)
domains = ["google.com", "wikipedia.org", "cnn.com"]
collector = DataCollector(archive_date="20050101")
collection = collector.collect_batch(domains, batch_size=10)  # Archive.org uses conservative defaults
historical_results = classifier.classify_from_collection(collection, method="text")

How archive analysis works

Snapshot discovery and retrieval go through the wayback library (CDX + Memento):

• Only status-200 captures are used. An archived redirect or 404 is never classified as if it were content; the domain reports no_archive_snapshot instead.
• The capture actually used is reported as snapshot_timestamp — not the date you asked for. Requesting 20100101 for cnn.com yields 20100101041727.
• Text is fetched raw via Wayback's id_ playback mode: no injected Wayback JavaScript, no rewritten URLs, and no browser required — so the text path is fast.
• Screenshots render via if_, which hides the Wayback toolbar but keeps archived CSS and images, so the page looks as it did. (id_ would render an unstyled skeleton.)
• Rate limiting, retries and exponential backoff are handled by the wayback session.

The cache key includes the archive date, so a live fetch and snapshots from different years coexist rather than overwriting one another:

cache/html/cnn.com.html            # live
cache/html/cnn.com@20050101.html   # 2005 snapshot
cache/html/cnn.com@20150101.html   # 2015 snapshot

Configure via piedomains.config: archive_max_parallel, archive_window_days, archive_search_rate, archive_memento_rate, archive_retries, archive_backoff.

LLM Classification

# Configure LLM provider
classifier.configure_llm(
    provider="openai",
    model="gpt-4o",
    api_key="sk-...",
    categories=["news", "shopping", "social", "tech"]
)

# LLM-powered classification
result = classifier.classify_by_llm(["example.com"])

# With custom instructions
result = classifier.classify_by_llm(
    ["site.com"],
    custom_instructions="Classify by educational value"
)

Set API keys via environment variables:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="..."

Categories

47 categories: news, finance, shopping, education, government, adult, gambling, social networks, search engines and others. Derived from Shallalist, with classes that describe hosting rather than content removed and the grab-bag categories split — see training/taxonomy.py.

Security & Docker

v0.5.0 includes production-ready Docker containerization for secure domain analysis:

# Build secure sandbox container
docker build -t piedomains-sandbox .

# Run with security constraints (2GB RAM, 2 CPU, read-only filesystem)
docker run --rm --memory=2g --cpus=2 --read-only \
  --tmpfs /tmp --tmpfs /var/tmp \
  piedomains-sandbox python -c "
from piedomains import DomainClassifier
classifier = DomainClassifier()
run = classifier.classify(['example.com'])
print(run['results'][0]['category'])
"

Batch Processing in Container:

# Use the included secure classification script
cd examples/sandbox
echo -e "wikipedia.org\ngithub.com\ncnn.com" > domains.txt
python3 secure_classify.py --file domains.txt

For testing, use known-safe domains: ["wikipedia.org", "github.com", "cnn.com"]

Documentation

• API Reference
• Examples
• Security Guide

Development

git clone https://github.com/themains/piedomains
cd piedomains
uv sync --all-groups
uv run pytest tests/ -v

License

MIT License

Citation

@software{piedomains,
  title={piedomains: AI-powered domain content classification},
  author={Chintalapati, Rajashekar and Sood, Gaurav},
  year={2024},
  url={https://github.com/themains/piedomains}
}