linktrace

LinkTrace is a document-oriented crawler. Every crawled page becomes a rich Document object containing metadata, content, and discovered relationships.

Perfect for: Site structure analysis, link tracking, concurrent page fetching, HTML document transformation.

Not: A Scrapy replacement. Scrapy is a powerful full-featured framework — linktrace is deliberately lightweight with no pipelines, middleware, or project scaffolding to configure. If you want crawling results in minutes rather than hours of setup, and a gentler learning curve, linktrace is for you.

Key Features

• ⚡ Async/await native — Built on asyncio + aiohttp for concurrent requests
• 🔗 Automatic link classification — Distinguishes internal vs external links by domain
• 🎯 URL filtering rules — Include/exclude by regex, path prefix, extension, query param, or domain
• 🎚️ Configurable concurrency — Tune max_concurrency and connection-pool limits for throughput
• 🗺️ Sitemap discovery — Seed from sitemap.xml / robots.txt Sitemap:, including nested indexes
• 📄 Rich document model — Full HTML source, parsed links, metadata, headers
• 🔄 Persistent sessions — Connection pooling for 10-100x faster same-domain crawls
• 🔁 Retries + backoff — Exponential backoff for transient errors (timeouts, 5xx)
• ⏱️ Rate limiting — Per-domain rate limiting with asyncio.Lock, no thundering herd
• 🤖 robots.txt support — Automatically respect Crawl-delay directives and Disallow rules per domain
• 🔍 Broken link tracking — Audit 404s and 5xx errors for site structure validation
• 💾 Optional caching — Disk-based cache (1-day TTL) for repeat crawls
• 🔐 SSL verification — Secure by default, with corporate proxy support
• 🍪 Automatic cookies — Set-Cookie extraction and sending built-in
• 🔀 Traversal strategies — BFS (broad) or DFS (deep) crawling
• 📊 Multi-format export — JSON, Pandas, Polars, PyArrow for data analysis
• 📍 Callbacks & streaming — Process results as crawled without memory buildup

Quick Start

import asyncio
from linktrace import Spider

async def main():
    spider = Spider(start_url="https://example.com", max_depth=2)
    documents = await spider.run_async()
    
    for doc in documents:
        print(f"{doc.url}")
        print(f"  Internal links: {len(doc.internal_links)}")
        print(f"  External links: {len(doc.external_links)}")

asyncio.run(main())

Installation

pip install linktrace

Optional export formats:

pip install linktrace[serializers]  # pandas + polars + pyarrow
pip install linktrace[pandas]       # Just pandas

Core Concepts

Spider

High-level orchestrator that crawls multiple pages using BFS (breadth-first) or DFS (depth-first) traversal.

Crawler

Low-level engine that fetches and parses individual documents. Handles retries, caching, SSL, cookies, sessions.

Document

Rich object containing:

• url — page URL
• title — HTML title tag
• source — raw HTML
• internal_links — links to same domain
• external_links — links to other domains
• status_code, response_headers, domain — metadata

CrawlRules

Declarative include/exclude policy for which discovered URLs the Spider follows (regex, path prefix, extension, query param, domain). Passed as Spider(rules=...).

SitemapParser

Fetches and parses XML sitemaps (urlsets and nested indexes). Used when Spider(use_sitemaps=True), or directly for sitemap inspection.

See Core Concepts for more.

Configuration

Basic Crawl

spider = Spider(
    start_url="https://example.com",
    max_depth=3,              # How deep to follow links
    traversal_strategy="bfs"  # "bfs" (default) or "dfs"
)
documents = await spider.run_async()

Retries & Timeouts

spider = Spider(
    start_url="https://example.com",
    request_timeout=15,       # Seconds per request (default: 30)
    max_retries=5,            # Retry transient errors (default: 3)
)

Caching

spider = Spider(
    start_url="https://example.com",
    cache_dir=".linktrace_cache"  # Enable disk caching (default: None/disabled)
)
# 2nd run will be 10-50x faster for same URLs

URL Filtering Rules

By default the spider follows every internal link until max_depth is reached. Pass a CrawlRules object to keep the crawl focused and skip crawl traps (login pages, sort/calendar links, binaries, off-site domains):

from linktrace import Spider, CrawlRules

rules = CrawlRules(
    allowed_domains=["realty.example.com"],      # stay on-site (subdomain-aware)
    include_path_prefixes=["/homes-for-sale/"],  # only follow listing pages...
    exclude_path_prefixes=["/login", "/privacy"],# ...never these
    blocked_extensions=["pdf", "jpg", "png"],    # documents, not downloads
    exclude_query_params=["sort", "page"],        # avoid faceted-search explosion
    exclude_patterns=[r"/print/?$"],              # regex escape hatch
)

spider = Spider("https://realty.example.com/", max_depth=4, rules=rules)

A populated allow list (allowed_domains, include_path_prefixes, allowed_extensions, include_patterns) is a whitelist; a block/exclude list rejects matches. Exclusions always win, and an empty CrawlRules() (the default) allows everything. rules.allows(url) is pure and unit-testable.

Concurrency

spider = Spider(
    start_url="https://example.com",
    max_concurrency=50,            # URLs fetched per batch (default: 10)
    max_connections=200,           # total aiohttp pool size (default: 100)
    max_connections_per_host=8,    # connections to one host (default: 10)
)

Raise max_concurrency / max_connections to go faster across many hosts; keep max_connections_per_host modest (and pair with request_delay or respect_robots_txt=True) to stay polite to any single server.

Sitemap Discovery

Seed the queue from a site's published document inventory before link-following begins. linktrace reads Sitemap: declarations from robots.txt (falling back to /sitemap.xml), follows sitemap indexes and nested sitemaps, and filters every discovered URL through your CrawlRules:

spider = Spider(
    start_url="https://realty.example.com/",
    max_depth=2,
    use_sitemaps=True,   # discover URLs from sitemap.xml / robots.txt Sitemap:
    rules=rules,
)
documents = await spider.run_async()

Sitemap discovery is best-effort — if none exists, ordinary link-following still runs. You can also inspect sitemaps directly via SitemapParser or Crawler.discover_sitemap_urls(base_url).

SSL & Corporate Proxies

# Default: verify SSL with system CA
spider = Spider(start_url="https://example.com")

# Corporate proxy with custom CA bundle
spider = Spider(
    start_url="https://example.com",
    ssl_verify="/path/to/corporate-ca.pem"
)

# Self-signed certs (testing only)
spider = Spider(
    start_url="https://example.com",
    ssl_verify=False  # ⚠️ Insecure
)

Cookies are handled automatically — no configuration needed.

Callbacks: Process Results in Real-Time

For large crawls, avoid memory buildup by processing documents as they're crawled:

# Stream results to disk
async def save_result(doc):
    with open("results.jsonl", "a") as f:
        f.write(json.dumps({"url": doc.url, "title": doc.title}) + "\n")

spider = Spider(
    start_url="https://example.com",
    on_page_crawled=save_result,
    accumulate_results=False,  # Don't keep in memory
)
await spider.run_async()  # Returns [], file has results

Callback Hooks:

• on_page_crawled(doc) — Called after each successful crawl. Return value accumulated if accumulate_results=True
• on_error(url, exc) — Called on crawl failures
• on_crawl_complete() — Called when crawl finishes (cleanup hook)

Async Callbacks Supported:

async def save_to_db(doc):
    await db.insert(doc.url, doc.title)
    return doc.url

spider = Spider(
    start_url="https://example.com",
    on_page_crawled=save_to_db,       # Async callback
    accumulate_results=True,
)
results = await spider.run_async()  # Returns list of URLs

Return Logic:

• No callback → returns all documents (default)
• Callback + accumulate_results=False → returns [] (streaming mode)
• Callback + accumulate_results=True → returns callback results

Traversal Strategies

BFS (Breadth-First) — Default

# Explores level by level: all depth-1 links, then depth-2, etc.
spider = Spider(start_url="https://example.com", max_depth=3, traversal_strategy="bfs")

DFS (Depth-First)

# Follows single paths all the way down before exploring siblings
spider = Spider(start_url="https://example.com", max_depth=5, traversal_strategy="dfs")

Use DFS for deep hierarchies (documentation sites, nested directories). Use BFS for broad exploration.

Rate Limiting & robots.txt

By default, linktrace automatically respects robots.txt Crawl-delay directives and Disallow rules, enforcing per-domain rate limiting:

# Automatic robots.txt respect (default)
spider = Spider(
    start_url="https://example.com",
    user_agent="MyBot/1.0",  # Identifies your bot to robots.txt rules
)
await spider.run_async()

Customize rate limiting:

# Enforce explicit delay (ignores robots.txt)
spider = Spider(
    start_url="https://example.com",
    request_delay=1.0,           # 1 second between requests to same domain
    respect_robots_txt=False,    # Don't fetch robots.txt
)

# Concurrent requests to different domains, serialized to same domain
await spider.run_async()

Track Crawl Status

Monitor which pages returned error status codes:

spider = Spider(start_url="https://example.com", max_depth=2)
documents = await spider.run_async()

# Find pages with error responses
error_pages = [doc for doc in documents if doc.status_code >= 400]
for doc in error_pages:
    print(f"Error: {doc.url} (HTTP {doc.status_code})")

# Monitor disallowed pages (403 from robots.txt)
disallowed = [doc for doc in documents if doc.status_code == 403]
print(f"Disallowed by robots.txt: {len(disallowed)} pages")

Stream crawl status in real-time:

async def track_errors(doc):
    if doc.status_code >= 400:
        print(f"❌ {doc.url} (HTTP {doc.status_code})")

spider = Spider(
    start_url="https://example.com",
    on_page_crawled=track_errors,
    accumulate_results=False,
)
await spider.run_async()

Export Data

from linktrace import Spider, Serializers

spider = Spider(start_url="https://example.com", max_depth=2)
documents = await spider.run_async()

# Export to JSON
serializer = Serializers(documents)
serializer.to_json("crawl.json", include_html=False)

# Export to Pandas (one row per link)
df = serializer.to_pandas()
print(df[["url", "title", "link_url", "link_type"]])

# Export to Polars (faster for large datasets)
df_polars = serializer.to_polars()

# Export to PyArrow (for data pipelines)
table = serializer.to_arrow()

Link Analysis

from collections import Counter

spider = Spider(start_url="https://example.com", max_depth=2)
documents = await spider.run_async()

# Count external domains
external_domains = Counter()
for doc in documents:
    for link in doc.external_links:
        domain = link.url.split("/")[2]
        external_domains[domain] += 1

print(external_domains.most_common(10))

See Examples for more patterns.

Notebooks

Interactive examples in notebooks/:

• crawl_cnn.ipynb — Crawls CNN.com, analyzes link structure, demonstrates all export formats
• crawl_tax_assessor.ipynb — Crawls a municipal property/GIS site, with callbacks and aggregation
• config_rules_sitemaps.ipynb — URL filtering rules, configurable concurrency, and sitemap discovery

API Reference

See API Reference for complete method documentation.

Troubleshooting

"SSL: CERTIFICATE_VERIFY_FAILED"

Use ssl_verify=False for self-signed certs (testing only), or ssl_verify="/path/to/ca.pem" for corporate proxies.

"Too many connections"

Lower max_concurrency and/or max_connections_per_host to reduce parallel requests, and consider adding a request_delay. Defaults are conservative.

"Crawler hits timeout on deep sites"

Try DFS traversal instead of BFS, or increase request_timeout.

See Troubleshooting for more.

Performance

Typical performance (single-domain crawl):

• First run: ~50-500ms per page (network-bound)
• Cached run: ~1-10ms per page (2-50x faster)
• Memory: ~1MB per 100 pages

With persistent sessions + connection pooling, same-domain requests are 10-100x faster than per-request session setup.

Architecture

Spider (orchestrator)
  └─ Crawler (persistent session)
      ├─ aiohttp (HTTP requests + connection pooling)
      ├─ lxml (HTML parsing)
      ├─ ResponseCache (optional disk caching)
      └─ CookieJar (automatic cookie handling)

Spider manages the crawl queue and traversal. Crawler handles individual document fetching/parsing. All requests share one persistent aiohttp session per Spider instance, so connection pooling, cookies, SSL configuration, and DNS caching are reused across the crawl.

Why linktrace?

Scrapy is an excellent full crawling and extraction framework. linktrace is designed for a narrower job: fast async link analysis with minimal setup.

Instead of building a Scrapy project around spiders, requests, responses, callbacks, items, pipelines, middleware, and settings, linktrace gives you a direct document-centric API. Each crawled URL becomes a Document object containing the page source, title, status code, response headers, domain, internal links, external links, and crawl status metadata.

That makes linktrace useful when your goal is to inspect site structure, trace links, audit crawl status, or export crawl results to dataframe-oriented tools without creating a larger scraping project.

linktrace also reuses a persistent aiohttp session during a crawl. Connection pooling, cookie reuse, SSL configuration, request timeouts, per-host limits, and DNS caching are carried across requests, which can make repeated same-domain crawls much faster than creating a fresh client/session per URL.

Use Scrapy when: you need a mature scraping framework with item pipelines, middleware, schedulers, broad ecosystem support, and complex extraction workflows.

Use linktrace when: you want a focused async crawler that turns URLs into analyzable Document objects with automatic link classification and simple exports.

vs requests + BeautifulSoup: Built-in async concurrency, automatic session reuse, retries, caching, rate limiting, and structured document objects. Better for crawling multiple pages.

vs Selenium: Pure HTTP crawler (no JS execution). Faster, lighter, but can't handle dynamic sites.

Testing

just test          # Run all tests
just test-cov      # Run with coverage report

All 122 tests pass. 100% of core crawling paths tested (rate limiting, broken link tracking, robots.txt, callbacks, URL filtering rules, sitemap discovery).

Contributing

Bug reports and pull requests welcome on GitHub.

License

MIT

Documentation:

• Getting Started
• Core Concepts
• API Reference
• Examples
• Troubleshooting