lurkers 0.1.0

Convenient API + CLI to fetch web content for agents

lurkers

Convenient API + CLI to fetch web content for agents.

Status: early PoC. APIs will change.

Sources

• HTML pages (any URL) — extracted to markdown via trafilatura
• YouTube videos — title + channel + transcript
• Twitter/X posts — via fxtwitter (no auth required)
• RSS/Atom feeds — list of entries, each fetched through the unified dispatch

Install

pip install lurkers

Python

import lurkers

# unified entry point — auto-detects source type
doc = lurkers.fetch("https://news.ycombinator.com/item?id=1")
doc = lurkers.fetch("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
doc = lurkers.fetch("https://x.com/elonmusk/status/12345")

# RSS / Atom: returns list[Document]
docs = lurkers.feed("https://news.ycombinator.com/rss", limit=10)

# async siblings
import asyncio
docs = asyncio.run(lurkers.afeed("https://example.com/rss.xml"))

Every fetch returns a Document:

class Document(BaseModel):
    source: str               # canonical URL
    source_type: str          # "html" | "youtube" | "twitter"
    title: str | None
    content: str              # markdown / plain text
    fetched_at: datetime
    metadata: dict[str, Any]  # source-specific (video_id, author_handle, ...)

CLI

lurkers fetch https://example.com/article            # JSON to stdout
lurkers fetch https://youtu.be/dQw4w9WgXcQ --pretty
lurkers feed https://news.ycombinator.com/rss -n 5

License

Apache-2.0.