fetchkit

A YAML-configured data-fetching library for agentic applications.

fetchkit collects posts and comments from sources (Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT, SEC EDGAR, Polymarket, Manifold) into a single canonical Post model, with de-duplication, deterministic sorting, and a shared HTTP client with retries and rate limiting. It is designed as the data-collection layer for LLM/agent pipelines — feed it configs, get back clean typed data.

• YAML-first configuration with strict validation: unknown or misspelled keys are rejected up front, so a bad config fails loudly instead of silently.
• Builtin fetchers: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT (global news firehose), SEC EDGAR (securities filings), and Polymarket/Manifold (prediction markets) — all zero-auth.
• Relative time windows: say window: "last 6 hours" instead of computing timestamps.
• Open metadata: each Post carries a metadata dict for source-specific detail (arXiv categories/DOI, GitHub language/stars, tags) without bloating the core model.
• Robust HTTP: shared session pooling, exponential backoff + retries, Retry-After handling, and optional per-host rate limiting.
• Typed end-to-end: Pydantic v2 models throughout.
• Deterministic output: dedup by (source, id), sorted descending by (created_at, id).
• Agent-friendly CLI: fetchkit run config.yaml emits clean JSON on stdout — no Python required.
• RSS feed discovery: fetchkit discover "<use case>" maps a natural-language query onto real RSS feeds an agent can fetch — closing the "which feed URL?" gap.
• Per-source discovery: fetchkit suggest <source> lists the selectable knobs for any source (tags, sites, arXiv categories, trending hashtags, Bluesky feeds) — all zero-auth — so an agent can fill a config without guessing.

Install

pip install fetchkit-agents     # PyPI distribution name

# or install the latest from source:
pip install "git+https://github.com/metemorris/fetchkit.git"

The PyPI package is fetchkit-agents (the fetchkit name was taken), but you still import fetchkit and the CLI command is fetchkit. Requires Python ≥ 3.10.

Quick start

1. YAML config

config.yaml:

window: "last 24 hours"   # or set explicit start_time / end_time
fetchers:
  - type: hackernews
    posts:
      max_items: 50
      order: new
  - type: arxiv
    categories: ["cs.AI", "cs.LG"]
    max_items: 40
  - type: github
    resource: releases
    repos: ["python/cpython", "pydantic/pydantic"]
  - type: lobsters
    listing: hottest
  - type: stackexchange
    site: stackoverflow
    tagged: ["python", "asyncio"]
  - type: bluesky
    resource: search
    query: "large language models"
  - type: mastodon
    instance: mastodon.social
    tag: ai
  - type: edgar
    companies: ["AAPL", "MSFT"]
    forms: ["8-K", "10-Q"]
  - type: reddit
    subreddits: ["news", "wallstreetbets"]
    listing: new
  - type: googlenews
    resource: search
    query: "nvidia earnings when:6h"
  - type: gdelt
    query: '"interest rates"'
    languages: ["english"]
  - type: polymarket
    query: "fed"
  - type: rss
    feeds:
      - url: "https://feeds.bbci.co.uk/news/rss.xml"
        name: "BBC News"
    max_items_per_feed: 40
    max_total_items: 200
http:           # optional
  timeout: 15
  max_retries: 5
  rate_limit_per_host: 2.0

2. Run

from fetchkit import load_config, collect_all

config = load_config("config.yaml")
result = collect_all(config)

print(f"Collected {len(result.posts)} posts")
for post in result.posts:
    print(post.created_at, post.source, post.title, post.url)

if result.has_errors:
    for source, err in result.errors:
        print(f"  {source}: {err}")

3. Command line (for agents & scripts)

fetchkit installs a CLI so an agent can shell out and parse JSON without writing Python. run prints only JSON to stdout (diagnostics go to stderr), so it pipes cleanly into jq:

fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "sources", "errors"}
fetchkit run config.yaml -o out.json        # write JSON to a file (stdout stays empty)
fetchkit run config.yaml --window "6h"      # override the time window at runtime
fetchkit run config.yaml --compact          # single-line JSON
fetchkit run config.yaml --fail-on-error    # exit 1 if any source failed
echo "$CONFIG_YAML" | fetchkit run -        # read the config from stdin
fetchkit validate config.yaml               # validate a config without fetching (also accepts -)
fetchkit schema                             # JSON Schema for every config/output model
python -m fetchkit run config.yaml          # module form, identical behavior

fetchkit run --help and fetchkit validate --help print the YAML config format inline (top-level keys, a minimal example, and the fetcher type list), so simple configs need no schema round-trip.

fetchkit schema lets an agent discover what it can fetch — it prints the JSON Schema (with field descriptions) for the top-level config, the shared HTTP settings, every builtin fetcher's typed config, and the canonical Post output — so a config can be written without knowing the YAML format out of band. Pass a section name to print just that part instead of the full ~50 KB document:

fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","edgar","gdelt","github","googlenews","hackernews","lobsters","manifold","mastodon","polymarket","reddit","rss","stackexchange"]
fetchkit schema config                      # just the top-level config schema
fetchkit schema rss                         # just one fetcher's config schema
fetchkit schema post                        # just the output Post model
fetchkit schema -o schema.json              # write to a file; supports --compact too

Output shape — sources reports per-fetcher success/failure (one entry per enabled fetcher, in config order), so partial failures are visible without diffing domains against posts. A source can partially fail, e.g. one dead feed among many: it keeps its posts, records the error, and reports ok: false.

{
  "count": 2,
  "posts": [ { "id": "...", "source": "rss", "title": "...", "url": "...", "created_at": "..." } ],
  "sources": [
    { "source": "rss", "name": null, "ok": false, "posts": 2,
      "errors": ["Failed to fetch feed https://dead.example/feed.xml: ..."] }
  ],
  "errors": [ { "source": "rss", "error": "Failed to fetch feed https://dead.example/feed.xml: ..." } ]
}

Exit codes: 0 success · 1 a source failed (only with --fail-on-error) · 2 configuration error.

4. Programmatic (no YAML)

from datetime import datetime, timezone, timedelta
from fetchkit import FetchKitConfig, HackerNewsFetchConfig, PostFetchConfig, SortOrder, collect_all

config = FetchKitConfig(
    start_time=datetime.now(timezone.utc) - timedelta(days=1),
    end_time=datetime.now(timezone.utc),
    fetchers=[
        HackerNewsFetchConfig(posts=PostFetchConfig(max_items=30, order=SortOrder.NEW)),
    ],
)
result = collect_all(config)

Discovering RSS feeds

RSS is the hardest source to discover for: a feed is an arbitrary URL, so an agent that wants "central-bank policy" or "rust programming" has no way to know which feeds exist. The optional discovery module closes that gap — give it a use case, get back ranked feeds you can drop straight into an rss fetcher. (The other sources have their own, simpler discovery surface — see Discoverability for every source below.)

from fetchkit.discovery import discover, to_rss_config

matches = discover("news and topics regarding AI safety research", top_k=5)
for m in matches:
    print(m.score, m.name, m.url)

config = to_rss_config(matches)   # an RSSFetchConfig, ready for collect_all

fetchkit discover "AI safety research news" --top-k 5            # ranked feeds as JSON
fetchkit discover "central bank policy" --as-config -o rss.json  # emit a runnable config…
fetchkit run rss.json                                            # …then fetch (discover → config → run)
fetchkit find-feeds https://example.com                          # autodiscover a site's feeds

How it works (and what it can't do)

Feeds are matched, not searched. You can't embed feeds you've never seen, so embeddings only ever rank a set of candidates you already hold. Discovery assembles that candidate set from three sources, then ranks it against your query:

1. Curated catalog — a shipped, versioned directory of ~50 high-quality, long-lived feeds across news, research, programming, finance, and more. The offline, deterministic floor.
2. Autodiscovery (find_feeds(url) / --from-urls) — given a site, it reads the <link rel="alternate"> RSS-autodiscovery tags, probes common feed paths, and validates each candidate. This reaches the open-web long tail. The topic → which sites step is left to your web search: an agent already has one, so it finds the sites and hands them to fetchkit, which extracts the feeds. fetchkit never bundles a search engine.
3. External index (--external, opt-in) — queries a third-party feed-search service for recall over millions of feeds. Off by default (network + ToS).

Ranking has two backends behind one interface:

• lexical (default) — pure-Python BM25 over each feed's metadata. Zero extra dependencies, fully deterministic, works offline and in CI.

• embedding — a local sentence-transformers model for stronger semantic matching. Opt in with the extra:

  pip install "fetchkit-agents[discovery-embeddings]"
  fetchkit discover "papers on diffusion models" --backend embedding
  

  --backend auto (the default) uses the embedding ranker when the extra is installed and falls back to lexical otherwise.

Discovery matches a feed's description/metadata (what the feed is about), not the live article stream — for the latter, fetch the feed with the rss fetcher and match individual posts. Catalog quality is the main lever on result quality; see src/fetchkit/discovery/data/CATALOG.md to extend it.

Because discovery only ranks candidates it already holds, a niche query (a specific region, product, or community) can come back filled with generic feeds that matched only the query's broad terms. The CLI flags this instead of letting it pass: the JSON payload carries unmatched_terms (query terms no returned feed mentions) plus a hint, and the same hint is printed to stderr. When you see it, feed discovery better candidates — --from-urls with sites from your own web search, fetchkit find-feeds <site-url>, or --external.

Discoverability for every source

discover solves RSS's "which feed URL?" problem. The other sources have a different discoverability gap — which tag / site / instance / feed do I put in the config? — answered by fetchkit suggest <source> (and run_suggester() in Python). Each suggester is no-auth and returns JSON-ready rows:

fetchkit suggest lobsters                                  # all Lobsters tags
fetchkit suggest stackexchange --site stackoverflow        # popular SO tags
fetchkit suggest stackexchange --what sites                # available SE sites
fetchkit suggest arxiv --query vision                      # matching arXiv categories
fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to watch
fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
fetchkit suggest bluesky                                   # popular custom feeds
fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
fetchkit suggest edgar --query apple                       # tickers/CIKs for an edgar config
fetchkit suggest reddit --query finance                    # subreddits for a topic
fetchkit suggest gdelt --query country                     # GDELT languages/countries/operators
fetchkit suggest googlenews                                # Google News topics + query operators
fetchkit suggest polymarket --query fed                    # most active prediction markets
fetchkit suggest manifold --query "rate cut"               # Manifold markets for a topic
fetchkit suggest rss --query "AI safety news"              # delegates to discover()

Most suggesters call the source's live API (all no-auth); hackernews/arxiv are static, and rss reuses the offline catalog ranker. Network suggesters fail gracefully (the CLI reports the error on stderr and exits non-zero).

Configuration reference

Top-level (FetchKitConfig)

Set the time window with either a relative window or both start_time/end_time — the two are mutually exclusive. If you specify none of them, the window defaults to the last 24 hours. start_time <= end_time is enforced, and unknown top-level keys are rejected. Per-fetcher start_time/end_time default to None and inherit the run window from these top-level keys at runtime (there is no global: key — the top level is the global scope).

Relative windows

window accepts (case-insensitive): "last 6 hours", "past 30 minutes", "last 7 days", "last week", "last month", "today", "yesterday", or a bare duration like "6h", "2d", "90m". It resolves once to a concrete (start_time, end_time) pair (end = now), so collection stays deterministic for that resolved pair. The same parsing is available programmatically via fetchkit.resolve_window(spec) and fetchkit.parse_duration(text).

Hacker News (type: hackernews)

RSS / Atom (type: rss)

feeds[].url is restricted to HTTP(S) URLs by default, and each URL's host must resolve to a public address — feeds pointing at loopback, private (RFC-1918), link-local, or cloud-metadata (169.254.169.254) hosts are refused to prevent SSRF. Setting allow_local_files: true additionally permits local file paths and file:// URLs and opts out of the SSRF guard (it declares the config trusted) — see the security note below before enabling it.

⚠️ Security — untrusted configs. fetchkit is built to run YAML that may be produced by an LLM/agent. By default it blocks two abuse vectors in RSS feed URLs: local file reads (e.g. file:///etc/passwd) and SSRF to internal addresses (e.g. http://169.254.169.254/... or http://127.0.0.1). Both protections are on by default. allow_local_files: true turns both off, so enable it only for configs and fixtures you trust. The SSRF guard is reasonable, not perfect — it checks the host at request time and does not defend against DNS rebinding or redirects to private hosts.

arXiv (type: arxiv)

Uses the arXiv export API (Atom, parsed with feedparser). Authors, categories, DOI, PDF link, and primary category are preserved in post.metadata.

GitHub (type: github)

Public GitHub REST API (no auth; tighter rate limits without a token). Repo, language, stars, forks, and topics are preserved in post.metadata.

Lobsters (type: lobsters)

The lobste.rs public JSON endpoints (no auth). Tags are preserved in post.metadata.

Stack Exchange (type: stackexchange)

The Stack Exchange API (no auth; anonymous access is capped at 300 requests/day/IP). Questions become Posts; with comments.fetch: true, top answers are attached as Comments. Tags, answer count, and the accepted-answer id are kept in post.metadata.

Bluesky (type: bluesky)

The public Bluesky AppView (public.api.bsky.app), unauthenticated by design. uri, cid, and langs are kept in post.metadata; likes map to score and replies to comment_count.

Mastodon (type: mastodon)

Public and hashtag timelines on any instance (no auth, when the instance keeps public preview enabled — otherwise requests return 401). HTML content is reduced to plain text; tags, instance, and visibility are kept in post.metadata.

SEC EDGAR (type: edgar)

Securities filings from the SEC's free, keyless EDGAR APIs. resource: filings pulls recent filings per company from data.sec.gov/submissions (acceptance timestamps are near-real-time — new 8-Ks appear within minutes); resource: search runs EDGAR full-text search across all filers (date granularity only; the run window is passed to the API as a date range). Form type, CIK, accession number, and 8-K item codes are kept in post.metadata.

The SEC's fair-access policy asks clients to identify themselves with contact info and stay under 10 req/s — set user_agent and consider http.rate_limit_per_host.

Reddit (type: reddit)

Reddit's public .json endpoints with a descriptive User-Agent. listing: new surfaces posts seconds after submission (engagement is near zero at that point by definition); hot/top carry engagement at the cost of recency. Score, comment count, and upvote ratio are captured as of fetch time. Subreddit, flair, and upvote ratio are kept in post.metadata.

Terms note: unauthenticated access is tolerated by Reddit at low volume (~10 requests/minute per IP) but sits in a gray area of Reddit's Data API terms, which formally expect registered OAuth clients. fetchkit issues one request per fetch; keep polling rates low (http.rate_limit_per_host). Reddit also blocks many datacenter/cloud IPs outright (403 regardless of User-Agent), so this fetcher may work from a residential connection but not from CI or a cloud host.

GDELT (type: gdelt)

The GDELT Project's DOC 2.0 API — machine-reads worldwide news media in 65 languages, re-indexed every 15 minutes. A zero-auth global news firehose. Hits carry URL, title, timestamp, source domain/country/language (kept in post.metadata); there is no author, body text, or engagement, so those fields are None. The run window maps to the API's startdatetime/enddatetime.

Google News (type: googlenews)

Google News' public RSS endpoints: top stories, search, or a topic section. Minutes-fresh headlines aggregated from thousands of publishers. The query language supports when:2h (recency), site:, quoted phrases, -exclusions, and intitle:. Links are Google News redirect URLs (they resolve to the publisher); the publisher name maps to author.

Polymarket (type: polymarket) and Manifold (type: manifold)

Prediction markets as posts: the market question is the title, and live odds, volume, and close date land in post.metadata (odds/probability, volume_24h, …). Odds moving is often a signal about breaking news before articles are written. Both APIs are public read-only, no auth.

Snapshot semantics: markets are live state, not timestamped events, so these two fetchers deliberately ignore the run window — an active market created months ago is still current. created_at is the market's creation time, which also means snapshot posts sort below newer article-shaped posts in a mixed feed.

HTTP (http:)

Adding a fetcher

Fetchers live in the library itself, so each one ships with a typed config, validation, and tests. Add a new source via a PR or a local fork in four steps:

1. Add a typed config to src/fetchkit/schemas/fetcher.py (subclass FetcherBase, give it a Literal type), and register it in _BUILTIN_TYPES + FetcherConfig.

2. Write the fetcher module in src/fetchkit/fetchers/, returning a FetcherResult:

   from fetchkit.fetchers.base import FetcherResult
   from fetchkit.fetchers.registry import register_fetcher
   from fetchkit.schemas.post import Post
   
   @register_fetcher("mysource")
   def fetch(config) -> FetcherResult:
       posts = [Post(id="1", source="mysource", title="…", source_url="https://…")]
       return FetcherResult(posts=posts, errors=[])
   

3. Import the module in src/fetchkit/fetchers/__init__.py so it registers on import.

4. Add tests (mock HTTP with the responses library — see tests/fetchers/).

5. (Optional) register a discovery helper so agents can find your source's knobs:

   from fetchkit.fetchers.suggest_registry import register_suggester
   
   @register_suggester("mysource")
   def suggest(*, query=None, limit=20, **kwargs) -> list[dict]:
       return [{"tag": "…"}]   # JSON-ready rows; surfaced via `fetchkit suggest mysource`
   

Use Post.metadata for any source-specific fields that don't map to the canonical columns.

The Post model

class Post(BaseModel):
    id: str                       # unique within source
    source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
                                  #   | "stackexchange" | "bluesky" | "mastodon" | "edgar"
                                  #   | "reddit" | "gdelt" | "googlenews" | "polymarket" | "manifold"
    title: str | None
    text: str | None              # body / content
    url: str | None               # external link
    author: str | None
    score: int | None             # source-relative; NOT comparable across sources
    comment_count: int | None
    created_at: datetime | None   # UTC-aware
    source_url: str               # direct link on the source platform
    comments: list[Comment]       # nested threads (HN)
    metadata: dict[str, Any]      # source-specific extras (categories, stars, tags, …)

All datetimes are normalized to UTC. Posts are deduplicated by (source, id) and sorted descending by (created_at, id) for deterministic output.

score is source-relative. Each source defines it differently — Hacker News points, Lobsters score, GitHub stars (for search_repos), Stack Exchange question score, Bluesky likes, Mastodon favourites, Reddit score (fuzzed by Reddit, and near zero on brand-new posts), and None for arXiv, RSS, Google News, GDELT, EDGAR, and the prediction-market sources. The values are not comparable across sources, so don't rank a mixed feed by score directly. Compare within a single source, or use a source-aware ranking of your own. Output is ordered by recency (created_at), not by score.

Collector invariants

collect_all preserves three guarantees:

1. Window inheritance — per-fetcher start_time/end_time fall back to the global window when omitted.
2. Dedup — by (source, id); first occurrence wins.
3. Sort — descending by (created_at or UTC_MIN, id).

Partial failures are aggregated, not fatal: a failing source yields an entry in result.errors while other sources still collect. Check result.has_errors.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# tests (skip live/networked)
pytest -m "not live"

# live network smoke (opt-in)
pytest -m live

# strict typecheck
mypy src

License

MIT — see LICENSE. MIT is intentional: as a small, dependency-light utility meant to be embedded freely in agent pipelines (including commercial and closed-source ones), a permissive license maximizes adoption with no copyleft obligations. A weak-copyleft license (e.g. MPL-2.0) or strong copyleft (GPL/AGPL) would force redistribution terms on downstream users and discourage exactly the embedded use fetchkit is built for.