fetchkit

A YAML-configured data-fetching library for agentic applications.

fetchkit collects posts and comments from sources (Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT, SEC EDGAR, Polymarket, Manifold) into a single canonical Post model, with de-duplication, deterministic sorting, and a shared HTTP client with retries and rate limiting. It is designed as the data-collection layer for LLM/agent pipelines — feed it configs, get back clean typed data.

• YAML-first configuration with strict validation: unknown or misspelled keys are rejected up front, so a bad config fails loudly instead of silently.
• Builtin fetchers: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT (global news firehose), SEC EDGAR (securities filings), and Polymarket/Manifold (prediction markets) — all zero-auth by default. For the sources that block anonymous datacenter/cloud traffic there are opt-in authenticated variants (reddit-auth, bluesky-auth) whose credentials come from environment variables — see Authenticated fetcher types and secrets.
• Relative time windows: say window: "last 6 hours" instead of computing timestamps.
• Open metadata: each Post carries a metadata dict for source-specific detail (arXiv categories/DOI, GitHub language/stars, tags) without bloating the core model.
• Robust HTTP: shared session pooling, exponential backoff + retries, Retry-After handling, and optional per-host rate limiting.
• Identify your project: set http.user_agent (or $FETCHKIT_USER_AGENT) once and every request carries your project's identity — see Identify your project (User-Agent).
• Typed end-to-end: Pydantic v2 models throughout.
• Deterministic output: dedup by (source, id) — or across sources by normalized article URL with dedup: url — sorted descending by (created_at, id).
• Agent-friendly CLI: fetchkit run config.yaml emits clean JSON on stdout — no Python required. fetchkit example <type> prints a commented, runnable config for any source, and fetchkit guide prints an agent guide covering field semantics and per-source gotchas.
• Uniform text surface: Post.text carries the best available body at every source (arXiv abstract, GitHub description/README, RSS content, HN self-post…) — null only when the source genuinely has no body. See The Post model.
• RSS feed discovery: fetchkit discover "<use case>" maps a natural-language query onto real RSS feeds an agent can fetch — closing the "which feed URL?" gap.
• Per-source discovery: fetchkit suggest <source> lists the selectable knobs for any source (tags, sites, arXiv categories, trending hashtags, Bluesky feeds) — all zero-auth — so an agent can fill a config without guessing.

Install

pip install fetchkit-agents     # PyPI distribution name

# or install the latest from source:
pip install "git+https://github.com/metemorris/fetchkit.git"

The PyPI package is fetchkit-agents (the fetchkit name was taken), but you still import fetchkit and the CLI command is fetchkit. Requires Python ≥ 3.10.

Quick start

1. YAML config

config.yaml:

window: "last 24 hours"   # or set explicit start_time / end_time
fetchers:
  - type: hackernews
    posts:
      max_items: 50
      order: new
  - type: arxiv
    categories: ["cs.AI", "cs.LG"]
    max_items: 40
  - type: github
    resource: releases
    repos: ["python/cpython", "pydantic/pydantic"]
  - type: lobsters
    listing: hottest
  - type: stackexchange
    site: stackoverflow
    tagged: ["python", "asyncio"]
  - type: bluesky
    resource: search
    query: "large language models"
  - type: mastodon
    instance: mastodon.social
    tag: ai
  - type: edgar
    companies: ["AAPL", "MSFT"]
    forms: ["8-K", "10-Q"]
  - type: reddit
    subreddits: ["news", "wallstreetbets"]
    listing: new
  - type: googlenews
    resource: search
    query: "nvidia earnings when:6h"
  - type: gdelt
    query: '"interest rates"'
    languages: ["english"]
  - type: polymarket
    query: "fed"
  - type: rss
    feeds:
      - url: "https://feeds.bbci.co.uk/news/rss.xml"
        name: "BBC News"
    max_items_per_feed: 40
    max_total_items: 200
http:           # optional
  user_agent: "my-news-pipeline/1.0 (contact@example.com)"   # identify YOUR project
  timeout: 15
  max_retries: 5
  rate_limit_per_host: 2.0

2. Run

from fetchkit import load_config, collect_all

config = load_config("config.yaml")
result = collect_all(config)

print(f"Collected {len(result.posts)} posts")
for post in result.posts:
    print(post.created_at, post.source, post.title, post.url)

if result.has_errors:
    for source, err in result.errors:
        print(f"  {source}: {err}")

3. Command line (for agents & scripts)

fetchkit installs a CLI so an agent can shell out and parse JSON without writing Python. run prints only JSON to stdout (diagnostics go to stderr), so it pipes cleanly into jq:

fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "sources", "errors"}
fetchkit run config.yaml -o out.json        # write JSON to a file (stdout stays empty)
fetchkit run config.yaml --window "6h"      # override the time window at runtime
fetchkit run config.yaml --compact          # single-line JSON
fetchkit run config.yaml --fail-on-error    # exit 1 if any source failed
echo "$CONFIG_YAML" | fetchkit run -        # read the config from stdin
fetchkit validate config.yaml               # validate a config without fetching (also accepts -)
fetchkit schema                             # JSON Schema for every config/output model
fetchkit example arxiv                      # commented, runnable YAML config for a source
fetchkit example arxiv | fetchkit run -     # …which pipes straight into run
fetchkit guide                              # agent guide: field semantics + per-source gotchas
python -m fetchkit run config.yaml          # module form, identical behavior

fetchkit run --help and fetchkit validate --help print the YAML config format inline (top-level keys, a minimal example, and the fetcher type list), so simple configs need no schema round-trip.

fetchkit schema lets an agent discover what it can fetch — it prints the JSON Schema (with field descriptions) for the top-level config, the shared HTTP settings, every builtin fetcher's typed config, and the canonical Post output — so a config can be written without knowing the YAML format out of band. Pass a section name to print just that part instead of the full ~50 KB document:

fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","bluesky-auth","edgar","gdelt","github","googlenews","hackernews","lobsters","manifold","mastodon","polymarket","reddit","reddit-auth","rss","stackexchange"]
fetchkit schema config                      # just the top-level config schema
fetchkit schema rss                         # just one fetcher's config schema
fetchkit schema post                        # just the output Post model
fetchkit schema -o schema.json              # write to a file; supports --compact too

Output shape — sources reports per-fetcher success/failure (one entry per enabled fetcher, in config order), so partial failures are visible without diffing domains against posts. A source can partially fail, e.g. one dead feed among many: it keeps its posts, records the error, and reports ok: false.

{
  "count": 2,
  "posts": [ { "id": "...", "source": "rss", "title": "...", "url": "...", "created_at": "..." } ],
  "sources": [
    { "source": "rss", "name": null, "ok": false, "posts": 2,
      "errors": ["Failed to fetch feed https://dead.example/feed.xml: ..."] }
  ],
  "errors": [ { "source": "rss", "error": "Failed to fetch feed https://dead.example/feed.xml: ..." } ]
}

Exit codes: 0 success · 1 a source failed (only with --fail-on-error) · 2 configuration error.

4. Programmatic (no YAML)

from datetime import datetime, timezone, timedelta
from fetchkit import FetchKitConfig, HackerNewsFetchConfig, PostFetchConfig, SortOrder, collect_all

config = FetchKitConfig(
    start_time=datetime.now(timezone.utc) - timedelta(days=1),
    end_time=datetime.now(timezone.utc),
    fetchers=[
        HackerNewsFetchConfig(posts=PostFetchConfig(max_items=30, order=SortOrder.NEW)),
    ],
)
result = collect_all(config)

Discovering RSS feeds

RSS is the hardest source to discover for: a feed is an arbitrary URL, so an agent that wants "central-bank policy" or "rust programming" has no way to know which feeds exist. The optional discovery module closes that gap — give it a use case, get back ranked feeds you can drop straight into an rss fetcher. (The other sources have their own, simpler discovery surface — see Discoverability for every source below.)

from fetchkit.discovery import discover, to_rss_config

matches = discover("news and topics regarding AI safety research", top_k=5)
for m in matches:
    print(m.score, m.name, m.url)

config = to_rss_config(matches)   # an RSSFetchConfig, ready for collect_all

fetchkit discover "AI safety research news" --top-k 5            # ranked feeds as JSON
fetchkit discover "central bank policy" --as-config -o rss.json  # emit a runnable config…
fetchkit run rss.json                                            # …then fetch (discover → config → run)
fetchkit find-feeds https://example.com                          # autodiscover a site's feeds

How it works (and what it can't do)

Feeds are matched, not searched. You can't embed feeds you've never seen, so embeddings only ever rank a set of candidates you already hold. Discovery assembles that candidate set from three sources, then ranks it against your query:

1. Curated catalog — a shipped, versioned directory of ~50 high-quality, long-lived feeds across news, research, programming, finance, and more. The offline, deterministic floor.
2. Autodiscovery (find_feeds(url) / --from-urls) — given a site, it reads the <link rel="alternate"> RSS-autodiscovery tags, probes common feed paths, and validates each candidate. This reaches the open-web long tail. The topic → which sites step is left to your web search: an agent already has one, so it finds the sites and hands them to fetchkit, which extracts the feeds. fetchkit never bundles a search engine.
3. External index (--external, opt-in) — queries a third-party feed-search service for recall over millions of feeds. Off by default (network + ToS).

Ranking has two backends behind one interface:

• lexical (default) — pure-Python BM25 over each feed's metadata. Zero extra dependencies, fully deterministic, works offline and in CI.

• embedding — a local sentence-transformers model for stronger semantic matching. Opt in with the extra:

  pip install "fetchkit-agents[discovery-embeddings]"
  fetchkit discover "papers on diffusion models" --backend embedding
  

  --backend auto (the default) uses the embedding ranker when the extra is installed and falls back to lexical otherwise.

Discovery matches a feed's description/metadata (what the feed is about), not the live article stream — for the latter, fetch the feed with the rss fetcher and match individual posts. Catalog quality is the main lever on result quality; see src/fetchkit/discovery/data/CATALOG.md to extend it.

Because discovery only ranks candidates it already holds, a niche query (a specific region, product, or community) can come back filled with generic feeds that matched only the query's broad terms. The CLI flags this instead of letting it pass: the JSON payload carries unmatched_terms (query terms no returned feed mentions) plus a hint, and the same hint is printed to stderr. When you see it, feed discovery better candidates — --from-urls with sites from your own web search, fetchkit find-feeds <site-url>, or --external.

Discoverability for every source

discover solves RSS's "which feed URL?" problem. The other sources have a different discoverability gap — which tag / site / instance / feed do I put in the config? — answered by fetchkit suggest <source> (and run_suggester() in Python). Each suggester is no-auth and returns JSON-ready rows:

fetchkit suggest lobsters                                  # all Lobsters tags
fetchkit suggest stackexchange --site stackoverflow        # popular SO tags
fetchkit suggest stackexchange --what sites                # available SE sites
fetchkit suggest arxiv --query vision                      # matching arXiv categories
fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to watch
fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
fetchkit suggest bluesky                                   # popular custom feeds
fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
fetchkit suggest edgar --query apple                       # tickers/CIKs for an edgar config
fetchkit suggest reddit --query finance                    # subreddits for a topic
fetchkit suggest gdelt --query country                     # GDELT languages/countries/operators
fetchkit suggest googlenews                                # Google News topics + query operators
fetchkit suggest polymarket --query fed                    # most active prediction markets
fetchkit suggest manifold --query "rate cut"               # Manifold markets for a topic
fetchkit suggest rss --query "AI safety news"              # delegates to discover()

Most suggesters call the source's live API (all no-auth); hackernews/arxiv are static, and rss reuses the offline catalog ranker. Network suggesters fail gracefully (the CLI reports the error on stderr and exits non-zero).

Configuration reference

Top-level (FetchKitConfig)

Set the time window with either a relative window or both start_time/end_time — the two are mutually exclusive. If you specify none of them, the window defaults to the last 24 hours. start_time <= end_time is enforced, and unknown top-level keys are rejected. Per-fetcher start_time/end_time default to None and inherit the run window from these top-level keys at runtime (there is no global: key — the top level is the global scope).

Deduplication (dedup)

Posts are always deduplicated by (source, id). With dedup: url the collector additionally drops posts whose normalized article URL was already seen — tracking parameters (utm_*, fbclid, …), fragments, and trailing slashes are stripped before comparing — which collapses the same story fetched via different sources (e.g. a publisher's RSS feed and GDELT). First occurrence in config order wins, so put the richest source first. Note that Google News links are news.google.com redirect URLs, so they never URL-match the publisher's own links.

Authenticated fetcher types and secrets

The base fetchers are strictly no-auth. Sources whose anonymous endpoints are commonly blocked from datacenter/cloud IPs have a separate authenticated variant — reddit-auth and bluesky-auth — whose credential fields default to environment-variable references, so the minimal config is just the type:

fetchers:
  - type: reddit-auth            # reads ${REDDIT_CLIENT_ID} / ${REDDIT_CLIENT_SECRET}
    subreddits: ["news"]
  - type: bluesky-auth           # reads ${BLUESKY_IDENTIFIER} / ${BLUESKY_APP_PASSWORD}
    query: "cyprus"

Validation fails immediately (exit 2, naming the missing variable) when the environment isn't set — auth problems surface at load time, never as a mid-run 401. Any credential field also accepts a literal value or an explicit ${OTHER_VAR} reference (mastodon.access_token works the same way inline).

Relative windows

window accepts (case-insensitive): "last 6 hours", "past 30 minutes", "last 7 days", "last week", "last month", "today", "yesterday", or a bare duration like "6h", "2d", "90m". It resolves once to a concrete (start_time, end_time) pair (end = now), so collection stays deterministic for that resolved pair. The same parsing is available programmatically via fetchkit.resolve_window(spec) and fetchkit.parse_duration(text).

Hacker News (type: hackernews)

RSS / Atom (type: rss)

feeds[].headers sends extra HTTP headers with that feed's request — typically a custom User-Agent for sites that block non-browser clients:

feeds:
  - url: "https://www.havadiskibris.com/rss"
    headers: { User-Agent: "Mozilla/5.0 (compatible; my-pipeline)" }

feeds[].url is restricted to HTTP(S) URLs by default, and each URL's host must resolve to a public address — feeds pointing at loopback, private (RFC-1918), link-local, or cloud-metadata (169.254.169.254) hosts are refused to prevent SSRF. Setting allow_local_files: true additionally permits local file paths and file:// URLs and opts out of the SSRF guard (it declares the config trusted) — see the security note below before enabling it.

⚠️ Security — untrusted configs. fetchkit is built to run YAML that may be produced by an LLM/agent. By default it blocks two abuse vectors in RSS feed URLs: local file reads (e.g. file:///etc/passwd) and SSRF to internal addresses (e.g. http://169.254.169.254/... or http://127.0.0.1). Both protections are on by default. allow_local_files: true turns both off, so enable it only for configs and fixtures you trust. The SSRF guard is reasonable, not perfect — it checks the host at request time and does not defend against DNS rebinding or redirects to private hosts.

arXiv (type: arxiv)

Uses the arXiv export API (Atom, parsed with feedparser). The full abstract is in post.text; authors, categories, DOI, PDF link, and primary category are preserved in post.metadata. post.url is the human-readable abstract page by default (link_format: pdf for direct PDFs — the PDF link is always in metadata.pdf_url either way).

GitHub (type: github)

Public GitHub REST API (no auth; anonymous quota is 60 requests/hour). Repo, language, stars, forks, topics, description, pushed_at, and updated_at are preserved in post.metadata.

The query string is passed verbatim to the GitHub Search API, so the full repository-search syntax works: language:python topic:llm stars:>100. For an "active this week" digest use sort: updated + window_qualifier: pushed — the default (created) windows on repo creation date, which can surface repos untouched since day one.

Lobsters (type: lobsters)

The lobste.rs public JSON endpoints (no auth). Tags are preserved in post.metadata.

Stack Exchange (type: stackexchange)

The Stack Exchange API (no auth; anonymous access is capped at 300 requests/day/IP). Questions become Posts; with comments.fetch: true, top answers are attached as Comments. Tags, answer count, and the accepted-answer id are kept in post.metadata.

Bluesky (type: bluesky, type: bluesky-auth)

bluesky is the public AppView (public.api.bsky.app), unauthenticated by design — but it blocks some datacenter/cloud IP ranges (403). bluesky-auth takes the same resource/query/actor fields and calls the same XRPC endpoints, but through a session created on service (default bsky.social) with an app password — never the account password. (Bluesky is migrating to full AT Protocol OAuth and calls app passwords legacy, but they remain the supported path for headless scripts.) uri, cid, and langs are kept in post.metadata; likes map to score and replies to comment_count.

Mastodon (type: mastodon)

Public and hashtag timelines on any instance. Anonymous access works while the instance keeps public preview enabled, but it is the first to be rate-limited (429 under load) or disabled (401). Setting access_token sends authenticated requests, which lifts both restrictions — create a token on the instance under Settings → Development → New application (read scope is enough). HTML content is reduced to plain text; tags, instance, and visibility are kept in post.metadata.

SEC EDGAR (type: edgar)

Securities filings from the SEC's free, keyless EDGAR APIs. resource: filings pulls recent filings per company from data.sec.gov/submissions (acceptance timestamps are near-real-time — new 8-Ks appear within minutes); resource: search runs EDGAR full-text search across all filers (date granularity only; the run window is passed to the API as a date range). Form type, CIK, accession number, and 8-K item codes are kept in post.metadata.

The SEC's fair-access policy asks clients to identify themselves with contact info and stay under 10 req/s — set user_agent and consider http.rate_limit_per_host.

Reddit (type: reddit, type: reddit-auth)

Subreddit listings or search. listing: new surfaces posts seconds after submission (engagement is near zero at that point by definition); hot/top carry engagement at the cost of recency. Score, comment count, and upvote ratio are captured as of fetch time. Subreddit, flair, and upvote ratio are kept in post.metadata.

• reddit — anonymous, via the public .json endpoints. Tolerated at low volume (~10 requests/minute per IP) but formally a gray area of Reddit's Data API terms, and Reddit blocks many datacenter/cloud IPs outright (403 regardless of User-Agent) — expect this to fail from CI or a cloud host.
• reddit-auth — same fields plus OAuth through oauth.reddit.com with a token from a registered app (reddit.com/prefs/apps) — the sanctioned path, and the one that works from cloud infrastructure. Without username/password an application-only token is used (fine for reading public listings); set both to use the password grant of a script-type app (2FA accounts append the code as password:123456). Reddit asks OAuth clients for a descriptive user_agent like linux:my-pipeline:v1 (by /u/you).

GDELT (type: gdelt)

The GDELT Project's DOC 2.0 API — machine-reads worldwide news media in 65 languages, re-indexed every 15 minutes. A zero-auth global news firehose. Hits carry URL, title, timestamp, source domain/country/language (kept in post.metadata); there is no author, body text, or engagement, so those fields are None. The run window maps to the API's startdatetime/enddatetime.

Google News (type: googlenews)

Google News' public RSS endpoints: top stories, search, or a topic section. Minutes-fresh headlines aggregated from thousands of publishers. The query language supports when:2h (recency), site:, quoted phrases, -exclusions, and intitle:. Links are Google News redirect URLs (they resolve to the publisher); the publisher name maps to author and a headline snippet to text.

Boolean queries: Google News ANDs space-separated terms, and long X OR Y OR Z chains are parsed unreliably — results silently degrade into generic headlines (fetchkit warns on stderr when it spots the pattern). To OR several topics, use queries: each entry is fetched as its own RSS request and the results are merged and deduplicated. queries × required_domains is capped at 8 requests per fetcher; validation rejects configs beyond that.

Polymarket (type: polymarket) and Manifold (type: manifold)

Prediction markets as posts: the market question is the title, and live odds, volume, and close date land in post.metadata (odds/probability, volume_24h, …). Odds moving is often a signal about breaking news before articles are written. Both APIs are public read-only, no auth.

Snapshot semantics: markets are live state, not timestamped events, so these two fetchers deliberately ignore the run window — an active market created months ago is still current. created_at is the market's creation time, which also means snapshot posts sort below newer article-shaped posts in a mixed feed.

HTTP (http:)

Timeouts and connection errors are retried max_retries times with exponential backoff (429/503 honor Retry-After), and errors that exhaust their retries say so (... (after 4 attempts with backoff)). If a host throttles parallel runs (e.g. GDELT from one IP), set rate_limit_per_host to smooth requests out instead of racing them.

Identify your project (User-Agent)

fetchkit is a library — traffic it generates belongs to your project, so set a User-Agent that describes your use case, ideally with contact info. Site operators use it to attribute (and tolerate) automated traffic; anonymous generic clients are the first thing they block. Two equivalent ways:

http:
  user_agent: "cyprus-news-pipeline/1.0 (ops@example.com)"

export FETCHKIT_USER_AGENT="cyprus-news-pipeline/1.0 (ops@example.com)"   # no config change needed

The YAML field wins over the environment variable. Your identity is sent first, with the library's product token appended per HTTP convention: cyprus-news-pipeline/1.0 (ops@example.com) fetchkit-agents/0.4.0. If neither is set, a generic fetchkit-agents (+https://pypi.org/project/fetchkit-agents/) identifier is used — fine for experiments, too anonymous for anything running on a schedule. Escape hatches that send exactly what you write (no token appended): per-feed rss headers and the per-fetcher user_agent fields on reddit and edgar.

Adding a fetcher

Fetchers live in the library itself, so each one ships with a typed config, validation, and tests. Add a new source via a PR or a local fork in four steps:

1. Add a typed config to src/fetchkit/schemas/fetcher.py (subclass FetcherBase, give it a Literal type), and register it in _BUILTIN_TYPES + FetcherConfig.

2. Write the fetcher module in src/fetchkit/fetchers/, returning a FetcherResult:

   from fetchkit.fetchers.base import FetcherResult
   from fetchkit.fetchers.registry import register_fetcher
   from fetchkit.schemas.post import Post
   
   @register_fetcher("mysource")
   def fetch(config) -> FetcherResult:
       posts = [Post(id="1", source="mysource", title="…", source_url="https://…")]
       return FetcherResult(posts=posts, errors=[])
   

3. Import the module in src/fetchkit/fetchers/__init__.py so it registers on import.

4. Add tests (mock HTTP with the responses library — see tests/fetchers/).

5. (Optional) register a discovery helper so agents can find your source's knobs:

   from fetchkit.fetchers.suggest_registry import register_suggester
   
   @register_suggester("mysource")
   def suggest(*, query=None, limit=20, **kwargs) -> list[dict]:
       return [{"tag": "…"}]   # JSON-ready rows; surfaced via `fetchkit suggest mysource`
   

Use Post.metadata for any source-specific fields that don't map to the canonical columns.

The Post model

class Post(BaseModel):
    id: str                       # unique within source
    source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
                                  #   | "stackexchange" | "bluesky" | "mastodon" | "edgar"
                                  #   | "reddit" | "gdelt" | "googlenews" | "polymarket" | "manifold"
    title: str | None
    text: str | None              # body / content
    url: str | None               # external link
    author: str | None
    score: int | None             # source-relative; NOT comparable across sources
    comment_count: int | None
    created_at: datetime | None   # UTC-aware
    source_url: str               # direct link on the source platform
    comments: list[Comment]       # nested threads (HN)
    metadata: dict[str, Any]      # source-specific extras (categories, stars, tags, …)

All datetimes are normalized to UTC. Posts are deduplicated by (source, id) and sorted descending by (created_at, id) for deterministic output.

The text contract

text carries the best available body text at the source, so a pipeline can summarize/keyword-extract every post without a second HTTP request and without special-casing fetchers. It is null only when the source genuinely has no body — never because fetchkit skipped it.

Window semantics per source

The run window filters on created_at, but each source's clock means something different — a "past week" window is repo-creation time on GitHub (window_qualifier: pushed switches it to push activity), submission time on HN/Lobsters, publication time on news sources, and is deliberately ignored by the prediction-market fetchers (live state). Run fetchkit guide for the full per-source table of what created_at timestamps and whether the window is applied server-side or client-side.

score is source-relative. Each source defines it differently — Hacker News points, Lobsters score, GitHub stars (for search_repos), Stack Exchange question score, Bluesky likes, Mastodon favourites, Reddit score (fuzzed by Reddit, and near zero on brand-new posts), and None for arXiv, RSS, Google News, GDELT, EDGAR, and the prediction-market sources. The values are not comparable across sources, so don't rank a mixed feed by score directly. Compare within a single source, or use a source-aware ranking of your own. Output is ordered by recency (created_at), not by score.

Collector invariants

collect_all preserves three guarantees:

1. Window inheritance — per-fetcher start_time/end_time fall back to the global window when omitted.
2. Dedup — by (source, id); first occurrence wins.
3. Sort — descending by (created_at or UTC_MIN, id).

Partial failures are aggregated, not fatal: a failing source yields an entry in result.errors while other sources still collect. Check result.has_errors.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# tests (skip live/networked)
pytest -m "not live"

# live network smoke (opt-in)
pytest -m live

# strict typecheck
mypy src

License

MIT — see LICENSE. MIT is intentional: as a small, dependency-light utility meant to be embedded freely in agent pipelines (including commercial and closed-source ones), a permissive license maximizes adoption with no copyleft obligations. A weak-copyleft license (e.g. MPL-2.0) or strong copyleft (GPL/AGPL) would force redistribution terms on downstream users and discourage exactly the embedded use fetchkit is built for.