metatron-cli 0.2.1

Multi-project web feed manager with cross-outlet deduplication

Metatron

Multi-project web feed manager with cross-outlet deduplication.

You point Metatron at sources, organized by project. It polls them, dedups the articles (URL canonicalization → normalized title → token overlap → LLM tiebreaker), and serves you the unique articles via a small HTTP API. RSS is the source it speaks today; the architecture is medium-agnostic and grows to other web sources over time.

It does this one thing. It doesn't curate, it doesn't summarize, it doesn't have opinions about what's interesting. That's for whatever calls it.

Install

The CLI is metatron. The published package is metatron-cli (the bare metatron name was already taken on PyPI):

pipx install metatron-cli                                   # from PyPI
pipx install git+https://github.com/mattweberio/metatron    # from GitHub (latest main)

Configure

metatron config init       # writes ~/.config/metatron/config.toml
metatron config show       # prints the resolved config

Edit ~/.config/metatron/config.toml. The LLM tiebreaker — needed to catch "same story, different outlet" duplicates — requires an Anthropic API key:

[llm]
api_key = "sk-ant-..."
model = "claude-sonnet-4-6"

[api]
api_token = "your-bearer-token"

Without an LLM key, dedup falls back to URL + normalized-title matching only. Cross-outlet duplicates from different sources will leak through.

Run

metatron serve             # HTTP API on 127.0.0.1:8765

Use

Bearer-token auth on everything except /health.

TOKEN="your-bearer-token"
BASE="http://127.0.0.1:8765"

# Create a project
curl -s -X POST "$BASE/projects" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "ai-news"}'

# Add feeds
PROJECT_ID=...
curl -s -X POST "$BASE/projects/$PROJECT_ID/feeds" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hnrss.org/frontpage"}'

# List deduplicated articles
curl -s -H "Authorization: Bearer $TOKEN" \
  "$BASE/projects/$PROJECT_ID/articles?limit=20"

# Force a synchronous refresh
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  "$BASE/projects/$PROJECT_ID/refresh"

Bulk feed import

metatron seed-feeds my-project ./feeds.json

Where feeds.json is:

{
  "feeds": [
    { "url": "https://hnrss.org/frontpage", "name": "Hacker News", "category": "tech" },
    { "url": "https://www.theverge.com/rss/index.xml", "name": "The Verge", "category": "tech" }
  ]
}

How dedup works

Cheapest-to-most-expensive, short-circuits on first match:

1. Canonical URL. Strip utm_*, fbclid, etc. Same canonical URL → exact duplicate, dropped.
2. Normalized title. Lowercase, drop punctuation, strip newsroom prefixes ("BREAKING: ", "UPDATE - "). Same → joins the existing article's cluster.
3. Token overlap. Jaccard similarity on title + summary tokens. ≥0.12 → shortlist candidate (top 5).
4. Sonnet tiebreaker. Claude Sonnet reads both articles and decides "same story" or "different story." Same → joins cluster.

Articles in a cluster are stored individually (you can see the alternative sources via GET /articles/{id}) but /projects/{id}/articles returns only the canonical from each cluster.

Storage

Single SQLite file at ~/.local/share/metatron/metatron.db. WAL mode, FK constraints on. Override with [database].path in config.

API endpoints

Method	Path	Description
GET	/health	Liveness; reports LLM enablement
POST	/projects	Create a project
GET	/projects	List projects
DELETE	/projects/{id}	Delete a project (cascades feeds + articles)
POST	/projects/{id}/feeds	Add a feed
GET	/projects/{id}/feeds	List feeds
DELETE	/feeds/{id}	Remove a feed
GET	/projects/{id}/articles?since=&limit=	List deduped articles
POST	/projects/{id}/refresh	Synchronously poll project's feeds
GET	/articles/{id}	Article detail + cluster members

Development

python3 -m pytest -q

Tests cover normalization, the DB layer, the dedup pipeline (with a fake LLM judge), the feed poller (with mocked RSS), and the API surface.

License

MIT. See LICENSE.