CATS — Contextual Ambiguity & Trust Scoring

Trust intelligence for OSINT sources — not fact-checking, but source reliability over time.

What CATS measures

CATS scores how reliable a source's behaviour has been over time — narrative consistency, sentiment volatility, publishing gaps, signs of algorithmic manipulation — not whether any single piece of content is true. It does not fact-check, does not label sources as "disinformation", and does not replace human judgement: every score is an ordinal ranking with a full explanation attached, meant to prioritise where a human looks closer.

See it run

$ COHERENCE_BACKEND=sbert cats score ansa.it --source-type news
Source     https://ansa.it   (feed: https://ansa.it/rss.xml, 28 messages, 2026-09-20 → 2026-09-21)
Score      62.87   band: medium_high
Driver     silence (share 39.8%)
Signals    coherence 20.42 · volatility 7.41 · silence 0.0 · gaming 39.0 · domain_provenance 0.0
Language   italian (1.0)      Evidence  28 ≥ 3 ✓
Note       Ordinal score, not a probability. Cross-validate key claims. See docs/architecture.md.

COHERENCE_BACKEND=sbert is not decoration: why.

Try it in 30 seconds

pip install cats-scoring
python -m spacy download it_core_news_lg   # Italian NLP model (large download), needed by `coherence`
cats score <url>                       # any source URL — feed autodiscovery included

Without the spaCy model coherence is not measured: it enters the score as a neutral 50, which can move the score by up to ~20 points. cats score then prints a warning and forces human review. If feed autodiscovery fails on a site (no RSS/Atom feed found), pass the feed URL itself, e.g. cats score https://www.repubblica.it/rss/homepage/rss2.0.xml.

To rank several sources side by side (failures are listed, not fatal):

cats compare ansa.it ilpost.it https://www.repubblica.it/rss/homepage/rss2.0.xml --source-type news
cats compare --file sources.txt --json      # one URL per line, '#' for comments

The table shows score, band, main driver, the four raw signals, message count and review flags. Compare only sources scored together, with the same --source-type and install: a high rank describes publishing behaviour, not truthfulness.

The coherence backend matters

data/calibrated_weights.json was calibrated with the SBERT coherence backend. On the default spaCy-NER backend coherence — ~0.40 of the weight for news sources — is close to inert (mean 1.7 / sd 5.3 against SBERT's 23.3 / 11.6), so a default install forfeits its contribution and lands nearer ≈0.62 concordance than the 0.750 the calibrated weights reach.

For the numbers quoted here:

pip install -r requirements-sbert.txt   # pulls torch + transformers, ~2 GB
export COHERENCE_BACKEND=sbert

It falls back to NER silently when the model is unavailable — check the logs for sbert_loaded to confirm it took effect. Measurement.

Or as a library:

from cats.lite import score_feed

result = score_feed("https://example-news-outlet.it", source_type="news")
print(result["trust_score"], result["band"])
print(result["explanation"]["primary_driver"])

Try it in the browser instead:

A real use case

An OSINT analyst vetting a list of public source URLs before citing them in a report runs cats score <url> on each one and reads the explanation — which signal drove the score, and why — rather than treating trust_score as a verdict. A medium or low band means "look closer, cross-validate", not "discard": CATS surfaces behavioural patterns (irregular cadence, narrative drift, a clone domain) for a human to weigh alongside the actual content. It never outputs a "disinformation" label.

Signals

On top of the four behavioural signals, an asymmetric domain-provenance penalty lowers the score of impersonation/clone domains (rare/cheap TLDs, free-hosting subdomains, brand typo-squats) when a source URL is supplied. It only ever lowers a score, never rewards a clean domain — see architecture.

Trust score bands

⚠️ Scores are ordinal rankings of source reliability, not absolute probabilities. A high band means consistent publishing behaviour, not truthful content: a source that publishes disinformation on a steady schedule can still score high.

Honest limits

• Default NLP accuracy is roughly 55–62% (spaCy NER + TextBlob); optional SBERT/BERT backends do better.
• The default install does not reproduce the calibrated numbers: the shipped weights assume COHERENCE_BACKEND=sbert, and on the default NER backend coherence is close to inert (≈0.62 concordance instead of 0.750). See the backend note.
• Ranking still leans heavily on one signal (silence), with coherence as the main tie-breaker.
• The default stack is Italian-optimised — other languages are detected and flagged, not blocked, but accuracy degrades.
• Calibration is validated on a 56-source train / 53-source future-holdout split — informative, not large-scale.
• Scores are ordinal, not probabilities, and are not a substitute for human review.

Full detail, numbers and the validation history: docs/architecture.md and the calibration findings.

Full deployment

Beyond the zero-infrastructure library, CATS also ships a full multi-tenant FastAPI service — PostgreSQL-backed audit logging, Redis rate limiting, batch scoring, and a GDPR Art. 13–22 contest/appeal flow — behind the same four signals and the same guarantees. See docs/api.md for the REST reference and docs/architecture.md for the deployment topology and security design.

Documentation

Roadmap

CATS has shipped through v1.6 (calibrated weights, the domain-provenance penalty, and the language/evidence guardrails); active work is on adoption (CLI, MCP server, a public demo) before any further signal changes. Full plan: docs/piano_sviluppo_roadmap_2026-09.md.

Contributing

Issues and PRs are welcome — see CONTRIBUTING.md for the dev setup, code standards, and the checklist for adding a new signal. Please also read the Code of Conduct.

If CATS is useful to you, consider starring the repo ⭐ — it helps others doing OSINT/disinformation work find it.

License

MIT — technical@cats-system.org