title: "FeedSpine README" type: readme status: active tags: [feed-spine, pipeline, data-model, deduplication, medallion] created: 2025-01-15 updated: 2026-04-12

What Is FeedSpine?

FeedSpine is a Python framework for collecting structured data from feeds — RSS, JSON APIs, CSV files, SEC EDGAR, financial data providers — and storing it with automatic deduplication, version tracking, and quality-layer promotion.

Every record is identified by a normalized natural key and a SHA-256 content hash. Collect the same feed a thousand times — each item is stored exactly once, with a full sighting history of when and where it was seen.

When to use it

Quick Start

Install

uv add feedspine                      # Core
uv add "feedspine[duckdb]"            # + DuckDB analytical storage
uv add "feedspine[api]"               # + FastAPI REST server
uv add "feedspine[elasticsearch]"     # + Elasticsearch search
uv add "feedspine[entity]"            # + Entity resolution
uv add "feedspine[all]"               # Everything

Collect and deduplicate in 10 lines

import asyncio
from feedspine import create_feed_spine, MemoryStorage, RSSFeedAdapter

async def main():
    storage = MemoryStorage()
    app = create_feed_spine(storage)
    app.register_feed(RSSFeedAdapter(name="news", url="https://news.ycombinator.com/rss"))

    # First run — all items are new
    outcome = await app.collection_service.run_collection("news")
    print(f"New: {outcome.stats.new}, Duplicates: {outcome.stats.duplicates}")

    # Second run — duplicates detected automatically
    outcome = await app.collection_service.run_collection("news")
    print(f"New: {outcome.stats.new}, Duplicates: {outcome.stats.duplicates}")

asyncio.run(main())

Or from the CLI

uv run feedspine collect run --feed news
uv run feedspine feeds list-types        # Show available adapters
uv run feedspine health summary          # Feed health (RAG status)

Architecture

                    ┌──────────────────────────────────────────────┐
                    │              PRESENTATION LAYER              │
                    │  CLI (Typer) · REST API (FastAPI) · MCP      │
                    │  Thin wrappers — delegate to Ops layer       │
                    └──────────────────┬───────────────────────────┘
                                       │
                    ┌──────────────────▼───────────────────────────┐
                    │                OPS LAYER                     │
                    │  OperationContext → OperationResult[T]       │
                    │  query · feed · enrich · schedules · runs    │
                    │  Pure business logic — no transport imports  │
                    └──────────────────┬───────────────────────────┘
                                       │
                    ┌──────────────────▼───────────────────────────┐
                    │             PIPELINE LAYER                   │
                    │  RecordCandidate → dedup → Record + Sighting │
                    │  stages · runner · stats · dedup             │
                    └──────────────────┬───────────────────────────┘
                                       │
┌───────────┐       ┌──────────────────▼───────────────────────────┐
│  Sources   │──────▶│            STORAGE LAYER                    │
│ RSS · JSON │       │  Protocols: StorageBackend, SearchBackend   │
│ CSV · File │       │  Repository pattern + dialect abstraction   │
│ SEC EDGAR  │       ├─────────┬──────────┬──────────┬─────────────┤
│ Polygon.io │       │ Memory  │  SQLite  │  DuckDB  │  PostgreSQL │
└───────────┘       └─────────┴──────────┴──────────┴─────────────┘

Core data flow

1. FeedAdapter fetches raw data from a source and yields RecordCandidate objects
2. Pipeline deduplicates each candidate (natural key + content hash), creating or updating a Record
3. Sighting is logged for every observation — full audit trail of when and where each item was seen
4. Enricher can promote records through medallion layers and add metadata
5. StorageBackend persists everything — swap backends without changing pipeline code

Key primitives

Built-in Feed Adapters

All adapters implement the FeedAdapter protocol — a @runtime_checkable interface with fetch(), initialize(), and close() methods. Write your own adapter in ~30 lines.

Storage Backends

All backends implement the StorageBackend protocol — CRUD, batch operations, natural-key lookup for dedup, sighting tracking, and query with filtering/pagination.

# Swap storage in one line — pipeline code stays the same
from feedspine.storage.backends.duckdb import DuckDBStorage

storage = DuckDBStorage("feeds.duckdb")
app = create_feed_spine(storage)

Enrichment

Enrichers transform records and promote them through medallion layers:

Enrichment is orchestrated through FeedEnrichmentWorker with batch support.

Protocols

FeedSpine defines @runtime_checkable protocols for every extension point:

Implement any protocol to extend FeedSpine — no subclassing, no registration boilerplate.

REST API / CLI / MCP

FeedSpine ships with three transport layers. All delegate to the same ops/ business logic.

FastAPI REST API

uv run feedspine api serve --port 11300
# → OpenAPI docs at http://localhost:11300/docs

15 route modules: records, feeds, sightings, search, enrichment, health, metrics, stats, timeline, export, schedules, syndication (RSS/OPML), observations, runs, storage, and collection.

Typer CLI

uv run feedspine collect run --feed sec-filings     # Collect from a feed
uv run feedspine feeds list-types                   # List available adapters
uv run feedspine feeds list                         # List configured feeds
uv run feedspine health summary                     # Feed health (RAG: Red/Amber/Green)
uv run feedspine stats summary                      # Record counts, layer distribution
uv run feedspine query records --limit 10           # Query stored records
uv run feedspine export json output.json            # Export to JSON/CSV/Parquet
uv run feedspine info                               # System info

MCP Server (Model Context Protocol)

13 tools for LLM integration — feed collection, enrichment, timeline queries, search, health, and storage stats:

uv run feedspine-mcp                                # Start MCP server (stdio)

Search

Both implement the SearchBackend protocol with index(), search(), delete(), exists(), and initialize().

Examples

25 runnable examples across 7 categories:

uv run python examples/01_getting_started/01_quickstart.py
uv run python examples/run_all.py                          # Run all 25

Development

uv sync --dev                # Install all dependencies
uv run pytest                # 1217 tests
uv run ruff check .          # Lint
uv run ruff format .         # Format
uv run mypy src              # Type check
uv run mkdocs serve          # Local docs site

Project stats

Stability

Contributing

See CONTRIBUTING.md.

License

MIT — see LICENSE.