Autofeeder

Local-first intelligence extraction. Point it at RSS feeds, websites, or APIs. Define what you want to pull out. Get a structured DuckDB table on your machine — no cloud, no accounts, no data leaving your box.

Install

One command. Opens in your browser when done. Works on macOS, Linux, and Windows — including locked-down enterprise machines.

macOS / Linux

curl -fsSL https://raw.githubusercontent.com/arunavdaniel/Autofeeder/main/install.sh | sh

Windows (PowerShell)

iwr -useb https://raw.githubusercontent.com/arunavdaniel/Autofeeder/main/install.ps1 | iex

No curl? Any machine with Python works

python3 install.py

Download install.py first, then run it. That's it.

After install, Autofeeder starts automatically and opens http://127.0.0.1:8765 in your browser.

What it does

Autofeeder is an end-to-end extraction pipeline that runs entirely on your machine:

SOURCES  →  FETCH  →  SNAPSHOT  →  TEXT EXTRACTION  →  LLM  →  DUCKDB
  RSS            HTTP/browser     trafilatura          any      queryable
  Atom           or Playwright    clean text        OpenAI-    structured
  Websites                                         compat     table
  APIs

1. Subscribe to sources — RSS/Atom feeds, arbitrary public websites, or JSON APIs. Group into folders.
2. Snapshot — Capture a reproducible point-in-time set of articles.
3. Extract text — Pull clean readable body text from raw HTML using trafilatura.
4. Define a schema — Describe what structured fields you want out of each article (name, type, description).
5. Run LLM extraction — Send article text + your prompt to any OpenAI-compatible model. Get validated JSON back.
6. Write to DuckDB — Records land in a local DuckDB table with metadata columns stamped automatically.
7. Query — Open the built-in SQL viewer, run any query, export to CSV/Parquet/JSON.
8. Automate — Schedule any pipeline on an interval or daily timer.

What you can build with it

Installer options

# Skip Playwright browser download (faster; JS-heavy article fetching unavailable)
python3 install.py --no-browser

# Custom install location
python3 install.py --dir /opt/autofeeder

# Install a specific version
python3 install.py --version 0.2.1

# Corporate SSL inspection / custom CA certificate
python3 install.py --ca-bundle /etc/ssl/corporate-ca.pem

# Fully offline / air-gapped — requires a pre-downloaded bundle zip
python3 install.py --offline --bundle ~/Downloads/autofeeder-bundle.zip

# Preview what would happen without making any changes
python3 install.py --dry-run

# Uninstall
python3 install.py --uninstall

Enterprise compatibility

The installer works on locked-down machines with no admin rights:

Re-running after install

~/.autofeeder/autofeeder          # macOS / Linux
~/.autofeeder/autofeeder.bat      # Windows

If Autofeeder is already running, the launcher detects it and just opens your browser to http://127.0.0.1:8765 without starting a second instance.

Features

Pipeline stages (only use what you need)

SOURCE → FETCH → SNAPSHOT → CHANGE DETECTION → TEXT EXTRACTION
                                    ↓
                       CHUNKING → EMBEDDING → LOCAL VECTOR SEARCH
                                    ↓
                       LLM EXTRACTION → VALIDATION → DEDUPLICATION
                                    ↓
                            DUCKDB → EXPORT / REPLICATION

DuckDB output schema

Every extracted record carries automatic metadata columns:

ingested_at   TIMESTAMP   -- when the record was written
feed_title    VARCHAR     -- source feed or website name
article_url   VARCHAR     -- original article URL
run_id        BIGINT      -- links back to the pipeline run

Privacy

• LLM API keys are used for the outbound request only. They are never written to disk or the local database.
• Endpoint, model, and prompt live in browser localStorage.
• No telemetry. No accounts. No network calls except to feeds and the LLM endpoint you configure.

LLM configuration

Autofeeder works with any OpenAI-compatible chat completions endpoint:

Configure in Settings → LLM. The key is used only for the duration of the request.

Optional extras

# Playwright browser (for JS-heavy article fetching)
pip install 'autofeeder[browser]'
python -m playwright install chromium

# Database sync / upsert (Postgres, MySQL, MSSQL, Oracle)
pip install 'autofeeder[sync]'

# Sentence-level embeddings via sentence-transformers
pip install 'autofeeder[embeddings]'

Connection strings: postgres://user:pass@host:5432/db, mysql://user:pass@host:3306/db, mssql://user:pass@host:1433/db, oracle://user:pass@host:1521/ORCL.

Developer setup

git clone https://github.com/arunavdaniel/Autofeeder.git
cd Autofeeder
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e .
rss-text-reader                  # starts server + opens browser

Build the frontend (only needed when editing frontend source):

cd frontend
npm install
npm run build                    # output goes to rss_reader/frontend_dist/

Architecture

flowchart TB
  subgraph Frontend["React 19 + Vite SPA (served from rss_reader/frontend_dist/)"]
    UI["Overview · Sources · Discover · Websites · Schemas · Pipelines\nDuckDB · Embeddings · Semantic Search · Exports · Schedules · Run History · Stats · Settings"]
  end

  subgraph Backend["Flask backend (:8765)"]
    Web["web.py — REST API + static SPA"]
    Feeds["feeds.py — RSS/Atom + website fetch"]
    Extract["extractor.py / llm.py — LLM JSON extraction"]
    Pipe["pipeline.py — orchestration + scheduler"]
    Duck["duckstore.py — DuckDB engine"]
    Vec["vectorstore.py — local embeddings + cosine search"]
    DB[("database.py — SQLite metadata")]
  end

  subgraph Storage["Local data (no cloud)"]
    DDB[("DuckDB files (.duckdb)")]
    SQL[("SQLite metadata DB")]
    ART[("Article text snapshots")]
  end

  UI <-->|HTTP /api| Web
  Web --> Feeds & Extract & Pipe & Duck & Vec & DB
  Extract --> LLM["OpenAI-compatible LLM\n(Ollama / LM Studio / OpenAI / any)"]
  Duck --> DDB
  DB --> SQL
  Feeds --> ART

Project structure

Autofeeder/
├── rss_reader/                 # Python backend (Flask)
│   ├── web.py                  # REST API + SPA serving
│   ├── database.py             # SQLite metadata (feeds, schemas, pipelines, runs)
│   ├── duckstore.py            # DuckDB engine (query, import, alter, export)
│   ├── extractor.py            # extraction orchestration
│   ├── llm.py                  # OpenAI-compatible LLM client
│   ├── feeds.py                # RSS/Atom fetch + parse
│   ├── fetchers.py             # HTTP + Playwright fetcher abstraction
│   ├── pipeline.py             # pipeline runner + scheduler loop
│   ├── chunker.py              # paragraph/sentence chunking
│   ├── embeddings.py           # embedding provider abstraction
│   ├── vectorstore.py          # local cosine similarity search
│   ├── website.py              # website monitor + change detection
│   ├── catalog.py              # source catalog / discovery
│   ├── json_mapping.py         # JSON/API extraction without LLM
│   ├── publish.py              # RSS/JSON feed output
│   ├── backup.py               # backup / restore
│   └── frontend_dist/          # pre-built React SPA (committed, ships with pip)
├── frontend/                   # React 19 + Vite + TypeScript source
│   └── src/
│       ├── pages/              # one file per page/route
│       ├── components/         # ui primitives, layout, charts
│       └── lib/                # api.ts, types.ts, utils
├── install.py                  # cross-platform one-line installer
├── install.sh                  # macOS/Linux curl wrapper
├── install.ps1                 # Windows PowerShell wrapper
├── packaging/                  # PyInstaller spec (native app builds)
├── tests/
└── pyproject.toml

Data location

DuckDB files are stored in the DuckDB viewer folder you configure in the app. SQLite metadata and article snapshots go to the data location above.