English | 简体中文

Sluice v4

Sluice collects RSS and webpage content, runs YAML-defined task graphs, and delivers results through independently retried Sinks. SQLite stores publications, execution state, shared Stage results, and Delivery attempts. Local commands and the optional Prefect flow use the same runtime.

Typical uses are watching a webpage for changes, forwarding RSS entries to Telegram, email, or Feishu, and sending an LLM-written digest of a feed. Each use is a YAML definition that you run with the sluice command.

This branch is the v4 clean-room implementation. Use a separate database rather than pointing it at an older Sluice deployment. The runtime uses fcntl file locking and targets Unix-like systems; only one App can own a database at a time. See Operate Sluice.

Install

Sluice needs Python 3.12 or later. Choose one of the three routes below.

From source

Install uv, then run this command from the repository root:

uv sync --locked --extra dev --extra web --extra ui --extra prefect

The command installs sluice as .venv/bin/sluice. The examples below use that path.

With pip

The distribution is named sluice-pipeline; the Python import and command are sluice. To install into the current Python environment, run pip install '.[web,ui,prefect]' from the repository root, or pass the checkout path instead of .. Omit extras you do not use. Email, Telegram, and LLM integrations need their own configuration and credentials; the console example does not exercise them.

The release workflow publishes sluice-pipeline to PyPI when a v*.*.* tag is pushed. This repository does not record which versions are currently on PyPI, so install from the checkout as shown above rather than assuming a published v4 release.

With Docker

Build the image from the repository root and check that it starts:

docker build -f scripts/docker/Dockerfile -t sluice-pipeline:local .
docker run --rm sluice-pipeline:local --help

The release workflow pushes ghcr.io/nerdneilsfield/sluice-pipeline and nerdneils/sluice-pipeline when a v*.*.* tag is pushed. This repository does not record which tags are currently published; the local build above is the verified route.

The image runs sluice as its entrypoint from /app and includes the web, ui, and prefect runtime extras. The directories /app/configs and /app/data exist for definitions and state. The default database path is relative to the YAML file, so for a read-only definition mount, set an absolute database such as /app/data/state.db in the YAML:

docker run --rm \
  -v "$PWD/configs:/app/configs:ro" \
  -v "$PWD/data:/app/data" \
  sluice-pipeline:local tick /app/configs/definition.yml \
  --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

Replace configs/definition.yml with your definition. The image sets no USER, so the container writes files in the mounted state directory as root. A definition that uses a custom Sink or other dotted-path callable needs that module importable inside the container, for example by mounting it under /app/configs and adding -e PYTHONPATH=/app/configs.

Run the webpage example

From the repository root, after installing from source:

PYTHONPATH="$PWD" .venv/bin/sluice tick examples/webpage.yml \
  --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

The example fetches https://example.com/, extracts visible content, converts it to text, and prints a JSON object containing delivery_key and payload. It does not send email or Telegram messages. Internet access is required. PYTHONPATH makes the example's custom Sink, examples.console.send, importable; it is not needed for definitions using only installed built-ins.

State is stored in examples/webpage.db, relative to the YAML file. To inspect the result without relying on the CLI exit status:

.venv/bin/python - <<'PY'
import sqlite3
with sqlite3.connect('examples/webpage.db') as db:
    print(db.execute('SELECT kind, state, error FROM executions').fetchall())
    print(db.execute('SELECT state FROM deliveries').fetchall())
PY

On a successful first run, Source, Stage, and Delivery states are succeeded. The CLI can return normally after recording an execution failure, so check these states when diagnosing a run. The Source has a one-hour schedule; an immediate second tick need not fetch again. Unchanged content does not create another delivery.

Pipeline concepts

examples/webpage.yml contains an explicit HTTPX provider, fetch chain, extractor chain, Source, Stage, and console Delivery. YAML version: 1 is the definition format, not the package major version.

• sources collect immutable Items and publish changes. Built-ins include sluice.sources.rss and sluice.sources.webpage.
• stages run each or batch task chains.
• pipelines connect Sources and Stages through from edges. Every Pipeline needs a Delivery endpoint.
• web declares named providers and ordered fallback chains. HTTPX and BeautifulSoup work with core dependencies; the web extra adds curl_cffi, trafilatura, newspaper4k, and resiliparse.

Direct web providers reject private and loopback targets to prevent SSRF. Hosted service endpoints have a separate policy. Keep credentials out of YAML values; provider credential fields accept environment references such as api_key: env:FIRECRAWL_API_KEY.

For Crawl4AI, use the 0.9.x configuration guide; older API formats are not supported.

Rules for task chains and transient artifacts, the authored binding identity, and the extension contracts, including dotted-path callable requirements, are in Write a definition.

Scenario examples

Webpage. examples/webpage.yml fetches a page, extracts its text, and prints it through a console Sink. It is the example run above.

RSS notifications. RSS 2.0, Atom, and RSS 1.0 feeds can deliver to Telegram, SMTP email, and Feishu custom bots using built-in senders. See notification setup and the three-channel RSS example. This example sends real messages and needs your credentials; the webpage example above only prints to the console.

LLM digest. examples/rss-llm-digest.yml runs RSS → fetch → llm_extract → llm_summary → console, using one OpenAI-compatible /chat/completions endpoint configured in a top-level llm block. See LLM digest setup.

Common commands

sluice is a one-shot runner, not a background scheduler. Supply an ISO-8601 timestamp with a timezone for execution commands:

sluice tick definition.yml --at 2026-09-22T00:00:00Z
sluice reconcile definition.yml pipeline_name node_name --at 2026-09-22T00:00:00Z
sluice rebuild definition.yml 42 --at 2026-09-22T00:00:00Z
sluice retry-delivery definition.yml DELIVERY_KEY --at 2026-09-22T00:00:00Z
sluice ui definition.yml --host 127.0.0.1 --port 8000

Replace definition.yml, Pipeline/Node names, execution ID 42, and DELIVERY_KEY with values from your deployment. Retrying a Delivery can repeat an external side effect if the Sink did not honor the stable Delivery key. The management UI has mutation actions and no authentication; keep it on a trusted local interface.

Operate Sluice covers scheduling tick with cron, rebuild and retry semantics, the UI, Prefect, the database lock, and inspecting state.

Contributing

Contributions are welcome. CONTRIBUTING.md describes the development setup, the verification commands, the CI matrix, and the release process. v4 verification records the checked scope and remaining limits.