percolate-core

The processes that sit in front of the database. The database itself, and the specs all of this is built against, live in p8-subsystems; nothing here is required to use it.

pip install percolate-core                 # the worker -- asyncpg, httpx, typer
pip install 'percolate-core[content]'      # + Content Server (boto3, fastapi)
pip install 'percolate-core[agent]'        # + Agent Runtime (pydantic-ai, mcp)
pip install 'percolate-core[all]'

percolate worker --queue http    # claim and execute tasks
percolate content serve          # uploads, scraping, ingestion
percolate agent serve            # agents, streaming, delegation
percolate agent --help           # the runtime's own operator commands

docs-sample.md is the one to read next: a real REST call to the Agent Runtime, the events it streams back, the rows it leaves behind, and every header and option a caller can send — captured output, not illustration.

Layout

The import package matches the distribution name exactly, so there is no mapping to remember. (p8 was taken on PyPI.)

One distribution rather than three. Three packages means a version matrix (percolate-content 0.3 requiring percolate-core >=0.2,<0.3) resolved forever, for services that release together and are written by the same people. Splitting later is mechanical; merging two that have drifted is not.

The base install stays small on purpose. The common case is someone writing their own worker, and they should not pull boto3 and pydantic-ai to do it. The root CLI mounts each subpackage's own command group and tolerates its absence, so percolate worker runs with neither installed and percolate content says which extra to install — naming the module that was actually missing, rather than guessing.

percolate_core.core — the one that matters

It decides whose RLS applies, and it exists because that logic was previously written three times on two different database drivers.

from percolate_core.core import as_caller

async with as_caller(claims) as conn:      # claims = the VERIFIED JWT payload
    rows = await conn.fetch("select * from content.resources")

Every service connects as a low-privilege role — set explicitly by as_caller, not inherited from whatever the connection string happens to log in as — and sets the caller's claims per transaction, exactly as PostgREST does. A service that queried as itself would bypass every policy in the collection — not by exploiting anything, just by never presenting an identity for the policies to filter on.

Transaction-local (set_config(..., true)) is not a detail: an unregistered GUC left at session scope survives into the next transaction on a pooled connection, so the following request would inherit the previous caller's identity.

as_service() exists for work with genuinely no user behind it — a scheduled poll, a reconciliation sweep. Deliberately a separate function rather than as_caller(None), so "this query has no user" is something someone wrote down.

One pool per process, with explicit ownership. A service's lifespan owns it (open_pool/close_pool, reference-counted); every short-lived helper borrows it (pool()). Several mounted services in one process share it, and the last one out closes it — without that, one service's shutdown closes the pool another is still streaming through.

Writing your own worker

Most steps need no worker from you:

When you do need one, it is this loop with a handler registered — not a different program:

from percolate_core.worker import handler, run

@handler("transcode")
async def transcode(spec, ctx):
    return {"duration": await ffmpeg(ctx["run_input"]["file_key"])}

run(queue="media")

ctx comes from workflow.get_task_context(): run_input, the accumulated context (so a later step reads an earlier step's output), task_input, step_key, and trace_id/span_id.

The worker holds no table grants. Every interaction is a SECURITY DEFINER function call — claim_task, get_task_context, complete_task, fail_task — which is why "bring your own worker" is safe to offer: a compromised worker can claim and complete tasks, and nothing else.

Raise TerminalError for what will not get better. A bad argument, a missing credential, a 404. Anything else is retried with backoff. The worker is the only thing that knows what a failure means, so it decides and the engine honours the verdict.

Configuration

Environment only. Credentials by reference, never by value: credential_ref: "LLM_API_KEY" on a task names a variable this process resolves, so workflow.tasks stays inspectable and replayable.

Deployment

One image, several entrypoints — they share percolate_core.core, so separate images would be separate builds of the same base and separate tags to keep in step. ENTRYPOINT ["percolate"], and the command selects the service:

content: { image: percolationlabs/percolate-core:0.1.1, command: ["content","serve"] }
agent:   { image: percolationlabs/percolate-core:0.1.1, command: ["agent","serve"] }
gateway: { image: percolationlabs/percolate-core:0.1.1, command: ["agent","gateway"] }
worker:  { image: percolationlabs/percolate-core:0.1.1, command: ["worker","--queue","http"] }

The agent gateway is its own entrypoint because delegation is an MCP call into it: an agent that delegates references it as an ordinary tool_servers row.

For the packaging decision record see PACKAGING.md; for what packaging requires of the design, specs/agentic/brief.md §10.

Status

• core, worker, content — built, and exercised against a live PG19 instance with MinIO.
• agentic — built and verified under the new namespace: tests/smoke.py (6 checks, no database or credentials) and dev/verify.py (38 assertions against a live model, covering streaming, what is and is not persisted, the delegation and span trees, the reload round trip, mounting, session resume, tenant isolation, and a scheduled run completing its own workflow task).
• Still open: no service-surface entries in the specs repo's surface.sql, which by meta/skills/spec-driven-development §7 should exist before the endpoints they describe; and no startup check that the deployed schema is the one this version expects (specs/agentic/brief.md §10.4).

Development

. dev/env.sh          # DSN, test user, JWT, and the LLM key
./dev/db.sh up        # a PG19 cluster of its own, loaded from the specs repo
./dev/stack.sh up     # PostgREST, the retrieval service, the agent gateway
uv run --all-extras python dev/verify.py

dev/db.sh loads specs/*/schema.sql from a p8-subsystems checkout directly — never a copy — so a schema that only works because a harness did something extra fails here. Set P8_SPECS if your checkout is elsewhere.