litellm-ducklake-sink

An out-of-tree LiteLLM proxy logging callback that batches request telemetry and appends it to DuckLake tables through an Arrow Flight SQL endpoint. Metrics land in llm_requests, optional payloads in llm_payloads, both partitioned by day and queryable with SQL the moment they land.

It works with any DuckLake database exposed through an Arrow Flight SQL server (for example QoD - quack-on-demand). The SQL it emits is plain CREATE TABLE IF NOT EXISTS, INSERT ... VALUES, and DELETE, plus DuckLake's ALTER TABLE ... SET PARTITIONED BY at bootstrap, so the backend must be DuckDB with the DuckLake extension, but nothing in the write path is specific to one server. A few optional settings integrate with QoD deployments and are marked as such below.

Install

pip install litellm-ducklake-sink

Install it into the same environment that runs the LiteLLM proxy.

Configure

Reference the callback instance in the proxy YAML:

litellm_settings:
  callbacks: litellm_ducklake_sink.callback.instance

All settings come from DUCKLAKE_SINK_* environment variables.

QoD-only settings

These settings only apply when the endpoint is a QoD (quack-on-demand) gateway. Leave them all unset on any other Flight SQL server.

Startup and failure behavior

The callback instance is created when the proxy imports it at startup, and misconfiguration fails fast: if the sink is enabled but a required setting is missing or a value cannot be parsed, the proxy refuses to boot rather than silently dropping telemetry. Set DUCKLAKE_SINK_ENABLED=false to keep the callback referenced in config.yaml but fully inert: every other setting is ignored, nothing is validated, and no spool directory, connection, or shutdown hook is created. An explicitly empty value (DUCKLAKE_SINK_ENABLED=) is rejected as malformed rather than silently falling back to enabled.

An unreachable or wrong endpoint does not block startup. Connections are opened lazily at first flush, so with bad connectivity the proxy serves traffic normally while batches spool to disk and replay on later flush cycles once the endpoint is reachable. Batches spooled before the sink is disabled stay on disk untouched until it is enabled again.

Table creation

No manual DDL is needed. On the first flush of each process the sink runs CREATE TABLE IF NOT EXISTS for llm_requests and llm_payloads in schema_name, then ALTER TABLE ... SET PARTITIONED BY (request_day) on each, before the first insert. The schema itself must already exist (main always does). The connecting user therefore needs DDL and write privileges on that schema; under QoD with ACL enabled, grant CREATE/ALTER alongside INSERT (and DELETE for the retention job).

Delivery semantics

Delivery is at-least-once, not exactly-once. Duplicates are possible when a drain times out, the process is killed mid-flush, or several proxy workers replay spooled batches concurrently after a crash. Exact spend accounting downstream should deduplicate by request_id.

The disk spool lives at spool_dir and is capped at spool_max_bytes; once full, the oldest batches are evicted first, and evicted batches are dropped by design rather than retried.

The default spool location under the system tempdir is ephemeral in most container setups: a pod restart discards any batches spooled during an outage. Point DUCKLAKE_SINK_SPOOL_DIR at a persistent volume when spooled telemetry must survive restarts. Workers on the same host share one spool safely; separate proxy replicas each keep their own.

Replay of spooled batches runs on the periodic flush cycle, which starts on the first logged request after process start rather than immediately at boot. A proxy that restarts and then sits idle only replays its spool once it handles its first request.

Multi-worker deployments share one spool safely: workers claim a batch before replaying it, so the same batch is never replayed twice, and a claim left behind by a worker that crashes mid-replay is automatically recovered by another worker after 15 minutes.

Retention

python -m litellm_ducklake_sink.retention

Run it out-of-band (cron or a Kubernetes CronJob). It deletes rows older than DUCKLAKE_SINK_RETENTION_DAYS. When DUCKLAKE_SINK_ENABLED=false it exits 0 without connecting, so a shared env file disables the proxy sink and the retention job together.

Deletes in DuckLake are logical: old snapshots still reference the deleted rows' Parquet files until snapshot expiry and file cleanup run. On QoD, set DUCKLAKE_SINK_MAINTENANCE_URL, DUCKLAKE_SINK_TENANT, and DUCKLAKE_SINK_TENANT_DB and the retention job triggers QoD's managed maintenance chain after the deletes so the files are physically removed. On any other DuckLake setup, run ducklake_expire_snapshots and ducklake_cleanup_old_files yourself on whatever schedule your deployment uses.