duckless 0.4.2

Documentation · Quickstart · Examples · Benchmarks · Releases · Contributing

Serverless DuckDB on GCP. Submit SQL or your own code; DuckLess runs it in your project on a right-sized machine (Cloud Batch VM with local SSD for spill, or Cloud Run Jobs for small jobs), reads and writes GCS with the job's own credentials (no HMAC keys), then tears it down. Nothing runs between jobs. Jobs read Parquet, CSV, JSON, Excel and BigQuery tables, and write BigQuery tables; optional DuckLake tables on a Cloud SQL catalog, and Agent Skills for coding agents.

Status: v0.4, young and tested on real projects. Changes are listed in the GitHub Releases; the benchmarks have the measurements behind the design.

Quick start

uv tool install duckless                     # or: pip install duckless

duckless init --project my-project --region europe-west1
# prints the lines to put in .envrc (DUCKLESS_PROJECT, DUCKLESS_BUCKET, DUCKLESS_SA, DUCKLESS_IMAGE)

duckless preflight --machine n2-highmem-32 --spot
duckless run job.sql --machine n2-highmem-32 --spot
duckless exec --image <your-image> --machine n2-highmem-16 -- dbt build
duckless status <job-id>
duckless logs <job-id> --follow
duckless result <job-id>
duckless destroy --project my-project       # removes what init created

duckless init is optional. It applies the Terraform module shipped with each release (duckless/terraform) through Infrastructure Manager: handy to start in five minutes. Teams that keep their infrastructure as code call the same module from their own Terraform and CI/CD instead, with their deployment identity, and never run init:

module "duckless" {
  source           = "git::https://github.com/tosun-si/duckless.git//duckless/terraform?ref=v0.4.2"
  project_id       = "my-project"
  region           = "europe-west1"
  runner_image_tag = "0.4.2"
}

The CLI only reads the settings the module outputs (envrc), however they were created.

Full guide: tosun-si.github.io/duckless, from the quickstart to the CLI reference.

Writing jobs

• SQL (.sql): ${VAR} placeholders come from the job env (DUCKLESS_BUCKET, --env K=V).

• Python (.py): from duckless_runtime import connect gives a DuckDB connection with GCS auth, spill on local SSD, memory and threads sized to the VM.

• Write to GCS in parallel: COPY … TO 'gs://…' (FORMAT parquet, PER_THREAD_OUTPUT, FILE_SIZE_BYTES '256MB') is 7-8x faster than the single-writer default (~900 MB/s vs ~110 MB/s on 32 vCPU).

• Vectorize Python logic: a row-wise Python UDF runs at ~8k rows/s on one thread; use an Arrow UDF over numpy (type="arrow") or SQL.

• Read BigQuery: bigquery_scan('project.dataset.table', filter = '…') in SQL, read_bigquery(con, table, columns, where) in Python; only the columns and rows asked for leave BigQuery (duckless init --bigquery-dataset <dataset> first).

• Write BigQuery: to_bigquery(con, query, "project.dataset.table", mode=...): Parquet, a free load job, then replace_partitions (deletes whole partitions for free, bills only the new rows) or a merge that reads only the partitions it touches (guide; duckless init --bigquery-write-dataset <dataset> first).

More in Writing jobs and Machines, Spot and spill.

Examples

Runnable pipelines in examples/, on data generated by the first one:

Agent Skills

DuckLess ships a Claude Code plugin with five skills: setup, writing-jobs, sizing, troubleshooting and ducklake. They carry what the docs and the CLI cannot decide for you (which machine, why a job failed) and the rules measured while building DuckLess.

duckless skills install          # into this project: .claude/skills + .agents/skills (--user: your home)

Or as a Claude Code plugin, kept up to date from this repository:

/plugin marketplace add tosun-si/duckless
/plugin install duckless@duckless

Layout

Hexagonal, kept light: a pure core, ports, adapters, and one wiring point.

Dependency rule: core imports nothing else from DuckLess, service only core and ports, and only wiring imports adapters.

Contributing

Issues and pull requests are welcome: bug reports, real use cases, docs, code. See CONTRIBUTING.md for the setup, the conventions and how changes are tested.

uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .

License

DuckLess is open source under the Apache License 2.0. You can use it, modify it and ship it, commercially or not; keep the license and the notices.