Skip to main content

ddxdb

Calculus in SQL: the derivative of a query with respect to a table's columns, as JAX takes the derivative of a function, computed by the engine.

from ddxdb import Context

ctx = Context()
ctx.sql("CREATE TABLE w (i BIGINT, val DOUBLE) AS VALUES (0, 1.0), (1, 2.0)")
ctx.sql("""
WITH f AS (SELECT SUM(val * val) AS l FROM w)
SELECT i, val, grad(f, val) AS df_dval FROM w     -- 2·val, row by row
""")

In a SELECT over a table, grad(f, val) is, at each row, the derivative of the number the CTE f computes with respect to that row's val (w.val when the SELECT reads several tables). The query is ordinary SQL; nothing in it is labelled for ddx, and the one function ddx gives a meaning to is ddx_stop_gradient(x), JAX's lax.stop_gradient.

A gradient is what an optimizer steps along, so a query scoring anything (a model's loss, a curve's misfit, an energy) can be optimized in SQL: a step, params - lr * grad(f)(params), is one expression.

jvp and vjp in SQL

Forward mode is jvp(e, table.column, tangent), a column of the SELECT that computes e: its tangent at each row along tangent, a CTE or table shaped like table (its dims, then the column's tangent), or a number, the same at every row, as jax.jvp returns it beside the value. Along 1, it is the derivative of an expression row by row: SELECT x, jvp(sin(x), t.x, 1) AS dx FROM t is cos(x). Along a table:

ctx.sql("""
WITH h AS (
  SELECT x.sample, w.out, tanh(SUM(x.val * w.val)) AS val
  FROM x JOIN w ON x.inp = w.inp GROUP BY x.sample, w.out),
dw AS (SELECT inp, out, 0.01 AS val FROM w)
SELECT sample, out, val, jvp(val, w.val, dw) AS val_tangent FROM h
""")

One SELECT's jvp columns share a tangent; tangents along two tables are two SELECTs, and add. vjp(f, column, cotangent) is the reverse: in a SELECT over a table, the pull-back to each row of a cotangent shaped like the CTE f's output. Calls nest: f, e's SELECT, or a tangent can read another call, so a jvp of a CTE of gradients is a Hessian-vector product, in SQL.

This is the Python distribution of ddx, a thin wrapper over the ddx-ad engine.

Install

pip install ddxdb                  # the programs, for any engine
pip install "ddxdb[datafusion]"    # + Context and ddxdb.ad, on DataFusion
pip install "ddxdb[duckdb]"        # + ddxdb.duckdb, on DuckDB

Context, for DataFusion

A real SessionContext subclass whose .sql() runs grad first: every inherited method, property and constructor argument works unchanged. It lives in ddxdb.datafusion (a subclass needs its base class at import time) and is re-exported as ddxdb.Context, imported on first use, so import ddxdb needs no engine.

On a plain SessionContext, ddxdb.ad.sql(ctx, statement) does the same, and ddxdb.ad.sql_all runs several statements that take grad of one loss for the price of one backward pass.

A training loop runs the same statements on every batch. ddxdb.ad.prepare prepares them once, as jax.jit traces a function once:

step = ad.prepare(ctx, [step_w, step_b])
for batch in batches:
    ctx.deregister_table("batch")
    ctx.register_record_batches("batch", [batch])
    w, b = step()                       # what sql_all would return

Its first call computes the calls as sql_all does. Later calls run the same plans again, asking DataFusion for no plan of a query, so they skip what a call of sql_all repeats: about a quarter of a small GPT's training step. A table may change its rows, or be registered again, between calls. A call after a table's schema, a view or a setting changes prepares the statements again.

On DuckDB

ddxdb.duckdb.sql(con, statement) is the same on a DuckDB connection, and ddxdb.ad.sql accepts one too:

import duckdb
from ddxdb import duckdb as ad

con = duckdb.connect()
con.register("w", arrow_table)          # or a table, a view, a temp table
ad.sql(con, """
WITH loss AS (SELECT SUM(val * val) AS l FROM w)
SELECT i, val - 0.1 * grad(loss, val) AS val FROM w
""")

Programs, and how they compose

ddxdb.ad.grad, vjp and jvp give the programs underneath, and each takes SQL or a program, and the columns to differentiate with respect to (wrt): a column as "table.column", or a list of them, which one backward pass differentiates together. They compose: ad.jvp(ctx, ad.grad(ctx, loss, wrt), wrt) gives Hessian-vector products, checked against jax.jvp(jax.grad(f)). Both kinds of program (BackwardProgram, ForwardProgram) list the tables you register before they run (inputs, as InputTables) and the tables they leave (value and gradients, as OutputTables). ad.run(ctx, program, inputs) runs one on its inputs (a cotangent or tangents, as DataFrames or Arrow tables, in the order of program.inputs) and returns what it computed, value and gradients, as DataFrames, tangents named {column}_tangent; nothing stays on the context:

hvp = ad.jvp(ctx, ad.grad(ctx, "SELECT SUM(val * val * val) AS l FROM w", "w.val"), "w.val")
out = ad.run(ctx, hvp, [ctx.sql("SELECT i, 1.0 AS val FROM w")])
out.gradients[0]   # i, val (the gradient), val_tangent (H·v)

Other engines

ddxdb.grad_plan(plan_bytes, wrt) differentiates the serialized Substrait plan an engine's producer writes, and ddxdb.run(backend, program) runs the result on any object with four methods, select_all, returns_rows, materialize and drop_table (the ddxdb.Backend protocol); vjp_plan and jvp_plan are the same for the other two. They are ddx-ad's own Rust, the same code the Rust adapter runs. Pass namespace="__ddx_mine_" to get the same program from the same plan every time.

ddxdb.to_sql(plan_bytes, dialect) writes a plan, its reads bound, as one SQL query in "duckdb" or "datafusion": for a backend that runs a program's steps as SQL rather than through its engine's Substrait consumer, as ddxdb's DuckDB backend does. On DataFusion, a backend of one's own can call ddxdb.ad.dataframe(ctx, plan_bytes) on each plan ddxdb.run hands it, to get a DataFrame the way ddxdb.ad.run does: as SQL when its column names would grow too large for the Substrait consumer, and through the consumer otherwise.

The SQL calls run on another engine too: ddxdb.rewrite(backend, statements) runs the programs the statements' grad, vjp and jvp calls need on a backend that also has plan_sql (SQL to a serialized Substrait plan), and returns the statements rewritten to read what they computed. Run them on the engine inside the with block; the tables they read are dropped when it ends:

with ddxdb.rewrite(backend, [step_w, step_b]) as rewritten:
    for sql in rewritten.statements:
        engine.sql(sql)

In a training loop, pass the same ddxdb.Programs() each step (ddxdb.rewrite(backend, statements, programs=programs)), and each program is built once and run again on later steps. A program is kept under the backend's plan_key(sql), a text that changes whenever the engine's plan of the query could (its tables' schemas, its views' definitions), so a backend without that method gets nothing kept. ddxdb.ad.sql and Context.sql keep programs for each context this way on their own. ddxdb.prepare(backend, statements) prepares them instead, as ddxdb.ad.prepare does: its run() returns a Rewritten like ddxdb.rewrite's.

What you can write

Inside the loss: + - * /; the chain rule for the trig / inverse-trig / exp / log / hyperbolic set plus abs and erf; power with a constant base or exponent; joins, GROUP BY, SUM, AVG, MAX, MIN, CASE, greatest and least.

ddx_stop_gradient(x) is x, held constant by differentiation, as JAX's lax.stop_gradient is. Context registers it, and so does ddxdb.duckdb, on the connection it's given, unless that already has one. On a plain SessionContext, call ddxdb.ad.register_stop_gradient(ctx). On DataFusion it keeps its argument's type. On DuckDB it takes DOUBLE, so a forward value computed through it on a REAL column is DOUBLE, though the gradient stays REAL.

A loss may call a function you register, such as erf, which ddx differentiates and neither engine defines. On DataFusion, register it with ctx.register_udf. On DuckDB, use ddxdb.duckdb.create_function(con, name, function, parameters, return_type, **kwargs), which takes con.create_function's arguments. ddx plans on a connection of its own, so it can't see a function registered with con.create_function alone. For a cursor to plan with those functions too, make it with ddxdb.duckdb.cursor(con).

import math
from ddxdb import duckdb as ad

ad.create_function(con, "erf", math.erf, ["DOUBLE"], "DOUBLE", side_effects=False)
ddxdb.supported_functions()               # ['abs', 'acos', 'asin', ...]

The unary functions ddx has a rule for, read from the engine rather than restated.

Errors are typed

An unsupported construct is always an error, never a silently wrong number — this is a numerical-correctness library, and a plausible-looking wrong derivative is the worst thing it could produce. The kind of failure is a class, so you can catch the one you can act on. All of them derive from ddxdb.DdxError:

  • UnsupportedExpression: no rule for something in the query.
  • NotScalar: grad of a query that is not a loss (one row, one column).
  • UnknownColumn: a column the loss does not read.
  • InvalidColumn: a column that cannot be differentiated (not a float, or in a table whose rows do not have unique dims).

Development

pip install maturin pytest
maturin develop --uv
python -m pytest tests/

Building compiles protoc from source for the substrait crate (it needs cmake and a C++ compiler), which takes a few minutes the first time.

License

Licensed under Apache-2.0, the same as the rest of ddx.

Metadata

Release files for ddxdb 0.4.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ddxdb 0.4.12
File Size Uploaded Provenance Actions
ddxdb-0.4.12.tar.gz 440.8 kB With provenance Details for ddxdb-0.4.12.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for ddxdb 0.4.12
File Actions
ddxdb-0.4.12-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for ddxdb-0.4.12-cp310-abi3-win_amd64.whl
ddxdb-0.4.12-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for ddxdb-0.4.12-cp310-abi3-manylinux_2_28_x86_64.whl
ddxdb-0.4.12-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for ddxdb-0.4.12-cp310-abi3-manylinux_2_28_aarch64.whl
ddxdb-0.4.12-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for ddxdb-0.4.12-cp310-abi3-macosx_11_0_arm64.whl
ddxdb-0.4.12-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for ddxdb-0.4.12-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 46.6 MB

Release files / ddxdb-0.4.12.tar.gz

Download URL ddxdb-0.4.12.tar.gz
Size 440.8 kB
Tags Source
SHA-256 checksum
How to use checksums
46e69f73e7f1d720a5472892048401a720a63c635827b8138268d6d39c2dda56
BLAKE2b-256 checksum
How to use checksums
8a76d1c1097d00246faa1c1354276ed13c34f3d77fde77d3df17e869a46ac949
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / ddxdb-0.4.12-cp310-abi3-win_amd64.whl

Download URL ddxdb-0.4.12-cp310-abi3-win_amd64.whl
Size 9.3 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
21bb88dbc57359c4adc890967b38f8ed2c45632818c9eaf55b578aca5699ae42
BLAKE2b-256 checksum
How to use checksums
9af74fc901e45502eb64d8ca3e3bc84f138528d20ae0d50fb69292e75898ceaf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / ddxdb-0.4.12-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL ddxdb-0.4.12-cp310-abi3-manylinux_2_28_x86_64.whl
Size 10.6 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
c14e5ed0fd2e1b541c35fb5568b4fc5417ae8d62ba68a1a4ca87b8faf6e58ada
BLAKE2b-256 checksum
How to use checksums
3fcc40d9ed774a4d771872307c7e6a81e1d41d6de96c7ea15195af640c96fbe6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / ddxdb-0.4.12-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL ddxdb-0.4.12-cp310-abi3-manylinux_2_28_aarch64.whl
Size 8.8 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
7f6d42fbfe35ca0cdeebd442c4c397927820794f2ce40d2deab3db51180b165a
BLAKE2b-256 checksum
How to use checksums
862faaefebd674db57a02fb5cc122c2e737550277cc695d82baa0acfb2537506
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / ddxdb-0.4.12-cp310-abi3-macosx_11_0_arm64.whl

Download URL ddxdb-0.4.12-cp310-abi3-macosx_11_0_arm64.whl
Size 8.6 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
661766c0fc60366c4f6ba32a272314c05856204d18fc65c39d68cac8758fbce1
BLAKE2b-256 checksum
How to use checksums
fb2f71f23de1bb7103459426d69cdfc1f2a9c70825bd204876c877830115f81c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / ddxdb-0.4.12-cp310-abi3-macosx_10_12_x86_64.whl

Download URL ddxdb-0.4.12-cp310-abi3-macosx_10_12_x86_64.whl
Size 8.8 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
caa7b3993627dc0291d123b0b4a9a1111007fe7eaaecc2a38c982976eec75abf
BLAKE2b-256 checksum
How to use checksums
287df7dd2cc6a494e0914f9b165c09e46c921e55b463fb839817e159488db0a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for ddxdb 0.4.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for ddxdb-0.4.12.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page