This release is a pre-release and may not be stable for production use.
einfold
Array algorithms for tensor computations written in SQL.
Matrix products, tensor contractions and attention layers can all be written in SQL, as joins and aggregates. A matrix product, for one:
SELECT a.i, b.k, SUM(a.v * b.v) AS v
FROM a JOIN b ON a.j = b.j
GROUP BY a.i, b.k
A SQL engine runs that as SQL: it builds every row of the join, then hashes each into its group. That's correct, but slow, and for attention it needs memory that grows with the square of the sequence length. einfold finds these computations in a query's plan and runs them with array algorithms instead:
- dense, positional kernels when the keys are small integers;
- sorting-based kernels for any other keys;
- a fused, tiled kernel for attention, that never holds more than a tile of its scores.
The rows are the same as the engine's, up to the rounding of floating-point sums. NULLs, NaN and infinities follow SQL's semantics exactly.
Use
einfold plugs into a SQL engine. Today that's Apache DataFusion's Python bindings:
python -m pip install --upgrade --pre "einfold[datafusion]"
einfold is in pre-release: --pre installs the latest release candidate, and --upgrade replaces an older one.
from datafusion import SessionContext
from einfold.datafusion import EinFold
einfold = EinFold()
ctx = SessionContext().with_extensions(einfold)
ctx.register_parquet("a", "a.parquet")
ctx.register_parquet("b", "b.parquet")
table = ctx.sql("SELECT a.i, b.k, SUM(a.v * b.v) AS v FROM a JOIN b ON a.j = b.j GROUP BY a.i, b.k").to_arrow_table()
einfold.plans_with_einfold() # 1: einfold ran the matrix product
Every query in that session is planned by einfold from then on. What it doesn't recognize runs as DataFusion would run it, planned with DataFusion's default physical planner and optimizer rules. A query planner or physical optimizer rule added to the session before einfold isn't applied.
Limits
- Very deep plans. A session hands einfold each plan through DataFusion's FFI, which decodes it recursively on the thread that asks. On Linux, einfold raises the main thread's stack limit, so plans of about 2,000 joins work there. On another thread, or on macOS, a plan of a few hundred joins can overflow the thread's stack, and the process exits rather than raising an error.
- Functions named
sumandmax. einfold replaces them with DataFusion's own, which it recognizes in plans. In a session with einfold, a function you register under either name isn't used.
What it finds
- Sums of products over a join, grouped by columns of either side: matrix products, contractions, weighted sums and regridding with a weight table. Also the same left-joined from a table, so that every row of the table gets its sum.
- A weighted sum of such a sum, fused with it. One query computes scores, a second sums values weighted by a function of them, and einfold runs both tile by tile, never holding all the scores at once. Attention is one: weights
exp(s - MAX(s))normalized by their sum, written with a window (OVER (PARTITION BY i)) or withGROUP BYjoined back. The weights can be an expression of the score built from+ - * /, negation,abs,exp,lnandpowerwith a positive constant exponent, such as a sigmoid's1 / (1 + exp(-s))or a Cauchy kernel's1 / (1 + s * s), normalized or not, so a plain chain of two matrix products fuses too. - Scores that are an expression of two rows' values, not only a sum of products: a Gaussian kernel's
exp(-power(x - y, 2) / h)over a cross join, for kernel smoothing. - Masks that compare keys, skipped tile by tile: causal (
q.t >= k.t), a sliding window (q.t >= k.t AND q.t - k.t < 128, or withBETWEEN), a band either side (abs(q.t - k.t) <= 64), and the same on a sum of products by itself.
How fast
From Python, on an Intel i7-8700 (6 cores), DataFusion 55, with and without einfold:
| query | DataFusion | with einfold |
|---|---|---|
| causal self-attention in SQL, 1,024 tokens of 64 features | 2.07 s | 17.5 ms |
| the same, 4,096 tokens | 109 s | 46 ms |
a matrix product, 50,000 × 16 by 16 × 8 |
37.9 ms | 20.9 ms |
SELECT 1, no tensor computation at all |
0.23 ms | 1.69 ms |
The last row is a fixed cost per query. The extension planner hands each plan across DataFusion's FFI, and that costs about 1.5–3 ms per query, which matters for very short queries.
Engines
- DataFusion's Python bindings:
einfold.datafusion, now. - DuckDB: planned.
einfold's algorithms work on Apache Arrow data and depend on no engine, so each engine needs only a way to hand einfold the parts of its plans.
Projects that use einfold
- xarray-sql queries xarray Datasets, such as climate and weather data, with SQL. Its
Contextis a DataFusion session, sowith_extensions(EinFold())works there too. - ddx differentiates SQL queries, so that models written in SQL can be trained. Its gradients are sums of products over joins, which einfold runs.
License
Apache-2.0. The source is at github.com/xqlsystems/einfold.
Metadata
Release files for einfold 0.1.0rc2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| einfold-0.1.0rc2.tar.gz | 224.3 kB | With provenance | Details for einfold-0.1.0rc2.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_x86_64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ x86-64 | Details for einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_x86_64.whl |
| einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_aarch64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ ARM64 | Details for einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_aarch64.whl |
| einfold-0.1.0rc2-cp310-abi3-macosx_11_0_arm64.whl | CPython 3.10 | abi3 | macOS 11.0+ ARM64 | Details for einfold-0.1.0rc2-cp310-abi3-macosx_11_0_arm64.whl |
| einfold-0.1.0rc2-cp310-abi3-macosx_10_12_x86_64.whl | CPython 3.10 | abi3 | macOS 10.12+ x86-64 | Details for einfold-0.1.0rc2-cp310-abi3-macosx_10_12_x86_64.whl |
Total release size: 182.3 MB
Release files / einfold-0.1.0rc2.tar.gz
| Download URL | einfold-0.1.0rc2.tar.gz |
|---|---|
| Size | 224.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
82df6a9e2e1954cc3ba90982b61a1236b0451e84a5f80ab4e7707c0e46f7950e
|
|
BLAKE2b-256 checksum How to use checksums |
76e9b534c6660e42ae62f3bd430611b299f406833cd82757e39f74d0161290e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logRelease files / einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 47.3 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
98cc8014c4e1f26101ecd492e9881ff3bafdddcf130692fa97a600a3f8047d6d
|
|
BLAKE2b-256 checksum How to use checksums |
86638dc06cb2ffc0e8bc982b83d1ffd60b9bba30f7a5990c0f11254f9faa5825
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logRelease files / einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | einfold-0.1.0rc2-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 48.7 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
55d10e500ef934322e1108fa24b85da37ce21b152a9b3dd0364bf604e93fb0e0
|
|
BLAKE2b-256 checksum How to use checksums |
d2bf9d73c11836034eceea3057e246e89beadc0b090b11c4bb88865e357102af
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logRelease files / einfold-0.1.0rc2-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | einfold-0.1.0rc2-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 42.2 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
1041aebdd7293dac48b680addfebb6bd90870e220a56263fe7fb43da4f3b1217
|
|
BLAKE2b-256 checksum How to use checksums |
46563045c7efff5a649f8f209d95175785d7ddb7a279925f2033627a62df0645
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logRelease files / einfold-0.1.0rc2-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | einfold-0.1.0rc2-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 43.9 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
907bd43840bd3f0006436705a1f9a62939c2e8410e10911753c72975ce8edba0
|
|
BLAKE2b-256 checksum How to use checksums |
9a19986bd602078909cfa9bba3133d6f9cf5372c0e9f49de891aeeb657334187
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logSecurity for einfold 0.1.0rc2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 5 files in this release have been uploaded with provenance metadata.
Attestation for einfold-0.1.0rc2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.