Skip to main content

Xorq is three interlinked things, plus the interfaces built on them.

  1. An IR. An agent-native intermediate representation for cross-engine, composable, cacheable, declarative execution. A pipeline is an expression that compiles to many engines, caches at any node, and composes with other expressions.
  2. A ledger. The IR makes every pipeline a durable artifact that is cheap to verify: one computation, defined before it runs, named by a hash of its definition, with human-interpretable lineage alongside. The ledger steers agent work. An agent selects a reviewed entry instead of re-deriving it from source files.
  3. A CLI. It governs execution against the ledger: build, add, run, check, pin, rebase, compose. Arbitrary compute enters the deterministic system the ledger builds through UDXFs, Python functions that exchange Arrow tables over Flight.

Interfaces on top serve different kinds of users. The CLI doubles as the headless mode for agents, and xorq catalog tui is the view for humans.

Below, the ten-entry ledger of an insurance pricing pipeline from A Commit vs. a Ledger Entry. The selected entry is the scored test set: its lineage runs through three fitted models down to the two source reads, next to its schema and a git log with one commit per change.

xorq catalog TUI: ten entries on the left; the lineage of the scored test set down to the two source reads, its schema, and the git log with one commit per change

Why a ledger steers agents

On DABStep, 450 data-analysis questions over payment transaction data, a Xorq ledger of 33 named expressions takes Haiku from 50% to 84%, 9 points above the Sonnet baseline (Orientation Over Reasoning).

DABStep accuracy: Haiku 4.5 50%, Sonnet 4.6 75%, Haiku 4.5 + Semantic Catalog 84%

One metric question, 100 runs per configuration (Faithful, Reproducible, Wrong). Two of the post's nine configurations, both Haiku running with Xorq, differing only in whether a reviewed definition of the metric was preloaded:

Re-derived from source files Selected from a reviewed entry
Correct 0 of 100 100 of 100
Median turns 21 4
Cost per run $0.09 $0.02

Getting started

❯ pip install "xorq[examples]"
❯ xorq init -t penguins && cd xorq-template
❯ xorq uv build expr.py
builds/4c915297a711
❯ xorq catalog init
Initialized catalog at ~/.local/share/xorq/catalogs/default
❯ xorq catalog add builds/4c915297a711/ -a penguins-predict
Added 4c915297a711
❯ xorq catalog run penguins-predict -o out.parquet

Build turns an expression into a directory named by its hash, holding the manifest, the project wheel, and pinned requirements. Add puts that directory in the ledger under its hash and an alias. Run executes the entry in the environment it pins. The template's build needs Python 3.11 or newer; if uv picks a 3.10 interpreter, set UV_PYTHON_PREFERENCE=only-managed.

The IR

Xorq expressions support caching, multi-engine execution, and UDFs. xo._ is the row reference, so xo._.species is the species column of the current table.

import xorq.api as xo

penguins = xo.examples.penguins.fetch()
expr = (
    penguins
    .filter(xo._.species.notnull())
    .group_by("species")
    .agg(avg_bill_length=xo._.bill_length_mm.mean())
)

.cache(xo.ParquetCache.from_kwargs()) materializes any node. .into_backend(xo.duckdb.connect()) moves its data to another engine. The same expression runs on DataFusion, DuckDB, SQLite, pandas, Snowflake, Databricks, BigQuery, Trino, Postgres, PyIceberg, and GizmoSQL. Operators exchange Arrow RecordBatches, with no DAG of tasks to checkpoint. xo.Pipeline.from_instance turns a scikit-learn pipeline into a deferred expression.

The ledger

The ledger is a git repo on the filesystem, which the CLI calls a catalog. Aliases are symlinks, entries are zipped builds, and metadata sidecars are plain YAML, so an agent that clones it can discover everything with file operations. The hashes below come from one build and will not match yours:

❯ tree git-catalogs/penguins
git-catalogs/penguins
├── aliases
│   └── penguins-agg.zip -> ../entries/b45d9db8189a.zip
├── catalog.yaml
├── entries
│   └── b45d9db8189a.zip
└── metadata
    └── b45d9db8189a.zip.metadata.yaml

❯ git -C git-catalogs/penguins log --format=%s
add: b45d9db8189a (aliases penguins-agg)
add catalog.yaml
initial commit

An entry is a manifest plus the environment to run it. The manifest is the content-addressed specification of the pipeline: one node per operation, each referenced by hash, with schema, lineage, and inputs read straight off it. The wheel and pinned requirements.txt next to it make the entry portable to another sandbox, your machine, or production.

Abridged expr.yaml for the expression above
definitions:
  nodes:
    '@read_dbd8d81198f1':
      op: Read
      method_name: read_parquet
      name: penguins
      schema_ref: schema_11039d461f5b

    '@filter_f7eedc0c6f85':
      op: Filter
      parent: { node_ref: '@read_dbd8d81198f1' }
      predicates:
        - { op: NotNull, arg: { op: Field, name: species, ... } }

    '@aggregate_b45d9db8189a':
      op: Aggregate
      parent: { node_ref: '@filter_f7eedc0c6f85' }
      by:
        species: { op: Field, name: species, ... }
      metrics:
        avg_bill_length: { op: Mean, arg: { op: Field, name: bill_length_mm, ... } }

expression:
  node_ref: '@aggregate_b45d9db8189a'
  schema_ref: { schema_ref: schema_cdf21e46eb8a }

The CLI

Verbs that act on an entry take its hash or an alias such as prod. The ones that produce a pipeline (pin, rebase, compose) write it back as a new entry; entries are never edited in place, though aliases can move. Writes push the catalog to its remotes by default.

❯ xorq catalog run prod -o out.parquet          # execute in the entry's pinned environment
❯ xorq catalog run-cached prod -o out.parquet   # run, wrapping the result in a parquet cache
❯ xorq catalog check                            # entries, aliases and metadata agree with disk
❯ xorq catalog check-sources prod               # did a source schema drift? reports, never repairs
❯ xorq catalog rebase prod                      # rebuild over drifted source schemas as a new entry
❯ xorq catalog pin prod -e --alias prod-pinned  # materialize and freeze the caches as a new entry
❯ xorq catalog compose src transform -a pipeline  # source entry + unbound transform entry, as one
❯ xorq serve-flight-udxf builds/<hash>          # serve a build's UDXF nodes over Arrow Flight

A UDXF is how arbitrary compute joins the system: a Python function that takes an Arrow table and returns one, run in its own process behind Flight, named and cataloged like any other node. See User-Defined Exchange Functions.

Learn more


Pre-1.0. Expect breaking changes with migration guides.

Metadata

Release files for xorq 0.4.11

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for xorq 0.4.11
File Size Uploaded Provenance Actions
xorq-0.4.11.tar.gz 2.3 MB Without provenance Details for xorq-0.4.11.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for xorq 0.4.11
File Interpreter ABI Platform Actions
xorq-0.4.11-py3-none-any.whl Python 3 none any Details for xorq-0.4.11-py3-none-any.whl

Total release size: 4.5 MB

Release files / xorq-0.4.11.tar.gz

Download URL xorq-0.4.11.tar.gz
Size 2.3 MB
Tags Source
SHA-256 checksum
How to use checksums
0175a224074e1b268cb3aa7b40123890f3986259ae701870d09bebf4183504ec
BLAKE2b-256 checksum
How to use checksums
90d1a43d8edbea5044373a7223c66bbc511aa004e5bbfbbc6528e1dda42f804a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / xorq-0.4.11-py3-none-any.whl

Download URL xorq-0.4.11-py3-none-any.whl
Size 2.2 MB
Tags Python 3
SHA-256 checksum
How to use checksums
ce816ec2575911369a437f27adf2d66a329916d4c73024ca8d92c53541193095
BLAKE2b-256 checksum
How to use checksums
37952e8a671b2910bcb004765c51aec9e66b0126832d0ef6bebb0702ecc16d89
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for xorq 0.4.11

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.11 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page