XQDB — Python Bindings

XQDB is independent and not affiliated with or endorsed by KX. kdb+ is a trademark of KX.

A Python interface to kdb+/q powered by Narwhals, with support for multiple dataframe backends (PyArrow, pandas, Polars).

Installation

Requirements: Python ≥ 3.10, Narwhals ≥ 2.10, PyArrow ≥ 20.0.0

Optional backend packages: pandas, polars

Install the published package:

python -m pip install xqdb

To build from source for development with a Rust toolchain:

python -m pip install -e .

Quick Start

import xqdb
import narwhals as nw

# Basic connection (PyArrow backend by default)
conn = xqdb.Q('localhost', 1800)

# Select an installed Narwhals output backend
conn = xqdb.Q('localhost', 1800, backend='polars')

# Authentication credentials require TLS unless the connection is already protected
conn = xqdb.Q(
    'localhost', 1800, user='user', passwd='password', enable_tls=True
)

# With TLS and retry
conn = xqdb.Q('localhost', 1800, enable_tls=True, retries=3, timeout=30)

Connection Parameters

q IPC authentication sends credentials in cleartext when TLS is disabled. Enable TLS for credentialed connections unless another trusted transport already protects the socket.

Narwhals DataFrames and Backend Selection

Results from conn.sync() and conn.receive() return Narwhals DataFrames or Series backed by the selected backend. The default is PyArrow; pandas and Polars are supported when their optional packages are installed. An unavailable or unknown backend raises an error—XQDB does not silently substitute another backend.

To extract the underlying native DataFrame:

result = conn.sync("select from trade")  # Narwhals DataFrame
native_df = nw.to_native(result)  # PyArrow, pandas, or Polars Table/DataFrame

Input Constraints

• Native or Narwhals eager DataFrames and Series are accepted.
• Lazy frames are rejected rather than collected implicitly.
• The Python/native boundary uses the Arrow C Stream interface; it does not serialize frames to Arrow IPC bytes.

Temporal range and precision

Python datetime, time, and timedelta values have microsecond precision. q timestamp or timespan atoms with non-zero sub-microsecond nanoseconds raise ValueError instead of being truncated. q date or datetime atoms outside Python's representable range raise OverflowError instead of being clamped.

Connect / Disconnect

# explicitly connect (auto-connects on first query)
conn.connect()

# disconnect (auto-disconnects on IO error)
conn.disconnect()

String Query

conn.sync("select from trade where date=last date")

Functional Query

Supports Python basic data types, Narwhals Series/DataFrame, and dict (with string keys).

from datetime import date, time

import pyarrow as pa

symbols = pa.chunked_array(
    [pa.array(["sym0", "sym1"]).dictionary_encode()]
)
conn.sync(
    ".gw.query",
    "table",
    {
        "date": date(2023, 11, 21),
        "syms": symbols,
        "startTime": time(9),
        "endTime": time(11, 30),
    },
)

Operators and Lambdas

Pass q primitives and arbitrary lambdas as first-class arguments:

from xqdb import XqdbQLambda, XqdbQOperator

conn.sync("{[op;a;b] .[op;(a;b)]}", XqdbQOperator.PLUS, 1, 2)
conn.sync("{[op;a;b] .[op;(a;b)]}", XqdbQLambda("{x+y}"), 1, 2)

# A non-root q context can be supplied explicitly.
scoped = XqdbQLambda("{x+y}", "analytics")

XqdbQOperator(name) accepts supported q primitive names such as "+"; it does not expose wire opcodes. XqdbQLambda(source, context="") preserves its source text, requires a brace-delimited UTF-8 body (optionally prefixed with k)), rejects NUL bytes in both fields, and rejects context values beginning with ".". The context "analytics" represents q namespace .analytics because the wire context omits the leading dot. Lambda source is executable q code: construct it only from trusted input.

Send DataFrame

import pyarrow as pa

frame = pa.table({"sym": ["a", "b"], "price": [10.5, 11.0]})
conn.sync("upsert", "table", frame)

Async Query

conn.asyn("upsert", "table", frame)

Subscribe

import pyarrow as pa

tables = pa.chunked_array(
    [pa.array(["table1", "table2"]).dictionary_encode()]
)
symbols = pa.chunked_array(
    [pa.array(["sym1", "sym2"]).dictionary_encode()]
)
conn.sync(".u.sub", tables, "")
conn.sync(".u.sub", tables, symbols)

while True:
    # returns ("upd", "table", Narwhals DataFrame)
    upd = conn.receive()
    print(upd)

Generate IPC Bytes

Serialize data as kdb+ IPC bytes without a connection.

import pyarrow as pa

from xqdb import serialize_as_ipc_bytes6

frame = pa.table({"sym": ["a", "b"], "price": [10.5, 11.0]})

# without compression
buffer = serialize_as_ipc_bytes6("sync", False, ["upd", "table", frame])

# with compression
buffer = serialize_as_ipc_bytes6("sync", True, ["upd", "table", frame])

msg_type: "async" | "sync" | "response"

Read Binary Table

Read a regular q binary table file directly into a Narwhals DataFrame. Select the native output backend independently of the file format.

from xqdb import read_binary6

df = read_binary6("/path/to/table.bin", backend="pandas")

Error Handling

from xqdb import XqdbError, XqdbIOError, XqdbAuthError

try:
    conn.sync("select from trade")
except XqdbAuthError:
    print("Authentication failed")
except XqdbIOError:
    print("Connection error")
except XqdbError:
    print("General xqdb error")

Data Type Mapping

Deserialization (q → Python)

q scalars map to Python scalars; q vectors/tables are returned as Narwhals DataFrames/Series backed by the selected backend.

Atom (scalar to Python)

Vector and Table (Arrow-backed Narwhals)

Other selected backends receive the equivalent representation that Narwhals can construct from this Arrow stream. Backend-specific dtypes may differ while values and q semantics remain the same.

real/float 0n is mapped to null, not NaN.

short/int/long null and infinity values (0Nh/i/j, 0Wh/i/j, -0Wh/i/j) are mapped to null.

Serialization (Python → q)

Basic Data Type

Series, DataFrame, and Dictionary

Dictionary serialization requires str keys.

Resources

• Narwhals Documentation — Unified dataframe interface
• PyArrow Documentation — Default backend
• Pandas Documentation — Optional backend
• Polars Documentation — Optional backend

License

XQDB is licensed under the BSD-3-Clause permissive open-source license, which permits use in proprietary and commercial applications.