XQDB — Python Bindings

XQDB is independent and not affiliated with or endorsed by KX. kdb+ is a trademark of KX.

A Python interface to kdb+/q powered by Narwhals, with support for multiple dataframe backends (PyArrow, pandas, Polars).

Upgrading to 0.1.4

Breaking change. q timestamp atoms are now naive datetime values with tzinfo is None. Version 0.1.3 returned UTC-aware values.

A q timestamp carries no timezone, and Arrow timestamp[ns] columns were already naive, so in 0.1.3 the same q value compared unequal depending on whether it was read as an atom or from a column, and a value read from a column could not be passed back as a query argument. Both now work.

If you need an aware value, attach the zone yourself. Do not assume UTC unless your q process guarantees it — a process using .z.P stores local wall-clock times, and 0.1.3 labelled those UTC.

# 0.1.3
ts = conn.sync("first exec time from trade")   # datetime(..., tzinfo=timezone.utc)

# 0.1.4
ts = conn.sync("first exec time from trade")   # datetime(...)  naive
ts = ts.replace(tzinfo=timezone.utc)           # only if the data really is UTC

Arguments now accept both shapes: a naive datetime keeps its wall clock, and an aware one normalizes through its UTC offset, including zoneinfo zones across DST boundaries. A tzinfo whose utcoffset() returns None counts as naive, per Python's own definition.

Installation

Requirements: Python ≥ 3.10 and < 3.15, Narwhals ≥ 2.10, PyArrow ≥ 20.0.0

Optional backend packages: pandas, polars

Install the published package:

python -m pip install xqdb

To build from source for development with a Rust toolchain:

python -m pip install -e .

Quick Start

import xqdb
import narwhals as nw

# Basic connection (PyArrow backend by default)
conn = xqdb.Q('localhost', 1800)

# Select an installed Narwhals output backend
conn = xqdb.Q('localhost', 1800, backend='polars')

# Authentication credentials require TLS unless the connection is already protected
conn = xqdb.Q(
    'localhost', 1800, user='user', passwd='password', enable_tls=True
)

# With TLS and retry
conn = xqdb.Q('localhost', 1800, enable_tls=True, retries=3, timeout=30)

Connection Parameters

q IPC authentication sends credentials in cleartext when TLS is disabled. Enable TLS for credentialed connections unless another trusted transport already protects the socket.

q stores symbols and strings as raw bytes and never validates them, while Arrow string columns must be valid UTF-8. With the default symbol_encoding="strict", a result carrying a stray Latin-1 or binary byte in a symbol, symbol column, string column, char column, or lambda fails as an XqdbError naming the offending value. Opt in to symbol_encoding="lossy" — at construction or later through conn.symbol_encoding = "lossy" — to decode such text with each maximal invalid sequence replaced by "\ufffd", so every other value in the result survives intact. Valid text decodes identically under both policies, and q error messages always surface with replacement characters rather than being hidden behind a decoding failure.

Narwhals DataFrames and Backend Selection

Results from conn.sync() and conn.receive() return Narwhals DataFrames or Series backed by the selected backend. The default is PyArrow; pandas and Polars are supported when their optional packages are installed. An unavailable or unknown backend raises an error—XQDB does not silently substitute another backend.

To extract the underlying native DataFrame:

result = conn.sync("select from trade")  # Narwhals DataFrame
native_df = nw.to_native(result)  # PyArrow, pandas, or Polars Table/DataFrame

Input Constraints

• Native or Narwhals eager DataFrames and Series are accepted.
• Lazy frames are rejected rather than collected implicitly.
• The Python/native boundary uses the Arrow C Stream interface; it does not serialize frames to Arrow IPC bytes.

Temporal range and precision

Python datetime, time, and timedelta values have microsecond precision. q timestamp or timespan atoms with non-zero sub-microsecond nanoseconds raise ValueError instead of being truncated. q date or datetime atoms outside Python's representable range raise OverflowError instead of being clamped.

A q timestamp carries no timezone, so it maps to a naive datetime. Timestamp atoms and Arrow timestamp[ns] columns therefore share the same timezone semantics, and for any value Python's datetime can represent they compare equal. XQDB does not label q values UTC, because a q process may hold local wall-clock times; apply your own zone when you know it.

Query arguments accept both shapes. A naive datetime is used as the q wall clock unchanged. An aware datetime is resolved to UTC by Python, so fixed offsets, zoneinfo zones, and other tzinfo implementations all normalize to the correct instant, including across DST boundaries.

Whole Series and DataFrame round-trips are always nanosecond-exact on every backend, because they cross the boundary as Arrow rather than as Python objects. Single scalars pulled out of a frame are backend-dependent, and the precision is decided by the backend before XQDB sees the value:

pandas.Timestamp keeps its sub-microsecond digits in .nanosecond rather than .microsecond, and XQDB reads that remainder so the full nanosecond reaches q. Polars materializes a plain datetime, so a nanosecond read out as a Polars scalar is already truncated and XQDB cannot recover it. When nanosecond fidelity matters, pass the frame or Series instead of a scalar.

Reading a sub-microsecond value back as an atom still raises ValueError, because datetime cannot represent it — select it as a one-row table instead.

Connect / Disconnect

# explicitly connect (auto-connects on first query)
conn.connect()

# disconnect (auto-disconnects on IO error)
conn.disconnect()

String Query

conn.sync("select from trade where date=last date")

Functional Query

Supports Python basic data types, Narwhals Series/DataFrame, and dict (with string keys).

from datetime import date, time

import pyarrow as pa

symbols = pa.chunked_array(
    [pa.array(["sym0", "sym1"]).dictionary_encode()]
)
conn.sync(
    ".gw.query",
    "table",
    {
        "date": date(2023, 11, 21),
        "syms": symbols,
        "startTime": time(9),
        "endTime": time(11, 30),
    },
)

Operators and Lambdas

Pass q primitives and arbitrary lambdas as first-class arguments:

from xqdb import XqdbQLambda, XqdbQOperator

conn.sync("{[op;a;b] .[op;(a;b)]}", XqdbQOperator.PLUS, 1, 2)
conn.sync("{[op;a;b] .[op;(a;b)]}", XqdbQLambda("{x+y}"), 1, 2)

# A non-root q context can be supplied explicitly.
scoped = XqdbQLambda("{x+y}", "analytics")

XqdbQOperator(name) accepts supported q primitive names such as "+"; it does not expose wire opcodes. XqdbQLambda(source, context="") preserves its source text, requires a brace-delimited UTF-8 body (optionally prefixed with k)), rejects NUL bytes in both fields, and rejects context values beginning with ".". The context "analytics" represents q namespace .analytics because the wire context omits the leading dot. Lambda source is executable q code: construct it only from trusted input.

Send DataFrame

import pyarrow as pa

frame = pa.table({"sym": ["a", "b"], "price": [10.5, 11.0]})
conn.sync("upsert", "table", frame)

Async Query

conn.asyn("upsert", "table", frame)

Subscribe

import pyarrow as pa

tables = pa.chunked_array(
    [pa.array(["table1", "table2"]).dictionary_encode()]
)
symbols = pa.chunked_array(
    [pa.array(["sym1", "sym2"]).dictionary_encode()]
)
conn.sync(".u.sub", tables, "")
conn.sync(".u.sub", tables, symbols)

while True:
    # returns ("upd", "table", Narwhals DataFrame)
    upd = conn.receive()
    print(upd)

Generate IPC Bytes

Serialize data as kdb+ IPC bytes without a connection.

import pyarrow as pa

from xqdb import serialize_as_ipc_bytes6

frame = pa.table({"sym": ["a", "b"], "price": [10.5, 11.0]})

# without compression
buffer = serialize_as_ipc_bytes6("sync", False, ["upd", "table", frame])

# with compression
buffer = serialize_as_ipc_bytes6("sync", True, ["upd", "table", frame])

msg_type: "async" | "sync" | "response"

Read Binary Table

Read a regular q binary table file directly into a Narwhals DataFrame. Select the native output backend independently of the file format, and pass symbol_encoding to decode text that is not valid UTF-8 the same way Q does.

from xqdb import read_binary6

df = read_binary6("/path/to/table.bin", backend="pandas")
df = read_binary6("/path/to/legacy.bin", symbol_encoding="lossy")

Error Handling

from xqdb import XqdbError, XqdbIOError, XqdbAuthError

try:
    conn.sync("select from trade")
except XqdbAuthError:
    print("Authentication failed")
except XqdbIOError:
    print("Connection error")
except XqdbError:
    print("General xqdb error")

Data Type Mapping

Deserialization (q → Python)

q scalars map to Python scalars; q vectors/tables are returned as Narwhals DataFrames/Series backed by the selected backend.

Atom (scalar to Python)

Vector and Table (Arrow-backed Narwhals)

Other selected backends receive the equivalent representation that Narwhals can construct from this Arrow stream. Backend-specific dtypes may differ while values and q semantics remain the same.

real/float 0n is mapped to null, not NaN.

short/int/long null and infinity values (0Nh/i/j, 0Wh/i/j, -0Wh/i/j) are mapped to null.

Serialization (Python → q)

Basic Data Type

Series, DataFrame, and Dictionary

Dictionary serialization requires str keys.

Resources

• Narwhals Documentation — Unified dataframe interface
• PyArrow Documentation — Default backend
• Pandas Documentation — Optional backend
• Polars Documentation — Optional backend

License

XQDB is licensed under the BSD-3-Clause permissive open-source license, which permits use in proprietary and commercial applications.