adbc-driver-monetdb

ADBC driver for MonetDB, written in Rust.

Arrow-native reads and writes for MonetDB: polars, pandas, and every other ADBC consumer get columnar result sets (MonetDB's binary result-set protocol decoded directly into Arrow record batches) and bulk ingestion (COPY BINARY ... ON CLIENT streamed from Arrow buffers) through one standard interface.

[!IMPORTANT] This release pins adbc_core and adbc_ffi 0.23 to a public backport. The crates.io Rust exporter cannot carry a known affected-row count alongside an Arrow result stream, although the standard ADBC C API returns both. The backport adds that missing internal Rust capability without changing the ADBC ABI. It is compiled into every wheel and DBC library; only source builds fetch the pinned commit.

TODO: Get this generic capability fixed upstream—by merging the backport or adopting Apache's alternative—and then remove the fork pin and return to the official Apache adbc_core and adbc_ffi crates.io releases.

Installation

Install the Python package:

uv add adbc-driver-monetdb

For standalone driver-manager use, download the archive for your platform from the GitHub Releases page, verify it against the release's SHA256SUMS, and install the unsigned local archive:

uvx --from dbc dbc install --no-verify /path/to/monetdb_PLATFORM_vVERSION.tar.gz

Usage

import polars as pl
from adbc_driver_monetdb import dbapi

with dbapi.connect("monetdb://user:password@localhost:50000/db") as conn:
    df = pl.read_database("SELECT * FROM trades", conn)
    df.write_database("trades_copy", conn, if_table_exists="append", engine="adbc")

# or resolved from the URI scheme:
df = pl.read_database_uri("SELECT 1", "monetdb://localhost:50000/db", engine="adbc")
# TLS URIs resolve through the bundled adbc_driver_monetdbs shim:
secure_df = pl.read_database_uri("SELECT 1", "monetdbs://localhost:50000/db", engine="adbc")

The DB-API connection starts with autocommit disabled, as required by PEP 249. Call conn.commit() to persist a transaction, or pass autocommit=True explicitly. Closing a connection, including by leaving its context manager, rolls back uncommitted work. Consume or close a query's result stream before executing another statement or changing transaction state on the same connection. Use independent connections for parallel queries; ADBC permits drivers to block or reject concurrent statements on one connection, and MonetDB's single MAPI channel shares transaction state between every statement on that connection.

The same connection works directly with pandas 3:

import pandas as pd

with dbapi.connect("monetdb://user:password@localhost:50000/db") as conn:
    df = pd.read_sql("SELECT * FROM trades", conn, dtype_backend="pyarrow")
    df.to_sql("trades_copy", conn, if_exists="append", index=False)

Credentials and read-only access

SQLAlchemy-style URIs work: userinfo in monetdb://user:password@localhost:50000/db is percent-decoded and stripped from the URI before it reaches the protocol layer. URIs end up in shell history, logs, and tracebacks, so prefer supplying credentials separately through the standard ADBC database options username and password (also available as adbc_driver_manager.DatabaseOptions.USERNAME / .PASSWORD), which override any URI userinfo:

import os

from adbc_driver_monetdb import dbapi

with dbapi.connect(
    "monetdb://localhost:50000/db",
    db_kwargs={
        "username": os.environ["MONETDB_USER"],
        "password": os.environ["MONETDB_PASSWORD"],
    },
) as conn:
    ...

The password option is write-only: reading it back through get_option is an error.

MonetDB has no per-connection read-only mode — the server rejects read-only transactions outright (42000!Readonly transactions not supported), so setting the ADBC option adbc.connection.readonly to true returns NotImplemented (see the waived-surface table below). For read-only access, connect as a user that holds only SELECT privileges; the server then enforces this for every statement on the connection:

CREATE USER reader WITH PASSWORD 'secret' NAME 'Reporting' SCHEMA sys;
GRANT SELECT ON sys.trades TO reader;

Configuration and timeouts

Timeout values are integer seconds. The default connection deadline is 30 seconds and the default idle write timeout is 60 seconds. Read and operation timeouts are disabled by default so a healthy long-running query is not terminated merely because it exceeds a client-side wall-clock limit. Zero explicitly selects no deadline; negative values and values above the portable socket limit of 4,294,967 seconds are rejected. A connection timeout covers DNS, every address attempt, TCP or Unix connection, TLS, authentication, redirects, and initial driver metadata. A timeout or cancellation closes the MAPI session, so the partially read connection cannot be reused.

The URI names are connect_timeout, read_timeout, write_timeout, and operation_timeout. Unknown query names are rejected so misspelled settings cannot be silently ignored. This is the only configuration channel available to pl.read_database_uri:

import polars as pl

df = pl.read_database_uri(
    "SELECT * FROM trades",
    "monetdb://localhost:50000/db?connect_timeout=10&operation_timeout=120",
    engine="adbc",
)

The same settings can be supplied separately through ADBC options. Database options override the URI; connection options override the database defaults; statement options override the connection for that statement. DatabaseOptions, ConnectionOptions, and StatementOptions enumerate the supported keys. DatabaseOptionValues, ConnectionOptionValues, and StatementOptionValues are exported TypedDict shapes for applications that want editor completion for dictionary-based configuration.

import polars as pl

from adbc_driver_monetdb import ConnectionOptions, DatabaseOptions, StatementOptions, dbapi

with dbapi.connect(
    "monetdb://localhost:50000/db",
    db_kwargs={
        DatabaseOptions.CONNECT_TIMEOUT: "10",
        DatabaseOptions.OPERATION_TIMEOUT: "120",
    },
    conn_kwargs={
        ConnectionOptions.READ_TIMEOUT: "30",
        ConnectionOptions.READ_PREFETCH: "true",
    },
) as conn:
    with conn.cursor(
        adbc_stmt_kwargs={
            StatementOptions.OPERATION_TIMEOUT: "15",
            StatementOptions.READ_BATCH_ROWS: "65536",
            StatementOptions.READ_PREFETCH: "true",
        }
    ) as cursor:
        df = pl.read_database("SELECT * FROM trades", cursor)

pl.read_database(query, connection) selects ADBC from the supplied DB-API connection or cursor; it has no engine="adbc" parameter. Polars calls connection.cursor() without adbc_stmt_kwargs, so use a preconfigured cursor for statement-specific settings. Its execute_options are forwarded to Cursor.execute: use a sequence for positional ? values and a dictionary for named :name values. adbc_stmt_kwargs is not an execute option. Polars' batch_size does not configure adbc.monetdb.read_batch_rows, and DataFrame.write_database(..., engine_options=...) supplies ingestion arguments rather than connection or timeout options. The read batch default is 131,072 rows, matching PyArrow Dataset Scanner's default. Read prefetch is enabled by default and can hold up to about three windows at once: one decoding, one buffered, and one in flight. Abandoning a stream can therefore waste up to two fetched windows; increasing read_batch_rows also increases this memory bound. Closing a reader waits briefly for its worker and then detaches a fetch that is still in flight; it does not implicitly cancel and permanently close the session. The next statement on that connection may wait for the detached fetch or its configured read/operation timeout. Use adbc_cancel() when session destruction is intended, or set adbc.monetdb.read_prefetch to "false" when prompt pool reuse matters more than fetch/decode overlap.

Ingestion automatically chooses the fewest COPY windows that fit a 512 MiB encoded-byte budget. For constrained append targets whose fixed-width wire rows are at most 16 bytes, the budget is raised to 2 GiB. The driver measures every upstream Arrow batch, coalesces small batches, and splits large ones without copying their buffers so each window stays within the byte budget. A single row larger than the budget is the only possible overrun. Tiered compaction keeps metadata bounded without repeatedly recopying the accumulated tail. The narrow-row constraint budget avoids repeated index maintenance across tens of millions of rows without imposing the larger memory bound on variable-width, wide, or ordinary append-only tables. On Linux both ceilings are reduced to a fixed fraction of a finite cgroup memory limit when needed.

Each requested column is encoded directly into bounded 1 MiB encoder chunks, which the protocol layer coalesces into 16 MiB upload messages; null-free signed integers and 32/64-bit floats borrow their Arrow value buffers after validation instead of copying them. The protocol scatter-writes MAPI headers with the borrowed message payload instead of constructing another message-sized buffer, so the reported conservative streaming-buffer peak is about 17 MiB plus framing headers, independent of logical window size.

Large logical windows do not imply equally large resident staging buffers. With wire_compression=none, each new window samples up to 256 KiB per column and compares ordinary LZ4 compression with byte-plane shuffle followed by LZ4 for fixed-width values. It retains the smaller representation only when compression saves at least 12.5%. Selection follows the observed bytes, not a type-name or sortedness heuristic; timestamp and smooth numeric byte patterns consequently benefit while random values do not. auto probes plain LZ4 so profitable frames can be sent directly, while lz4 forces that framed representation. During upload, at most one 1 MiB encoded piece is decompressed and unshuffled into reusable anonymous mappings before it is passed to the normal MonetDB binary protocol. Process-local chunks use direct LZ4 blocks to avoid frame construction and an extra copy; wire-eligible chunks use LZ4 frames. Neither representation is an on-disk cache.

If a column stops meeting the savings threshold, it falls back to raw retention; if the window as a whole is not materially compressible, the driver retains later Arrow batches and encodes bounded pieces on a worker while earlier pieces upload. On a low-latency target it automatically closes that window at 64 MiB. A connection metadata round trip of at least 0.5 ms, or a table with at least 512 columns, raises this floor to the effective 512 MiB budget because each additional COPY window otherwise repeats per-column network exchanges or wide server-plan compilation. A local random-float calibration found the larger floor neutral at 100 columns and 15–36% faster from 200 through 1,000 columns. The 512-column cutoff keeps the tighter default memory bound for medium-width tables while retaining the 33–36% gain measured from 512 through 1,000 columns. The measured round trip and chosen floor are reported in ingest stats.

Compared with an application that stages complete column files on disk, a very wide, incompressible stream may consequently produce more COPY windows and use more MonetDB memory while the server consolidates or reloads those appends. Raising the window would retain more Arrow data in client RAM; spilling the encoded columns would recreate the disk-staging path. The automatic cap deliberately favors bounded client memory and no staging files. Compressible streams avoid this tradeoff because their large logical windows remain small in physical memory.

adbc.monetdb.write_window_bytes changes both the main byte budget and the incompressible-data floor; zero selects the automatic default. adbc.monetdb.write_batch_rows remains a diagnostic row-count override and disables byte-window adaptation. Both options can be set on a connection or statement, with statement values taking precedence. write_window_bytes is also accepted as a URI query parameter for applications whose only configuration surface is a connection URI. Applications normally need none of these: producer batch boundaries do not define COPY boundaries.

MonetDB Dec2025 and newer can decompress LZ4 frames during client COPY. The default adbc.monetdb.wire_compression=auto sends a window's plain LZ4 frames on the wire only when every column passed the compression threshold; otherwise it retains Arrow and sends ordinary binary COPY. This avoids weak client-only compression on mixed time-series data while still reusing profitable frames for compressible windows. Byte-shuffled blocks remain client-local because the server does not undo that transform. Set the option to lz4 to compress every column for a bandwidth-constrained link, or none to prefer the lower-memory client-only shuffle path. The option is accepted at database, connection, and statement scope and as wire_compression in the URI. Ingest stats report the choice for every window in window_wire_compression.

A complete, single-batch ingest of at most 100 rows and 8 MiB uses one cached prepared INSERT script instead of binary COPY. This avoids one client-file exchange per column and is especially important for tiny wide appends. The threshold increases from the measured connection round trip on higher-latency links, while the byte cap remains fixed. Set adbc.monetdb.ingest_insert_rows=0 to force COPY, or set a positive floor on the connection or statement; the same key without the adbc.monetdb. prefix is accepted in a URI. An explicit write_batch_rows setting continues to force the COPY scheduler. Explicit values supplied for an identity column do not advance MonetDB's identity sequence on either INSERT or COPY; applications mixing explicit identity values with generated ones must manage that sequence.

After an ingest, the read-only statement option adbc.monetdb.ingest_stats returns JSON containing the chosen path, measured round trip, effective INSERT threshold, logical and physical stored bytes, storage and wire-compression mode per window, prepared-cache hits, input-batch and COPY counts, coalesced/split window counts, bounded in-flight bytes, window trajectories, transaction scope, and poison state.

Window construction runs one logical window ahead of the active COPY on a zero-capacity handoff. This overlaps Arrow production and encoding with the current upload while bounding lookahead to one window. Reader failures and worker panics are joined and returned before ingest completion.

Polars lazy queries do not expose an Arrow stream directly. Install the optional integration with uv add 'adbc-driver-monetdb[polars]' and wrap the query in the fully backpressured PolarsArrowStream. Its default producer batches are bounded independently of the driver's exact encoded-byte windows:

import polars as pl

from adbc_driver_monetdb import PolarsArrowStream, dbapi

frame = pl.scan_parquet("trades/*.parquet")

with dbapi.connect("monetdb://localhost:50000/db") as connection:
    with connection.cursor() as cursor:
        stream = PolarsArrowStream(frame)
        inserted_rows = cursor.adbc_ingest("trades", stream, mode="append")
        assert inserted_rows == stream.rows_read

The base driver remains importable without Polars or PyArrow. Install adbc-driver-monetdb[pyarrow] for PyArrow-specific driver-manager APIs without the Polars adapter.

An append to an existing table inside an explicit transaction executes directly for every Arrow stream window. This avoids MonetDB retaining and replaying a full table version for an operation savepoint. If a client-side stream or encoding error occurs after one or more completed COPY windows, subsequent reads remain available but commit() raises InvalidState until rollback() removes the partial append. The same guard applies to raw SQL COMMIT, and a successful raw SQL ROLLBACK clears it. Server errors retain their DB-API exception and SQLSTATE; MonetDB itself leaves that transaction aborted until rollback. Autocommit ingestion wraps the complete stream in an internal transaction, rolls it back on error, and restores the connection.

Two advanced statement or connection options change that contract. Setting adbc.monetdb.ingest_atomicity to "savepoint" preserves earlier caller work and rolls back only the failed ingest, but MonetDB Dec2025 may write an extra full copy of a sub-million-row append to its WAL. The default "transaction" scope avoids that amplification and blocks commit after a partial client-side failure. Setting adbc.monetdb.ingest_partial to "allow" permits committing completed windows after such a failure; the default "block" is the safe behavior.

Positional prepared statements are cached per connection by exact SQL text, so consumers such as SQLAlchemy can create a fresh cursor for each execution without making MonetDB compile the same plan again. The least-recently-used cache holds 512 plans by default; set adbc.monetdb.prepared_cache_capacity as a database/connection option or prepared_cache_capacity in the URI when a workload needs a different positive bound. Eviction queues server-side deallocation, and closing the connection releases the session and every remaining plan. Whitespace variants are intentionally different keys. Schema-changing statements issued through the connection invalidate the cache, and externally invalidated plans are prepared again and retried once when MonetDB can recover without rolling back user work. MonetDB aborts an explicit transaction when EXECUTE reports a missing prepared plan, so a stale plan in that state follows the normal database-error contract: roll back before retrying. One-row bound DML executes directly; multi-row bound DML retains a savepoint so the whole parameter batch remains atomic.

dbapi.Binary accepts bytes-like values (bytes, bytearray, and memoryview) and returns bytes. Text is rejected with TypeError; encode text explicitly before binding it as binary.

Performance expectations

The Arrow-native path is designed for columnar reads and bulk ingestion. A one-row parameterized DML execution takes the direct prepared-statement path; parameter batches with two or more rows use a savepoint so the complete batch stays atomic. The login keeps MonetDB's normal inline reply window, so small result sets are decoded from the initial response instead of forcing another fetch.

Very small queries can still be slower than pymonetdb. pymonetdb returns Python tuples directly, whereas an ADBC query must build an Arrow schema and buffers across the native boundary before the driver manager converts those buffers back to DB-API tuples. That fixed Arrow/FFI cost is inherent when a caller asks an Arrow-native ADBC driver for row-oriented Python objects; bypassing it would make DB-API results disagree with the native ADBC stream. Prefer Arrow consumers such as Polars, pandas with the PyArrow backend, or fetch_arrow_table() when result size makes that fixed cost material.

The small-query comparison is reproducible against the same server and host:

MONETDB_TEST_URI=monetdb://monetdb:monetdb@localhost:50000/test \
MONETDB_RUN_LATENCY_BENCHMARK=1 \
uv run pytest tests/test_local_benchmark.py::test_local_short_query_latency_against_pymonetdb -q -s

The examples use strings because adbc-driver-manager publishes string-valued type hints for database and connection option mappings. Statement options also accept native integers through the ADBC integer option path.

The DB-API module reports threadsafety = 1: threads may share the module, but each thread should use its own connection and cursors. Cancellation is connection-scoped: it interrupts the operation currently using that connection, not necessarily the statement object on which cancel was called. MAPI cannot safely resume a partially read response, so cancellation closes the session permanently; close that connection and open another one before issuing more work.

Client information

Client information is sent at login by default. The client value in sys.sessions identifies this driver and its protocol library, for example adbc_driver_monetdb 0.9.0 / monetdb-rust 0.2.2-wlaur.1. The Python shim uses the basename of sys.argv[0] as the default application. Hostname and process id are also sent by default, as they are by pymonetdb and libmapi; use client_info=false if that host metadata should not leave the client.

The URI parameters are client_application, client_remark, and client_info. The equivalent pre-connect database options are adbc.monetdb.client_application, adbc.monetdb.client_remark, and adbc.monetdb.client_info; database options override URI values. Application and remark values cannot contain newlines.

from adbc_driver_monetdb import DatabaseOptions, dbapi

with dbapi.connect(
    "monetdb://localhost:50000/db?client_application=nightly-load",
    db_kwargs={DatabaseOptions.CLIENT_REMARK: "warehouse refresh"},
) as conn:
    session = conn.execute(
        "SELECT hostname, application, client, clientpid, remark "
        "FROM sys.sessions WHERE sessionid = current_sessionid()"
    ).fetchone()

For a post-connect update, call sys.setclientinfo:

CALL sys.setclientinfo('ClientRemark', 'phase 2');

The native DB-API parameter style is qmark (?). Named :name parameters are also supported when a parameter dictionary is supplied, including SQLAlchemy expressions compiled to a SQL string:

from sqlalchemy import Integer, bindparam, cast, select

value = cast(bindparam("value", value=21), Integer)
compiled = select((value + value).label("value")).compile()
df = pl.read_database(
    str(compiled),
    conn,
    execute_options={"parameters": compiled.params},
)

Support policy

• MonetDB Dec2025 (11.55) and newer, little-endian servers only
• Python 3.13+ (one abi3 wheel per platform), polars 1.42+, pandas 3.0+, adbc-driver-manager 1.11+
• Platforms: Linux x86_64 + aarch64 (manylinux), macOS arm64, Windows x86_64

ADBC release baseline

Feature parity means all required rows below are implemented and tested, not that every optional ADBC entry point must be synthesized for a backend that cannot use it. This follows ADBC's own driver feature matrix: for example, the stable PostgreSQL driver does not support partitioned results or Substrait, while still covering the common SQL-driver baseline.

These optional or backend-inapplicable surfaces are explicitly outside the release gate. Their entry points are still tested to return ADBC NotImplemented, rather than a transport, parser, or argument error.

The reusable ADBC validation suite currently passes against Dec2025-SP3. Its skips cover explicitly waived cross-catalog/statistics behavior and negative-scale decimals that MonetDB itself does not support. Polars' announced future unknown-extension behavior is exercised in CI; HUGEINT and TIMETZ remain round-trippable when loaded as extension types. The monetdb.hugeint extension uses Arrow Decimal128(38, 0), so its supported domain is −(10^38−1) through 10^38−1; wider values in MonetDB's signed 128-bit domain return a bounded conversion error instead of silently changing the public Arrow type. Cast those wider values to VARCHAR in SQL when their full textual representation is required.

MonetDB JSON query results use Arrow's canonical arrow.json extension with UTF-8 string storage. Under the supported storage policy, Polars loads it as pl.String; the driver does not expand it to pl.Struct because one JSON column can contain objects, arrays, scalars, and JSON null with different shapes. Applications that know an object schema can opt in:

decoded = df.with_columns(
    pl.col("payload").str.json_decode(
        dtype=pl.Struct({"id": pl.Int64, "label": pl.String}),
    )
)

MonetDB functions declared to return JSON—including json.filter, json.keyarray, and json.valuearray—preserve arrow.json. Functions such as json.text, json.number, json."integer", json.length, and the JSON predicates return their declared scalar Arrow types. This follows MonetDB's JSON model, where JSON is a validated string subtype.

Backend-specific type boundaries are explicit:

Date64 ingest accepts only whole-day millisecond values; intra-day values must use an Arrow timestamp type. MonetDB does not expose compatible Xexportbin or COPY BINARY representations for the remaining waived paths. The driver returns bounded errors instead of adding a lossy text-protocol fallback.

dbc packages and driver-manager loading

Each release target also builds a standalone, non-Python cdylib for a flat dbc package. Installing that archive makes the driver discoverable as monetdb by C/C++, Go, R, Ruby, Rust, Python, and other ADBC driver managers. dbc writes the installed ADBC TOML manifest with an absolute shared-library path; a relative Driver.shared path is not portable.

Linux dbc libraries are built in the pinned manylinux_2_28 environment and support glibc 2.28 or newer. Windows dbc libraries statically link the Visual C++ runtime. Release CI audits both constraints before packaging.

uv run python packaging/generate_licenses.py
uv run python packaging/dbc/build_package.py \
    --library target/release/libadbc_monetdb.dylib \
    --platform macos_arm64 --out-dir dist/dbc --license THIRD_PARTY_LICENSES
ADBC_DRIVER_PATH="$PWD/.adbc-drivers" \
    uvx --from dbc dbc install --no-verify dist/dbc/monetdb_macos_arm64_v*.tar.gz

Locally built and GitHub Release archives are unsigned, hence --no-verify for direct archive installation. Every GitHub Release includes SHA256SUMS; verify the archive checksum before installing it.

Polars can use a dbc-installed driver without the adbc-driver-monetdb Python package when the connection is created by the Python driver manager (which is still required by Polars' ADBC engine):

import polars as pl
from adbc_driver_manager import dbapi

# The driver itself was installed from a downloaded GitHub Release archive as shown above.
with dbapi.connect(
    driver="monetdb",
    db_kwargs={"uri": "monetdb://user:password@localhost:50000/db"},
) as conn:
    df = pl.read_database("SELECT * FROM trades", connection=conn)
    df.write_database("trades_copy", connection=conn, engine="adbc")

The URI-string conveniences pl.read_database_uri(..., engine="adbc") and DataFrame.write_database(..., connection="monetdb://...") import the driver package by URI scheme and therefore require the Python distribution. The wheel includes both adbc_driver_monetdb and the TLS alias adbc_driver_monetdbs, so both monetdb:// and monetdbs:// work through those conveniences. This alias follows Polars' scheme-to-module lookup; it is not a second ADBC driver or Python distribution.

ADBC itself treats the canonical uri option and driver loading separately. The wheel aliases load the same native entrypoint, and the standalone DBC package has one monetdb manifest. DBC users select driver="monetdb" and may pass either URI scheme unchanged to that driver.

Repository layout

Development

See CONTRIBUTING.md for bug, security, feature-request, and contribution guidance.

git clone --recurse-submodules https://github.com/wlaur/adbc-driver-monetdb
uv sync                                # installs deps + builds the extension via maturin
uv run pytest -m "not integration and not local_only"  # python tests (no server needed)
cargo test --workspace                 # rust tests

# integration tests against a real server:
# compose.yaml pins the native ARM64 Dec2025-SP3 wlaur/monetdb-container image
docker compose -f compose.yaml up -d
MONETDB_TEST_URI=monetdb://monetdb:monetdb@localhost:50000/test \
    uv run pytest -m "integration and not local_only"

# manual ~30 GiB logical float32 ingest (8M rows x 1,000 columns, 100k batches);
# reports final and peak dbfarm growth plus peak filesystem growth:
MONETDB_RUN_LOCAL_BENCHMARK=1 \
MONETDB_TEST_URI=monetdb://monetdb:monetdb@localhost:50000/test \
    uv run pytest tests/test_local_benchmark.py -m local_only -q -s

# run the reusable ADBC conformance suite:
MONETDB_TEST_URI=monetdb://monetdb:monetdb@localhost:50000/test \
    uv run pytest tests/validation

Lint/typecheck: uv run ruff check ., uv run ruff format --check ., uv run pyright, cargo clippy --workspace --all-targets, cargo fmt --all --check.

License

The driver is MIT and the included monetdb protocol crate (monetdb-rust, our fork of MonetDB/monetdb-rust) is MPL-2.0. The distribution's license expression is therefore MIT AND MPL-2.0; both license texts and the corresponding-source notice are included in wheels and source distributions.