adbc-driver-spanner 0.5.0

ADBC (Arrow Database Connectivity) driver for Google Cloud Spanner

adbc-driver-spanner

A Python ADBC driver for Google Cloud Spanner.

It bundles the prebuilt native driver (a Rust cdylib) and exposes it through adbc_driver_manager, so you get a standard DBAPI 2.0 connection whose results come back as Apache Arrow — ready for pandas, polars, DuckDB, or PyArrow with no per-row conversion.

Install

pip install adbc-driver-spanner
# for the DataFrame / Arrow helpers used below:
pip install adbc-driver-spanner[dbapi] pandas

Prebuilt wheels are published for Linux (x86-64, aarch64), macOS (arm64, x86-64), and Windows (x86-64, arm64).

Quickstart

import adbc_driver_spanner.dbapi as spanner

with spanner.connect(
    database="projects/my-project/instances/my-instance/databases/my-db",
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers")
        df = cur.fetch_df()          # -> pandas.DataFrame

Connection options

Options mirror the driver's adbc.spanner.* keys:

kwarg	driver option
database=	adbc.spanner.database
endpoint=	adbc.spanner.endpoint
emulator=	adbc.spanner.emulator
keyfile=	adbc.spanner.keyfile
keyfile_json=	adbc.spanner.keyfile_json

Credentials default to Application Default Credentials; pass keyfile= / keyfile_json= for a service account, or point at the emulator:

# docs-test: skip
spanner.connect(database="projects/p/instances/i/databases/d",
                endpoint="localhost:9010", emulator=True)

Cookbook

Every snippet below is executed against the Spanner emulator in CI, so they stay correct. They assume a Singers(SingerId INT64, FirstName STRING) table.

Two things to know:

• DBAPI is autocommit-off by default, so DML and ingest need a conn.commit() (or pass autocommit=True). Reads need neither.
• The DataFrame / Arrow paths need the [dbapi] extra (pyarrow).

pyarrow — zero-copy Arrow:

import adbc_driver_spanner.dbapi as spanner

with spanner.connect(
    database="projects/my-project/instances/my-instance/databases/my-db",
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers ORDER BY SingerId")
        table = cur.fetch_arrow_table()      # -> pyarrow.Table

polars — read straight from the connection:

import polars as pl
import adbc_driver_spanner.dbapi as spanner

with spanner.connect(
    database="projects/my-project/instances/my-instance/databases/my-db",
) as conn:
    df = pl.read_database(
        "SELECT SingerId, FirstName FROM Singers ORDER BY SingerId",
        connection=conn,                     # an ADBC connection, not a URI
    )

DuckDB — query the fetched Arrow table in-process:

import duckdb
import adbc_driver_spanner.dbapi as spanner

with spanner.connect(
    database="projects/my-project/instances/my-instance/databases/my-db",
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers")
        singers = cur.fetch_arrow_table()

# `singers` is a pyarrow.Table; DuckDB queries it by variable name, no copy.
top = duckdb.sql("SELECT COUNT(*) AS n, MIN(FirstName) AS first FROM singers").fetchone()

Insert a DataFrame (bulk ingest):

import pandas as pd
import pyarrow as pa
import adbc_driver_spanner.dbapi as spanner

frame = pd.DataFrame({"SingerId": [10, 11], "FirstName": ["Carol", "Dave"]})

with spanner.connect(
    database="projects/my-project/instances/my-instance/databases/my-db",
    autocommit=True,                         # apply immediately; returns the row count
) as conn:
    with conn.cursor() as cur:
        # The target table must already exist — only append mode is supported.
        rows = cur.adbc_ingest("Singers", pa.Table.from_pandas(frame), mode="append")

Partitioned reads and Data Boost

A large scan can be split into independent partitions and read in parallel — optionally on Spanner's serverless Data Boost compute, so the work is isolated from your provisioned instance. This uses the ADBC partitioned-execution extension (adbc_execute_partitions / adbc_read_partition):

import adbc_driver_spanner.dbapi as spanner

with spanner.connect(
    database="projects/my-project/instances/my-instance/databases/my-db",
) as conn:
    with conn.cursor() as cur:
        # Optional statement options, set on the underlying ADBC statement:
        cur.adbc_statement.set_options(**{
            "adbc.spanner.data_boost_enabled": "true",  # run on Data Boost
            "adbc.spanner.max_partitions": "8",          # cap the partition count
        })
        partitions, schema = cur.adbc_execute_partitions("SELECT SingerId FROM Singers")

    # Each descriptor is opaque bytes; it can be shipped to another worker,
    # process, or connection and read independently.
    for token in partitions:
        with conn.cursor() as cur:
            cur.adbc_read_partition(token)
            table = cur.fetch_arrow_table()
            ...

The Data Boost choice is baked into each descriptor, so it is honoured wherever the partition is read. Only single-table scans are partitionable — queries with an ORDER BY or aggregation are not.

How this package is built

The wheel is data-only: it does not compile anything at install time and links nothing against Python. CI (.github/workflows/libraries.yml) builds the native library per platform, drops it into adbc_driver_spanner/, and packages a py3-none-<platform> wheel. See that workflow for the release wiring.