spanner-adbc

Experimental. AI-generated and not yet validated in production use.

A Python ADBC driver for Google Cloud Spanner.

Query Spanner through DBAPI 2.0 and fetch Apache Arrow results for pandas, Polars, DuckDB, or PyArrow.

Install

Requires Python 3.11 or later. Wheels bundle the native driver; see Supported platforms for OS requirements.

pip install "spanner-adbc[dbapi]"  # includes PyArrow and pandas

For the low-level ADBC API without DataFrame dependencies, install spanner-adbc. The Python import name is spanner_adbc.

Quickstart

This example assumes an existing Singers table with SingerId and FirstName columns.

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers")
        df = cur.fetch_df()          # -> pandas.DataFrame

Use GoogleSQL @name parameters with cur.execute(...); ? placeholders are unsupported. Pass a tuple in first-occurrence order instead of formatting values into SQL:

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/p/instances/i/databases/d"},
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT @name AS name, @id AS id", ("Alice", 42))
        assert cur.fetchone() == ("Alice", 42)

fetchone() and fetchall() return Python rows; Arrow and DataFrame helpers are shown below.

Driver manifest (driver="spanner")

The package's spanner.connect() loads its bundled library directly. To use the generic adbc_driver_manager by driver name, install an ADBC driver manifest. Use a current driver manager; Python manifest support requires version 1.8.0 or later.

pip install --upgrade adbc-driver-manager
python -m spanner_adbc.manifest install

The equivalent console command is spanner-adbc-install-manifest.

import adbc_driver_manager.dbapi

with adbc_driver_manager.dbapi.connect(
    driver="spanner",
    uri="spanner:///projects/my-project/instances/my-instance/databases/my-db",
) as conn:
    ...

Current driver managers can also infer spanner from the URI scheme when driver is omitted.

• Re-run the installer after upgrading, reinstalling, or moving the environment: the manifest contains the library's absolute path.
• The default directory is <sys.prefix>/etc/adbc/drivers inside a virtual environment, or the platform's user configuration directory otherwise. python -m spanner_adbc.manifest path prints the target path.
• Use install --dir /path/to/drivers for a custom directory on ADBC_DRIVER_PATH.
• For a standalone shared library, edit the Driver.shared paths in the repository's spanner.toml.

Authentication

With no credential options, the driver uses Application Default Credentials: run gcloud auth application-default login locally, set GOOGLE_APPLICATION_CREDENTIALS, or use an attached service account on Google Cloud.

Pass explicit credentials in db_kwargs alongside DatabaseOptions.URI:

Use each enum member directly as the key, or use its .value. ACCESS_TOKEN cannot be combined with key-file credentials or impersonation. See the option reference for scopes and other auth settings.

For the Spanner emulator, use anonymous mode:

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(db_kwargs={
    DatabaseOptions.URI: "spanner:///projects/p/instances/i/databases/d",
    DatabaseOptions.ENDPOINT: "localhost:9010",
    DatabaseOptions.EMULATOR: "true",
}) as conn:
    ...

Emulator mode rejects explicit credential options; ambient ADC is ignored.

For Spanner Omni, set DatabaseOptions.INSTANCE_TYPE to "omni" and DatabaseOptions.ENDPOINT to the server endpoint, such as "http://localhost:15000". Use spanner:///projects/default/instances/default/databases/<database> as the URI. Omni uses its own transport; leave DatabaseOptions.EMULATOR unset.

Options

Use DatabaseOptions, ConnectionOptions, and StatementOptions enums, or raw option keys. These are StrEnum members: str(DatabaseOptions.URI) and f"{DatabaseOptions.URI}" both produce "uri" on every supported Python version. Members work directly as dictionary keys; .value remains useful for driver-manager getters whose Cython arguments require an exact str rather than a subclass. The option reference lists types, defaults, and accepted values. Set cursor options with conn.cursor(adbc_stmt_kwargs={...}) or cur.adbc_statement.set_options(**{...}).

import spanner_adbc.dbapi as spanner
from spanner_adbc import ConnectionOptions, DatabaseOptions, StatementOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/p/instances/i/databases/d"},
    conn_kwargs={ConnectionOptions.READ_STALENESS: "max:10s"},
    autocommit=True,  # bounded staleness on a single-use read
) as conn:
    with conn.cursor(
        adbc_stmt_kwargs={StatementOptions.ROWS_PER_BATCH: "1024"}
    ) as cur:
        cur.execute("SELECT * FROM Singers")

In manual query transactions, max:<duration> and min:<timestamp> bounds become exact staleness and a fixed read timestamp respectively. Set ConnectionOptions.READONLY to "true" to reject DML, DDL, and ingest; rollback remains available.

Transactions

DBAPI defaults to autocommit=False. A manual transaction accepts either queries or writes, chosen by its first query or write; mixing them raises spanner.ProgrammingError until commit or rollback. Queries share a snapshot. DML is buffered until conn.commit(), so queries cannot read buffered writes.

Exiting with spanner.connect(...) closes the connection without committing. Call conn.commit() explicitly before leaving the block to keep buffered writes.

DDL always executes immediately: rollback cannot undo it, and it runs before buffered DML. Use autocommit=True for immediately committed DML, including THEN RETURN statements. See transactions for bulk-ingest and partitioned-DML behavior.

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
) as conn:
    with conn.cursor() as cur:
        # DDL applies immediately.
        cur.execute("DROP TABLE IF EXISTS Albums")
        cur.execute("CREATE TABLE Albums (Id INT64 NOT NULL) PRIMARY KEY (Id)")

        cur.execute("INSERT INTO Albums (Id) VALUES (1)")  # buffered until commit
    conn.commit()  # Commit writes before starting a query transaction.

    with conn.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM Albums")
        assert cur.fetchone()[0] == 1
    conn.rollback()  # end the query snapshot

Working with DataFrames

The quickstart uses pandas. The examples below use the same Singers table with SingerId and FirstName columns. Install polars or duckdb separately for those examples.

pyarrow — results as a native Arrow table:

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers ORDER BY SingerId")
        table = cur.fetch_arrow_table()      # -> pyarrow.Table

fetch_arrow_table() and DataFrame helpers collect the complete result in memory. For large queries, consume record batches while the cursor and connection remain open:

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/p/instances/i/databases/d"},
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers")
        with cur.fetch_record_batch() as batches:
            for batch in batches:
                print(batch.num_rows)  # process or write each batch here

polars — read straight from the connection:

import polars as pl
import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
) as conn:
    df = pl.read_database(
        "SELECT SingerId, FirstName FROM Singers ORDER BY SingerId",
        connection=conn,                     # an ADBC connection, not a URI
    )

DuckDB — query the fetched Arrow table in-process:

import duckdb
import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT SingerId, FirstName FROM Singers")
        singers = cur.fetch_arrow_table()

# DuckDB can query the Arrow table by variable name.
top = duckdb.sql("SELECT COUNT(*) AS n, MIN(FirstName) AS first FROM singers").fetchone()

Bulk insert a DataFrame

cur.adbc_ingest(table, data, mode=...) inserts Arrow-compatible data in bulk. Convert pandas DataFrames to Arrow as shown here:

import pandas as pd
import pyarrow as pa
import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

frame = pd.DataFrame({"SingerId": [10, 11], "FirstName": ["Carol", "Dave"]})

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
    autocommit=True,                         # apply immediately; returns the row count
) as conn:
    with conn.cursor() as cur:
        # `append` inserts into an existing table (the default mode is `create`).
        rows = cur.adbc_ingest("Singers", pa.Table.from_pandas(frame), mode="append")

The mode selects how the target table is handled:

• create — create the table from the data's Arrow schema first, erroring if it already exists (the default).
• append — insert into an existing table.
• create_append — create the table only if it is absent, then insert.
• replace — drop any existing table, recreate it from the schema, then insert.

Create modes omit a primary key, so Spanner supplies a hidden rowid column. For an explicit primary key, create the table with SQL first and use mode="append". In manual mode, inserts need conn.commit(); table creation or replacement still happens immediately. Autocommit loads may commit in multiple chunks, so a failed load can leave rows written.

Partitioned reads and Data Boost

A large scan can be split into independent partitions and read in parallel with the ADBC partitioned-execution extension (adbc_execute_partitions / adbc_read_partition):

import spanner_adbc.dbapi as spanner
from spanner_adbc import DatabaseOptions

with spanner.connect(
    db_kwargs={DatabaseOptions.URI: "spanner:///projects/my-project/instances/my-instance/databases/my-db"},
) as conn:
    with conn.cursor() as cur:
        partitions, schema = cur.adbc_execute_partitions("SELECT SingerId FROM Singers")

    # Each descriptor is opaque bytes; it can be shipped to another worker,
    # process, or connection and read independently.
    for token in partitions:
        with conn.cursor() as cur:
            cur.adbc_read_partition(token)
            table = cur.fetch_arrow_table()
            ...

Cloud Spanner can run partitions on serverless Data Boost compute. Enable it on the producing cursor with adbc_stmt_kwargs={StatementOptions.DATA_BOOST: "true"} after importing StatementOptions. Spanner Omni does not support Data Boost.

Spanner decides partitionability from the query plan. Simple scans are suitable; joins, ordering, and aggregation can prevent partitioning. See the partitionability rules.

Descriptors contain SQL and transaction identifiers and are unauthenticated. Only read descriptors from trusted sources: they execute using the receiving connection's credentials.

Supported platforms

pip selects a wheel matching the OS and architecture:

Building for an older OS requires a compatible native driver and Python dependencies. Source wheel builds require exactly one native driver library staged in python/spanner_adbc/; the Python build does not compile Rust. For example, on Linux:

cargo build --release
cp target/release/libspanner_adbc.so python/spanner_adbc/
pip install './python[dbapi]'

For local builds and releases, see CONTRIBUTING.md.