GDB Python Client (gdb-client)

Official high-performance Python client for GDB (Distributed HTAP In-Memory Graph Database).

Powered by Polars SIMD vectorized expressions and Apache Arrow Flight high-speed TCP streaming, gdb-client delivers ultra-high-throughput parallel scatter-ingest directly into in-memory partition stores of cluster nodes.

⚡ Key Features

• Polars-Powered Scatter-Ingest: Automatically partitions massive DataFrames using Polars SIMD expressions and streams partitions in parallel directly to primary replica nodes in the cluster hash ring.
• Dedicated Arrow Flight Ingestion Ports (:8860+): Bypasses HTTP JSON serialization overhead, streaming binary Arrow RecordBatch payloads directly into GDB's in-memory DeltaMemTable.
• Automatic Cluster Hash Ring Discovery: Queries cluster coordinator topology (/cluster) to resolve ring tokens ($u \pmod N$), partition mapping, and per-node Arrow Flight endpoints.
• Native Polars Query Results: Runs openCypher / GQL queries and yields results directly as polars.DataFrame via Arrow Flight do_get with zero-copy deserialization.
• Graceful Fallbacks: Works out of the box with standard library fallbacks if optional dependencies are missing.

📦 Installation

Install from PyPI:

pip install gdb-client

Or install the latest development version directly from GitHub:

pip install git+https://github.com/1orgar/gdb-py-client.git

Requirements

• Python >= 3.9
• polars >= 1.0.0
• pyarrow >= 14.0.0
• requests >= 2.28.0

🚀 Quickstart

import polars as pl
from gdb_client import GdbClient

# 1. Connect to any cluster node (auto-discovers all ring peers and Flight ports)
client = GdbClient(seed_url="http://localhost:8847")

# 2. Ingest 1,000,000 vertices in parallel directly into target nodes
users_df = pl.DataFrame({
    "id": range(1, 1_000_001),
    "name": [f"User_{i}" for i in range(1, 1_000_001)],
    "age": [20 + (i % 50) for i in range(1, 1_000_001)],
})

res_v = client.scatter_ingest_vertices(users_df, label="User", id_col="id")
print(f"Vertices: {res_v['rows_ingested']} rows in {res_v['elapsed_seconds']}s "
      f"({res_v['rows_per_second']:,.0f} rows/s)")

# 3. Ingest edges in parallel (automatically hashed by source vertex)
edges_df = pl.DataFrame({
    "src": range(1, 500_001),
    "dst": range(500_001, 1_000_001),
})

res_e = client.scatter_ingest_edges(edges_df, edge_type="FOLLOWS")
print(f"Edges: {res_e['rows_ingested']} rows in {res_e['elapsed_seconds']}s "
      f"({res_e['rows_per_second']:,.0f} rows/s)")

# 4. Trigger background compaction into immutable Chunked-CSR
client.compact()

# 5. Query graph using openCypher directly into a Polars DataFrame
df = client.query("MATCH (a:User)-[:FOLLOWS]->(b:User) RETURN a.name, b.name LIMIT 10;")
print(df)

# 6. Close Flight TCP connections
client.close()

🏗 Architecture: How Scatter-Ingest Works

In a traditional database setup, client libraries send bulk requests to a single coordinator or load balancer, which must unpack the data, hash every key, and re-transmit records over the network to the correct storage nodes.

gdb-client moves partition routing to the client side:

                  ┌─────────────────────────────────────────┐
                  │           polars.DataFrame              │
                  │       (10,000,000 Rows in RAM)          │
                  └────────────────────┬────────────────────┘
                                       │
                      Polars SIMD Hash: (col % TotalNodes)
                                       │
            ┌──────────────────────────┼──────────────────────────┐
            ▼                          ▼                          ▼
   [Partition Chunk 0]        [Partition Chunk 1]        [Partition Chunk 2]
            │                          │                          │
   PyArrow RecordBatch        PyArrow RecordBatch        PyArrow RecordBatch
            │                          │                          │
  Arrow Flight do_put        Arrow Flight do_put        Arrow Flight do_put
   (TCP Port :8860)           (TCP Port :8861)           (TCP Port :8862)
            ▼                          ▼                          ▼
   ┌─────────────────┐        ┌─────────────────┐        ┌─────────────────┐
   │  GDB Node #1    │        │  GDB Node #2    │        │  GDB Node #3    │
   │  DeltaMemTable  │        │  DeltaMemTable  │        │  DeltaMemTable  │
   └─────────────────┘        └─────────────────┘        └─────────────────┘

1. Client-Side Vectorized Hashing: Polars computes target partitions for all rows simultaneously using native CPU SIMD instructions.
2. Direct Multi-Node Streaming: Partitions are streamed over concurrent TCP Flight channels directly to the responsible nodes' --client-flight-port (:8860, :8861, :8862, etc.).
3. Zero Coordinator Bottleneck: Cluster inter-node network bandwidth is preserved for MPP shuffle exchange and graph traversal.

📖 API Reference

GdbClient

GdbClient(seed_url: str = "http://localhost:8847", auth_token: Optional[str] = None)

• seed_url: HTTP address of any healthy node in the GDB cluster.
• auth_token: Optional Bearer token for clusters with authorization enabled.

Methods

Ingestion Performance Metrics

scatter_ingest_vertices and scatter_ingest_edges return a detailed performance dictionary:

{
    "rows_ingested": 1000000,
    "elapsed_seconds": 0.4281,
    "rows_per_second": 2335902.8,
    "partitions": 3
}

🛠 Local Development & Building Wheel

To build the wheel and source distribution locally:

# Clone the repository
git clone https://github.com/1orgar/gdb-py-client.git
cd gdb-py-client

# Install build dependencies
pip install -e ".[dev]"

# Run tests
python -m unittest discover tests

# Build .whl and .tar.gz distributions
python -m build

# Validate package metadata with twine
twine check dist/*

Built artifacts will be located in dist/:

• dist/gdb_client-0.3.0-py3-none-any.whl
• dist/gdb_client-0.3.0.tar.gz

📄 License

Licensed under the Apache License, Version 2.0. See the LICENSE file for details.