arcane-bigquery-storage 1.3.2

A package to use the BigQuery Storage Write API

Arcane bigquery-storage

A thin wrapper around the BigQuery Storage Write API. The Storage Write API uses protocol buffers to describe and serialize rows, so to write to a table you bring a compiled protobuf message class that mirrors the table schema.

The flow is always the same:

1. Write a .proto that mirrors your BigQuery table schema.
2. Compile it with protoc to get a *_pb2.py module.
3. Use it with Client.create_proto_descriptor / Client.create_row / Client.write_rows.

The next three sections walk through each step.

1. Write a .proto

Create a file like my_table.proto next to your code. Each field must match a column in your BigQuery table by name; the proto type must be compatible with the BigQuery column type (see the protobuf → BigQuery type mapping).

syntax = "proto2";

message MyRow {
    optional string name  = 1;
    optional int64  value = 2;
}

A couple of rules worth knowing:

• Field names in the proto must equal column names in BigQuery.
• Field numbers (the = 1, = 2 tags) identify fields on the wire. Never re-use a number for a different field — that breaks compatibility with any rows already written.
• proto2 makes every field optional by default, which mirrors BigQuery's NULLABLE columns nicely. proto3 works too; just be aware of its default-value semantics.

2. Compile the .proto

You only need to do this once per message type — commit the generated *_pb2.py to your repo.

Install the compiler

Check the protoc downloads page for the current release and the Python compatibility matrix. At the time of writing, protoc 21.12 pairs with protobuf 3.20.3.

On macOS:

PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v21.12/protoc-21.12-osx-universal_binary.zip
unzip protoc-21.12-osx-universal_binary.zip -d protoc-21.12
sudo mv protoc-21.12/bin/protoc /usr/local/bin/
sudo cp -r protoc-21.12/include/* /usr/local/include/
protoc --version

Run the compiler

From the directory containing my_table.proto:

protoc -I. -I/usr/local/include --python_out=. --pyi_out=. my_table.proto

This generates my_table_pb2.py (and a .pyi stub). Commit both.

3. Write rows to BigQuery

from arcane.bigquery_storage.client import Client
from my_package.proto import my_table_pb2  # the file you just compiled

client = Client()  # picks up Application Default Credentials

# Get a stream to write to.
# Default stream = at-least-once, no per-hour limit. For exactly-once,
# use client.create_application_stream(...) instead.
stream_name = client.create_default_stream_name(
    project_id="my-project",
    dataset_id="my_dataset",
    table_id="my_table",
)

# Build a descriptor of your message type.
descriptor = Client.create_proto_descriptor(my_table_pb2.MyRow)

# Serialize each row as bytes.
rows = [
    Client.create_row(my_table_pb2.MyRow, {"name": "alice", "value": 1}),
    Client.create_row(my_table_pb2.MyRow, {"name": "bob",   "value": 2}),
]

# Append.
client.write_rows(stream_name, descriptor, rows)

Keys in the dict passed to create_row must match the field names in your .proto exactly; unknown keys raise ValueError from the protobuf runtime.

See the stream-type docs for the trade-offs between the default stream and application-created streams.

Legacy helpers (deprecated)

Earlier versions of this package shipped table-specific helpers (create_feed_boost_result_proto_descriptor, create_feed_boost_statistic_proto_descriptor, create_feed_boost_result_row, create_feed_boost_statistic_row) and bundled the matching .proto files. They still work but now emit a DeprecationWarning and will be removed in a future major release. New code should use Client.create_proto_descriptor and Client.create_row with its own compiled protobufs.

Release history

See CHANGELOG.md.