bigquery-spark-streaming-connector 0.1.0

spark structured streaming connector utility for gcp bigquery using storage api services.

bigquery-streaming-connector

This is a library for bigquery streaming connector for pyspark.

The underlying connector uses bigquery storage api services to pull bigquery table data at scale using spark workers.

The storage api services is cheaper and faster than the traditional Bigquery query api services enabling faster & cheaper Bigquery migration incrementally in a continuous fashion.

Pre-requisite

pip install build.whl

Pyspark usage:

from streaming_connector import bq_stream_register

query=(spark.readStream.format("bigquery-streaming")
 .option("project_id", <bq_project_id>)
 .option("incremental_checkpoint_field",<table_incremental_ts_based_col>)
 .option("dataset",<bq_dataset_name>)
 .option("table",<bq_table_name>)
 .load()
 ## The above will ingest table data incrementally using the provided timestamp based field and latest value is checkpointed using offset semantics.
 ## Without the incremental input field full table ingestion is done.