CloudSync MLBridge

Google Cloud Datastore to BigQuery sync toolkit with Pub/Sub events, ML scoring, and workflow-ready orchestration.

cloudsync-mlbridge helps teams move operational data from Google Cloud Datastore into BigQuery while keeping analytics tables current through event-driven sync patterns.

It is designed for modern Google Cloud architectures using Datastore, BigQuery, Pub/Sub, Google Workflows, Cloud Run, and ML-based decision systems.

Use Cases

• Datastore to BigQuery synchronization
• Pub/Sub event-driven change propagation
• BigQuery analytics table loading
• ML-assisted record scoring before analytics ingestion
• Workflow-ready sync payload generation
• Operational dashboards and reporting
• Agentic AI data movement orchestration

Install

pip install cloudsync-mlbridge

Python Usage

from cloudsync_mlbridge import SyncRecord, score_record_freshness, build_bigquery_row

record = SyncRecord(
    entity_key="customer-1001",
    kind="CustomerProfile",
    operation="UPSERT",
    data={"status": "active", "region": "US"}
)

print(score_record_freshness(record))
print(build_bigquery_row(record))

CLI Usage

cloudsync-mlbridge score --entity-key customer-1001 --kind CustomerProfile --operation UPSERT

Architecture Pattern

Recommended enterprise pattern:

1. Application writes to Datastore.
2. Application emits a Pub/Sub event.
3. Cloud Run, Dataflow, or Workflows receives the event.
4. cloudsync-mlbridge normalizes the record.
5. Optional ML scoring is applied.
6. Record is written to BigQuery raw/current tables.
7. Scheduled reconciliation compares Datastore export with BigQuery.

Build and Publish

python -m pip install --upgrade build twine
python -m build
twine check dist/*
twine upload dist/*

License

MIT

Author: Raghava Chellu