Data Commons Database Module
This module provides the database models and client primitives for the Data Commons project, implementing a graph database using Google Cloud Spanner and SQLAlchemy. It defines the core data models for nodes, edges, observations, and schema migration version tracking.
Features
- Direct Cloud Spanner Client (
SpannerClient): Client for Spanner database operations, DDL execution, parameterized DML, and point-in-time snapshot queries. - SQLAlchemy ORM models: Declarative models for nodes, edges, and observations.
- Graph database implementation: Built on top of Google Cloud Spanner.
- JSON-LD document support: Model support for data import/export.
- Efficient indexing & querying: Full-text and composite indexing for graph traversals.
- Provenance tracking: Complete auditability for all graph entities and relationships.
Data Model
NodeModel
- Primary key:
subject_id(String) - Properties:
name(Text)types(Array of Strings)
- Relationships:
outgoing_edges: One-to-many relationship with EdgeModel
EdgeModel
- Composite primary key: (
subject_id,predicate,object_id,object_hash,provenance) - Properties:
object_value(Text)object_value_tokenlist(Text, full-text search)
- Relationships:
source_node: Many-to-one relationship with NodeModel
- Indexes:
EdgeByObjectValue: Index onobject_valuefor efficient lookups
Observation Model
- Composite primary key: (
variable_measured,observation_about,import_name) - Properties:
observation_period(String)measurement_method(String)unit(String)scaling_factor(String)observations(LargeBinary)provenance_url(String)
Usage
Cloud Spanner Client
The package provides SpannerClient for direct Spanner operations, DDL execution, and query execution.
Initialization
from datacommons_db.clients import SpannerClient
client = SpannerClient(
project_id="your-gcp-project",
instance_id="your-spanner-instance",
database_id="your-spanner-database",
region="us-central1",
)
Checking Tables
# Check if a specific table exists in information_schema
if not client.table_exists("Node"):
print("Node table not found")
Executing DDL Statements
execute_ddl() accepts a list of DDL statement strings and waits for Spanner Long-Running Operations (LROs) to complete, returning a DdlResult:
from datacommons_db.clients import ExecutionStatus
ddl_result = client.execute_ddl([
"""
CREATE TABLE CustomTable (
id STRING(64) NOT NULL,
name STRING(MAX)
) PRIMARY KEY (id)
""",
"CREATE TABLE TableB (id INT64) PRIMARY KEY (id)",
])
if ddl_result.status != ExecutionStatus.SUCCESS:
print(f"DDL failed: {ddl_result.error_message}")
Executing Queries & DML
from google.cloud import spanner
from datacommons_db.clients import ExecutionStatus
# Parameterized DML transaction (returns DmlResult)
dml_result = client.execute_dml(
"UPDATE CustomTable SET name = @name WHERE id = @id",
params={"name": "New Name", "id": "123"},
param_types={"name": spanner.param_types.STRING, "id": spanner.param_types.STRING},
)
if dml_result.status == ExecutionStatus.SUCCESS:
print(f"Rows affected: {dml_result.rows_affected}")
else:
print(f"DML failed: {dml_result.error_message}")
# Point-in-time Snapshot query (returns QueryResult)
query_result = client.execute_query(
"SELECT id, name FROM CustomTable WHERE id = @id",
params={"id": "123"},
param_types={"id": spanner.param_types.STRING},
)
if query_result.status == ExecutionStatus.SUCCESS:
print(f"Queried rows: {query_result.rows}")
Schema Migrations
The package provides a timestamp-based schema migration framework with SchemaMigration and MigrationRunner.
Running Migrations
from datacommons_db.clients import SpannerClient
from datacommons_db.migrations import MigrationRunner
client = SpannerClient(
project_id="your-gcp-project",
instance_id="your-spanner-instance",
database_id="your-spanner-database",
region="us-central1",
)
runner = MigrationRunner(client)
# Check applied migrations in SchemaMigrations table
applied = runner.get_applied_migrations()
print(f"Applied migrations: {applied}")
# Run all pending migrations chronologically
applied_migrations = runner.run_migrations()
for migration in applied_migrations:
print(f"Applied: {migration.creation_timestamp} ({migration.description})")
Defining a Custom Migration
Create a file in datacommons_db/migrations/migration_scripts/ named YYYYMMDDHHMMSS_<description>.py (e.g. 20260920120000_add_custom_index.py):
from datacommons_db.clients import ExecutionStatus, SpannerClient
from datacommons_db.migrations import SchemaMigration
class Migration(SchemaMigration):
description: str = "Add custom index"
creation_timestamp: str = "2026-09-20T12:00:00Z"
def roll_forward(self, spanner_client: SpannerClient) -> None:
result = spanner_client.execute_ddl([
"CREATE INDEX CustomIndex ON Node (name)"
])
if result.status != ExecutionStatus.SUCCESS:
raise RuntimeError(f"Migration failed: {result.error_message}")
SQLAlchemy ORM Usage
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from datacommons_db.models.node import NodeModel
from datacommons_db.models.edge import EdgeModel
# Initialize database connection
engine = create_engine('spanner:///projects/your-project/instances/your-instance/databases/your-database')
# Create a session
Session = sessionmaker(bind=engine)
session = Session()
# Example: Query nodes
nodes = session.query(NodeModel).filter(NodeModel.types.contains(['Person'])).limit(100).all()
Namespaces
The module supports several predefined namespaces:
rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns#rdfs: http://www.w3.org/2000/01/rdf-schema#xsd: http://www.w3.org/2001/XMLSchema#dc: https://datacommons.org/schema: https://schema.org/
Performance Considerations
- Deferred loading of
object_value_tokenlistto optimize memory usage - Proper indexing on frequently queried fields
- Efficient relationship loading using SQLAlchemy's
joinedload - Support for pagination and filtering
Dependencies
- SQLAlchemy
- Google Cloud Spanner
Contributing
When contributing to this module:
- Ensure all database operations are properly indexed
- Maintain JSON-LD compatibility
- Add appropriate type hints
- Include docstrings for all public methods
- Add tests for new functionality
License
Metadata
Release files for datacommons-db 1.1.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datacommons_db-1.1.7.tar.gz | 27.6 kB | Without provenance | Details for datacommons_db-1.1.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datacommons_db-1.1.7-py3-none-any.whl | Python 3 | none | any | Details for datacommons_db-1.1.7-py3-none-any.whl |
Total release size: 63.6 kB
Release files / datacommons_db-1.1.7.tar.gz
| Download URL | datacommons_db-1.1.7.tar.gz |
|---|---|
| Size | 27.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
66978707e150beb1d515e1e458941689caf16f0893e832311dfc83fb88b9b9fa
|
|
BLAKE2b-256 checksum How to use checksums |
c7c8eea8154c2f576bc5b8ef030abb06bdd20e7570c18719ec8dc52a51c546b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / datacommons_db-1.1.7-py3-none-any.whl
| Download URL | datacommons_db-1.1.7-py3-none-any.whl |
|---|---|
| Size | 36.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
db7d2c2b63e64e23b5caaff807a011662ea1e0de9904dc949c0ee83853017cb6
|
|
BLAKE2b-256 checksum How to use checksums |
89a82881b8b037e850abbdfec55dc837bd71a583047e2ec1d1f5a0c91e5b3040
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for datacommons-db 1.1.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.