Skip to main content

Data Commons Database Module

This module provides the database models and client primitives for the Data Commons project, implementing a graph database using Google Cloud Spanner and SQLAlchemy. It defines the core data models for nodes, edges, observations, and schema migration version tracking.

Features

  • Direct Cloud Spanner Client (SpannerClient): Client for Spanner database operations, DDL execution, parameterized DML, and point-in-time snapshot queries.
  • SQLAlchemy ORM models: Declarative models for nodes, edges, and observations.
  • Graph database implementation: Built on top of Google Cloud Spanner.
  • JSON-LD document support: Model support for data import/export.
  • Efficient indexing & querying: Full-text and composite indexing for graph traversals.
  • Provenance tracking: Complete auditability for all graph entities and relationships.

Data Model

NodeModel

  • Primary key: subject_id (String)
  • Properties:
    • name (Text)
    • types (Array of Strings)
  • Relationships:
    • outgoing_edges: One-to-many relationship with EdgeModel

EdgeModel

  • Composite primary key: (subject_id, predicate, object_id, object_hash, provenance)
  • Properties:
    • object_value (Text)
    • object_value_tokenlist (Text, full-text search)
  • Relationships:
    • source_node: Many-to-one relationship with NodeModel
  • Indexes:
    • EdgeByObjectValue: Index on object_value for efficient lookups

Observation Model

  • Composite primary key: (variable_measured, observation_about, import_name)
  • Properties:
    • observation_period (String)
    • measurement_method (String)
    • unit (String)
    • scaling_factor (String)
    • observations (LargeBinary)
    • provenance_url (String)

Usage

Cloud Spanner Client

The package provides SpannerClient for direct Spanner operations, DDL execution, and query execution.

Initialization

from datacommons_db.clients import SpannerClient

client = SpannerClient(
    project_id="your-gcp-project",
    instance_id="your-spanner-instance",
    database_id="your-spanner-database",
    region="us-central1",
)

Checking Tables

# Check if a specific table exists in information_schema
if not client.table_exists("Node"):
    print("Node table not found")

Executing DDL Statements

execute_ddl() accepts a list of DDL statement strings and waits for Spanner Long-Running Operations (LROs) to complete, returning a DdlResult:

from datacommons_db.clients import ExecutionStatus

ddl_result = client.execute_ddl([
    """
    CREATE TABLE CustomTable (
        id STRING(64) NOT NULL,
        name STRING(MAX)
    ) PRIMARY KEY (id)
    """,
    "CREATE TABLE TableB (id INT64) PRIMARY KEY (id)",
])
if ddl_result.status != ExecutionStatus.SUCCESS:
    print(f"DDL failed: {ddl_result.error_message}")

Executing Queries & DML

from google.cloud import spanner
from datacommons_db.clients import ExecutionStatus

# Parameterized DML transaction (returns DmlResult)
dml_result = client.execute_dml(
    "UPDATE CustomTable SET name = @name WHERE id = @id",
    params={"name": "New Name", "id": "123"},
    param_types={"name": spanner.param_types.STRING, "id": spanner.param_types.STRING},
)
if dml_result.status == ExecutionStatus.SUCCESS:
    print(f"Rows affected: {dml_result.rows_affected}")
else:
    print(f"DML failed: {dml_result.error_message}")

# Point-in-time Snapshot query (returns QueryResult)
query_result = client.execute_query(
    "SELECT id, name FROM CustomTable WHERE id = @id",
    params={"id": "123"},
    param_types={"id": spanner.param_types.STRING},
)
if query_result.status == ExecutionStatus.SUCCESS:
    print(f"Queried rows: {query_result.rows}")

Schema Migrations

The package provides a timestamp-based schema migration framework with SchemaMigration and MigrationRunner.

Running Migrations

from datacommons_db.clients import SpannerClient
from datacommons_db.migrations import MigrationRunner

client = SpannerClient(
    project_id="your-gcp-project",
    instance_id="your-spanner-instance",
    database_id="your-spanner-database",
    region="us-central1",
)

runner = MigrationRunner(client)

# Check applied migrations in SchemaMigrations table
applied = runner.get_applied_migrations()
print(f"Applied migrations: {applied}")

# Run all pending migrations chronologically
applied_migrations = runner.run_migrations()
for migration in applied_migrations:
    print(f"Applied: {migration.creation_timestamp} ({migration.description})")

Defining a Custom Migration

Create a file in datacommons_db/migrations/migration_scripts/ named YYYYMMDDHHMMSS_<description>.py (e.g. 20260920120000_add_custom_index.py):

from datacommons_db.clients import ExecutionStatus, SpannerClient
from datacommons_db.migrations import SchemaMigration

class Migration(SchemaMigration):
    description: str = "Add custom index"
    creation_timestamp: str = "2026-09-20T12:00:00Z"

    def roll_forward(self, spanner_client: SpannerClient) -> None:
        result = spanner_client.execute_ddl([
            "CREATE INDEX CustomIndex ON Node (name)"
        ])
        if result.status != ExecutionStatus.SUCCESS:
            raise RuntimeError(f"Migration failed: {result.error_message}")

SQLAlchemy ORM Usage

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from datacommons_db.models.node import NodeModel
from datacommons_db.models.edge import EdgeModel

# Initialize database connection
engine = create_engine('spanner:///projects/your-project/instances/your-instance/databases/your-database')

# Create a session
Session = sessionmaker(bind=engine)
session = Session()

# Example: Query nodes
nodes = session.query(NodeModel).filter(NodeModel.types.contains(['Person'])).limit(100).all()

Namespaces

The module supports several predefined namespaces:

Performance Considerations

  • Deferred loading of object_value_tokenlist to optimize memory usage
  • Proper indexing on frequently queried fields
  • Efficient relationship loading using SQLAlchemy's joinedload
  • Support for pagination and filtering

Dependencies

  • SQLAlchemy
  • Google Cloud Spanner

Contributing

When contributing to this module:

  1. Ensure all database operations are properly indexed
  2. Maintain JSON-LD compatibility
  3. Add appropriate type hints
  4. Include docstrings for all public methods
  5. Add tests for new functionality

License

Apache License 2.0

Metadata

Release files for datacommons-db 1.1.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datacommons-db 1.1.7
File Size Uploaded Provenance Actions
datacommons_db-1.1.7.tar.gz 27.6 kB Without provenance Details for datacommons_db-1.1.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datacommons-db 1.1.7
File Interpreter ABI Platform Actions
datacommons_db-1.1.7-py3-none-any.whl Python 3 none any Details for datacommons_db-1.1.7-py3-none-any.whl

Total release size: 63.6 kB

Release files / datacommons_db-1.1.7.tar.gz

Download URL datacommons_db-1.1.7.tar.gz
Size 27.6 kB
Tags Source
SHA-256 checksum
How to use checksums
66978707e150beb1d515e1e458941689caf16f0893e832311dfc83fb88b9b9fa
BLAKE2b-256 checksum
How to use checksums
c7c8eea8154c2f576bc5b8ef030abb06bdd20e7570c18719ec8dc52a51c546b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / datacommons_db-1.1.7-py3-none-any.whl

Download URL datacommons_db-1.1.7-py3-none-any.whl
Size 36.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
db7d2c2b63e64e23b5caaff807a011662ea1e0de9904dc949c0ee83853017cb6
BLAKE2b-256 checksum
How to use checksums
89a82881b8b037e850abbdfec55dc837bd71a583047e2ec1d1f5a0c91e5b3040
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"13","id":"trixie","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for datacommons-db 1.1.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page