vnpt_data_governance

DBGraph (VNPT Data Governance SDK) helps you explore and find relevant data assets in a large, complex relational database. It introspects a schema, builds a navigable graph of tables/columns, classifies PII, flags quality issues, infers relationships, optionally enriches it with LLM-generated descriptions, and lets you search/render/traverse that graph.

This package is a standalone SDK extracted from the internal dbgraph project, packaged for reuse across teams/services (originally built by minhdenthedev, packaged as an SDK by hainamnguyen192).

Install

Core install (graph building/traversal/interactive HTML rendering; no network DB drivers or LLM client):

pip install vnpt_data_governance

Pick the extras you actually need — each one only pulls in the dependency for that piece:

pip install "vnpt_data_governance[postgres]"   # connect to postgres://... / postgresql://...
pip install "vnpt_data_governance[mysql]"      # connect to mysql://...
pip install "vnpt_data_governance[trino]"      # connect to trino://...
pip install "vnpt_data_governance[openai]"     # OAICompatibleLLM
pip install "vnpt_data_governance[search]"     # BM25SearchEngine
pip install "vnpt_data_governance[all]"        # everything above

SQLite is supported out of the box (Python's built-in sqlite3), no extra needed.

Quick start

The one-call entry point — introspects the database and returns a fully enriched graph (schema, statistics, PII classification, quality issues, normalization hints, inferred soft-FK/lineage relationships). Point it at a sqlite file (or a postgres:///mysql:///trino:// DSN) and it does the rest, including writing a JSON dump, an interactive HTML diagram, and a portable SQL export next to the source — each into its own subfolder, so the 3 output types don't pile up mixed together next to your source file(s):

import vnpt_data_governance

graph = vnpt_data_governance.analyze("data/northwind.db")
# -> data/json/northwind.json, data/html/northwind.html, data/sql/northwind.sql
#    written automatically (subfolders created if missing)

Open data/html/northwind.html directly in a browser (no server needed) — click any table/column/domain node or any relationship edge to see its full detail (description, dtype, stats, PII classification, FK/soft-FK, lineage, ...), drag/zoom the diagram, and search assets by name. Pass open_browser=True to have analyze() open it there for you as soon as it's written (the CLI does this by default - see --no-open below).

data/sql/northwind.sql is a standalone script (CREATE TABLE IF NOT EXISTS

• INSERT INTO, see SQLGraphWriter) that loads the same graph back into the user's own database as two tables (dbgraph_assets, dbgraph_links) so it can be queried/joined with SQL there — run it with sqlite3 mydb.db < data/sql/northwind.sql, psql, or mysql as appropriate.

For a network database, pass its DSN instead of a file path — same one-liner. There's no source file to write next to here, so output instead lands under a data/ folder relative to the current directory, named after the database/catalog (data/json/mydb.json, data/html/catalog-schema.html, ...):

graph = vnpt_data_governance.analyze("postgres://user:pass@host:5432/mydb")
graph = vnpt_data_governance.analyze("trino://user:pass@host:443/catalog/schema")

Pass an explicit save_json=/render_html=/save_sql= path to control where those land (or False to skip one), and db_schema= when a trino:///postgres:// DSN doesn't carry a schema segment:

graph = vnpt_data_governance.analyze(
    "trino://user:pass@host:443/catalog",  # no /schema in the DSN
    db_schema="app",
    save_json="out/graph.json",
    render_html=False,
    save_sql=False,
)

CLI reference

python -m vnpt_data_governance runs the same one-call analyze() from the terminal, no code needed — pass any source analyze() accepts and it does the rest, printing a [i/n] table (m columns) / [j/m] table.column progress line to stderr as it goes (useful on a database profiled over the network, where a single wide table can take minutes), then opens the rendered HTML diagram in your default browser:

python -m vnpt_data_governance data/northwind.db
python -m vnpt_data_governance "postgres://user:pass@host:5432/mydb"
python -m vnpt_data_governance "trino://user:pass@host:443/catalog/schema"

Flags: --db-schema SCHEMA (for a trino:///postgres:// DSN without one), --no-json / --no-html / --no-sql (skip that output), --no-open (don't open the HTML diagram in a browser), --quiet (don't print progress). On success it prints assets: N links: M and exits 0. Any failure (DB connection error, missing sqlite file, unrecognized DSN scheme, ...) is caught and printed as a single Error: ... line on stderr with exit code 1 — no Python traceback reaches the end user. Exit code 2 means a CLI usage error (missing <source>), caught by argparse before analyze() ever runs.

Full control: pass a gateway instead of a path

For anything the DSN auto-detection can't cover (a custom SQLAlchemy engine with connection-pool tuning, an engine built from a dialect this package doesn't special-case, ...), pass a ready-made SQLDataGateway instead of a string — analyze() then behaves exactly like before: no JSON/HTML/SQL unless you ask for them. SQLDataGateway wraps any SQLAlchemy Engine (built via create_engine(...) yourself, or via its own from_url(...) convenience constructor), so it works with every dialect SQLAlchemy has a driver for — sqlite/postgres/mysql/trino out of the box, anything else by just installing that dialect's driver package:

from pathlib import Path
from vnpt_data_governance import OAICompatibleLLM

graph = vnpt_data_governance.analyze(
    vnpt_data_governance.SQLDataGateway.from_url("sqlite:///data/northwind.db"),
    llm=OAICompatibleLLM(model=..., base_url=..., api_key=...),  # requires the `openai` extra
    save_json=Path("data/northwind-graph.json"),
    search_index_dir=Path("data/northwind-index"),  # requires the `search` extra
)

# load a previously saved graph back later
graph = vnpt_data_governance.JSONGraphLoader(json_path=Path("data/northwind-graph.json")).load()

# ...and re-render its interactive HTML straight from that JSON, without rebuilding the graph
vnpt_data_governance.GraphRenderer.render_json_file(
    "data/northwind-graph.json", "data/northwind-graph.html"
)

# search it (requires the `search` extra)
search_engine = vnpt_data_governance.BM25SearchEngine(Path("data/northwind-index"))
asset_ids = search_engine.search("restricted PII columns in the Sales domain")

Advanced: composing the pipeline yourself

analyze() is a thin wrapper around individually usable pieces — reach for these instead if you need custom prompts, want to skip/reorder steps, or want finer control than analyze()'s options give you:

from pathlib import Path

from vnpt_data_governance import (
    JSONGraphWriter,
    RGraphBuilder,
    SQLDataGateway,
    SoftForeignKeyDetector,
    LineageDetector,
)

gateway = SQLDataGateway.from_url("sqlite:///data/northwind.db")
graph_builder = RGraphBuilder(gateway)
graph = graph_builder.build_graph()
graph = SoftForeignKeyDetector(gateway).detect(graph)
graph = LineageDetector(gateway).detect(graph)

# optional: generate semantic descriptions for assets via an LLM, with your
# own prompts
# from vnpt_data_governance import GraphDescriptorV1, OAICompatibleLLM
#
# graph_descriptor = GraphDescriptorV1(
#     llm=OAICompatibleLLM(model=..., base_url=..., api_key=...),
#     system_prompt=..., formating_prompt=..., target_prompt=...,
# )
# graph = graph_descriptor.rfill_semantic_aspects(graph)

JSONGraphWriter(json_path=Path("data/northwind-graph.json"), indent=2).write(graph)

For visualization purposes, here is a graph saved as JSON:

{
  "assets": [
    {
      "asset_id": "8ab5a624-0596-497e-a0ee-3996d95dbe63",
      "name": "Categories",
      "type": "table",
      "aspects": {
        "schema_properties": { "name": "Categories_table_schema", "pks": ["CategoryID"], "indices": {} },
        "statistical_properties": { "name": "Categories_table_stats", "num_columns": 4, "num_rows": 8 },
        "semantic_properties": {
          "name": "Categories_semantic",
          "description": "Stores product category definitions and metadata, serving as a lookup table for classifying products in the inventory system.",
          "keywords": ["categories", "product classification", "category definitions", "inventory groups", "product types"]
        }
      }
    }
  ],
  "links": [
    {
      "link_id": "db6bea93-a02c-4426-a2db-449e4a7bba8f",
      "name": "Categories_CategoryID",
      "type": "contain",
      "source_id": "8ab5a624-0596-497e-a0ee-3996d95dbe63",
      "destination_id": "04c20046-2808-4021-bbf1-99876e0eea6e",
      "aspects": {}
    }
  ]
}

Use cases

• Manipulating database schema — build the schema graph, store it, and use it to traverse the database, find join paths, get referenced tables, ...
• Profiling database — the Aspect concept represents different kinds of properties attached to a data asset (schema, statistics, semantics, ...).
• Render graph — output a schema graph as a self-contained interactive HTML diagram (GraphRenderer) or Markdown/text to feed as LLM context.
• Export back into a database — write the graph as a portable .sql script (SQLGraphWriter) the user can load straight into their own database.
• LLM assistance — use an LLM to generate data assets' descriptions/keywords, and as input for downstream SQL generation.
• Search for data assets — search assets via BM25 indexing/retrieval, over a document built from every enriched aspect (description/keywords, PII classification, quality issues, domain), not just the description.

Architecture

DBGraph is designed to be easy to extend:

1. Core classes (entities) hold the shared business logic of database graphs (traversal, neighborhoods, ...) and core operations (building graphs, profiling databases, ...). The prefix R... stands for "Relational" (the only paradigm currently supported); D..., V..., G... are reserved for Document/Vector/Graph paradigms.
2. Interfaces (extensions) mark the parts of the system meant to be pluggable:
   • RGraphBuilder works against any RDBMS via the RDataGateway abstraction — SQLDataGateway is the bundled implementation, built on sqlalchemy.inspect()/SQLAlchemy Core so it works with SQLite/PostgreSQL/MySQL/Trino out of the box and with any other dialect SQLAlchemy has a driver for, no new gateway class needed.
   • LLM abstracts the model provider — OAICompatibleLLM is the bundled implementation ([openai] extra); bring your own by implementing LLM.generate/agenerate.
   • GraphWriter/GraphLoader abstract graph persistence — JSONGraphWriter/JSONGraphLoader are the bundled implementation.
   • GraphRenderer renders any DatabaseGraph (including one just loaded via JSONGraphLoader) as a self-contained, clickable interactive HTML schema diagram; also runnable as a script: python -m vnpt_data_governance.io.graph_renderer graph.json graph.html.
   • SQLGraphWriter (a GraphWriter) writes the graph as a portable .sql script the user can load back into their own database.
   • SearchEngine abstracts indexing/retrieval — BM25SearchEngine is the bundled implementation ([search] extra).

Development

uv sync --group dev --all-extras
uv run pytest
uv run pylint vnpt_data_governance
uv run mypy vnpt_data_governance
uv run flake8 vnpt_data_governance

Tests that talk to Postgres/MySQL/Trino/OpenAI need real credentials (see tests/) and are skipped/fail without them; the SQLite, entity and loader/writer tests run standalone.

License

MIT