kgmodule-utils

kgmodule-utils — Shared graph store, semantic index, pipeline base, and snapshot infrastructure for the KGModule SDK.

Author: Eric G. Suchanek, PhD

Flux-Frontiers, Liberty TWP, OH

Overview

kgmodule-utils is the shared SDK layer for the Flux-Frontiers knowledge-graph ecosystem. It provides everything a domain KG module needs — from type abstractions and SQLite graph storage through pluggable vector indexing (sqlite-vec by default, LanceDB optional) and a full build/query/pack pipeline — so domain authors implement only what is specific to their source domain.

Every KGModule implementation — PyCodeKG, DocKG, and others — subclasses KGModule from here and implements exactly three methods: make_extractor(), kind(), and analyze().

Features

• kg_utils.specs — NodeSpec, EdgeSpec, BuildStats, QueryResult, SnippetPack dataclasses
• kg_utils.extractor — KGExtractor ABC: extract(), node_kinds(), edge_kinds(), coverage_metric()
• kg_utils.store — GraphStore: SQLite-backed node/edge store with BFS expansion, symbol resolution, caller lookup, and provenance recording
• kg_utils.semantic — SemanticIndex, SentenceTransformerEmbedder, SeedHit, model registry, resolve_model_path()
• kg_utils.vector_backend — VectorBackend protocol with SqliteVecBackend (default, exact recall) and LanceDBBackend (legacy); make_backend(), resolve_backend_name()
• kg_utils.pipeline — KGModule: full build → query → pack pipeline base with hybrid semantic + lexical reranking and snippet extraction
• kg_utils.embedder — get_embedder(), wrap_embedder(), load_sentence_transformer() factory functions
• kg_utils.embed — Embedder protocol, DEFAULT_MODEL, KNOWN_MODELS, resolve_model_path()
• kg_utils.snapshots — Snapshot, SnapshotManager, SnapshotManifest for temporal metric tracking
• kg_utils.synthesis — Unified text + image synthesis: oMLX, Ollama, and OpenAI text backends; mflux-local, mflux-serve, and DALL-E image backends; all env-var configurable
• kg_utils.viz — Shared interactive-HTML graph rendering (viz extra): build_graph_html(), select_nodes(), GraphTheme, TooltipSpec — one renderer for code, document, and metabolic graphs, with domain differences supplied as data
• kg_utils.viz3d — Shared 3-D graph layout (viz3d extra): Layout3D, AlliumLayout, FunnelLayout, LayoutNode, LayoutEdge — coordinates only, no renderer, so each module keeps its own viewer and shares the spatial reasoning. kg_utils.viz3d.organic adds space-colonization tree skeletons (grow_tree, tree_mesh) for corpora that should read as wood rather than as a scatter plot
• kg_utils.viz3d.qt — Qt render lifecycle for light-field output (viz3d-qt extra): PovRenderSession and PovRenderWorker keep POV-Ray off the GUI thread and clean up safely on window close, ImagePopup previews the result, and cast_scene_to_looking_glass runs the build → render → write → cast path to a Looking Glass display
• kg_utils.analysis — Read persisted centrality back out of SQLite: load_scores(), available_metrics(), ScoreSet (raw score, dense rank, percentile, range scaling); stdlib only

Installation

Requirements: Python ≥ 3.12, < 3.14

Core only (stdlib, no optional deps)

pip install kgmodule-utils

With semantic search (sqlite-vec + sentence-transformers)

pip install 'kgmodule-utils[semantic]'

With legacy LanceDB support (only for an un-migrated LanceDB store)

As of 0.10.0 lancedb is no longer part of [semantic]. Install it explicitly only if you have a pre-existing, un-migrated LanceDB store on disk.

pip install 'kgmodule-utils[semantic,lancedb]'

With text + image synthesis (oMLX / Ollama / OpenAI / mflux-serve)

pip install 'kgmodule-utils[synthesis]'

With local mflux image generation (Apple Silicon, includes synthesis)

pip install 'kgmodule-utils[synthesis-mflux]'

With interactive graph rendering (pyvis / vis-network)

pip install 'kgmodule-utils[viz]'

With 3-D graph layout (numpy)

pip install 'kgmodule-utils[viz3d]'

With 3-D rendering (numpy + pyvista)

The layouts above return coordinates and draw nothing. To build meshes from them — smooth_paths, tree_mesh, leaf_glyphs — you need a renderer too:

pip install 'kgmodule-utils[viz3d-render]'

With the Qt render lifecycle (PyQt5 + quiltwright)

To ray-trace and cast to a Looking Glass display from a Qt viewer — the kg_utils.viz3d.qt worker thread, session lifecycle, and cast path:

pip install 'kgmodule-utils[viz3d-qt]'

In a Poetry project

[tool.poetry.dependencies]
kgmodule-utils = { version = ">=0.4.0", extras = ["semantic", "synthesis"] }

Quick Start

Build a domain KG module

from collections.abc import Iterator
from pathlib import Path

from kg_utils.extractor import KGExtractor
from kg_utils.pipeline import KGModule
from kg_utils.specs import EdgeSpec, NodeSpec


class MyExtractor(KGExtractor):
    def node_kinds(self) -> list[str]:
        return ["document", "section"]

    def edge_kinds(self) -> list[str]:
        return ["CONTAINS"]

    def meaningful_node_kinds(self) -> list[str]:
        return ["section"]

    def extract(self) -> Iterator[NodeSpec | EdgeSpec]:
        for doc in self.repo_path.glob("**/*.md"):
            doc_id = f"document:{doc}"
            yield NodeSpec(node_id=doc_id, kind="document",
                           name=doc.stem, qualname=doc.stem,
                           source_path=str(doc))
            # … yield sections and CONTAINS edges


class MyKG(KGModule):
    _default_dir = ".mykg"

    def make_extractor(self) -> KGExtractor:
        return MyExtractor(self.repo_root)

    def kind(self) -> str:
        return "my"

    def analyze(self) -> str:
        s = self.stats()
        return f"# MyKG\nnodes={s['total_nodes']}"


# Build and query
kg = MyKG("/path/to/repo")
kg.build(wipe=True)

result = kg.query("authentication flow", k=8, hop=1)
pack   = kg.pack("error handling", max_nodes=10)
print(pack.to_markdown())

Track metrics over time

from kg_utils.snapshots import SnapshotManager

mgr = SnapshotManager(".mykg/snapshots", package_name="my-kg")

snapshot = mgr.capture(
    version="1.0.0",
    branch="main",
    graph_stats_dict=kg.stats(),
)
mgr.save_snapshot(snapshot)

snaps = mgr.list_snapshots(limit=5)
delta = mgr.diff_snapshots(snaps[-1]["key"], snaps[0]["key"])

API Reference

kg_utils.specs

kg_utils.extractor

kg_utils.store

kg_utils.semantic

kg_utils.pipeline

kg_utils.snapshots

kg_utils.synthesis

Full reference: docs/synthesis.md

kg_utils.viz

Requires the viz extra.

kg_utils.viz3d

Requires the viz3d extra for layouts, or viz3d-render to build meshes.

A layout maps nodes and edges onto {node_id: [x, y, z]} and draws nothing, so the same layout feeds a PyVista desktop viewer, an off-screen light-field renderer, or a plain scatter plot. Which kind is a root, which relation means containment, and which kind sits on which Z level are all constructor arguments — one engine, every domain.

Organic trees — kg_utils.viz3d.organic

Lattice layouts place nodes; this half grows a skeleton toward them, so a corpus reads as wood rather than as a scatter plot. Leaf-level positions become attraction points and the branching comes from space colonization, so every limb is a real structural path and the canopy's shape is the graph's shape.

The engine takes crown attractors and a root — the hierarchy is yours to choose. A document corpus grows document → section → chunk; a diary grows trunk → one limb per year → entry cluster → leaves. Because the pipe model sizes a limb by what it carries, a prolific year grows visibly heavier wood.

The geometry above is NumPy-only and needs just viz3d. The three that return PyVista objects — smooth_paths, tree_mesh, leaf_glyphs — import it lazily and raise a ModuleNotFoundError naming the install if it is absent, so reach for viz3d-render when you intend to build meshes.

from kg_utils.viz3d import grow_tree, tree_mesh

skeleton = grow_tree(chunk_positions, root=[0, 0, 0], key="pepys")
wood = tree_mesh(skeleton)          # needs pyvista; see below

The viz3d extra installs NumPy only. The geometry above is pure NumPy; only smooth_paths, tree_mesh and leaf_glyphs need PyVista, which they import lazily. Install pyvista alongside if you want meshes — everything else works without it.

from kg_utils.viz3d import FunnelLayout, LayoutEdge, LayoutNode

layout = FunnelLayout(
    layer_gap=12.0,
    zlevels={"document": 0, "section": 1, "chunk": 2},
    level_sizes={0: 1.8, 1: 0.9, 2: 0.28},
)
positions = layout.compute(nodes, edges)   # {node_id: np.array([x, y, z])}

Qt render lifecycle — kg_utils.viz3d.qt

Requires the viz3d-qt extra. Not re-exported from kg_utils.viz3d: these classes subclass QThread/QDialog/QObject, so importing them requires PyQt5 at class-definition time, and importing a layout must not.

The machinery a Qt viewer needs to ray-trace a scene and cast it to a Looking Glass display, factored out of pycode_kg and gutenberg_kg. Which node becomes a trunk stays per-repo; the session takes its progress bar and status callback as constructor arguments and makes no assumptions about the host window.

kg_utils.analysis

Project Structure

KG_utils/
├── pyproject.toml
├── docs/
│   ├── synthesis.md          # Synthesis sub-package reference
│   ├── encode-batch-memory-postmortem.md
│   └── viz-bootstrap-selfcontainment.md
├── src/
│   └── kg_utils/
│       ├── __init__.py
│       ├── specs.py          # NodeSpec, EdgeSpec, BuildStats, QueryResult, SnippetPack
│       ├── extractor.py      # KGExtractor ABC
│       ├── store.py          # GraphStore (SQLite)
│       ├── semantic.py       # SemanticIndex, SentenceTransformerEmbedder, SeedHit
│       ├── vector_backend.py # VectorBackend protocol, SqliteVecBackend (default), LanceDBBackend
│       ├── pipeline.py       # KGModule concrete base class
│       ├── module.py         # Re-export shim
│       ├── embed.py          # Embedder protocol, model registry
│       ├── embedder.py       # SentenceTransformerEmbedder factory functions
│       ├── corpus_embedder.py # CorpusEmbedder / EmbeddingCache: multi-worker corpus embedding
│       ├── retrieval/        # Serialize + enrich KG hits (hit_to_dict, attach_content_by_sqlite)
│       │   ├── __init__.py
│       │   └── hits.py
│       ├── worker/           # RunPod /runsync client (WorkerClient, decode/handle helpers)
│       │   ├── __init__.py
│       │   ├── client.py
│       │   └── ops.py
│       ├── snapshots/
│       │   ├── __init__.py
│       │   ├── models.py     # Snapshot, SnapshotManifest, PruneResult
│       │   └── manager.py    # SnapshotManager
│       ├── synthesis/
│       │   ├── __init__.py   # Public API + factory functions
│       │   ├── _config.py    # TextBackend, ImageBackend, TextConfig, ImageConfig, env factories
│       │   ├── _text.py      # TextSynthesizer
│       │   ├── _image.py     # ImageSynthesizer
│       │   └── factory.py    # Per-request backend override helpers
│       ├── viz/              # Shared graph rendering (viz extra)
│       │   ├── __init__.py   # build_graph_html, select_nodes, GraphTheme, TooltipSpec
│       │   ├── graph_html.py # Interactive HTML renderer (vis-network inlined)
│       │   ├── theme.py      # GraphTheme, KindStyle, with_alpha
│       │   └── tooltip.py    # TooltipSpec, TooltipRow
│       ├── viz3d/            # Shared 3-D graph layout (viz3d extra)
│       │   ├── __init__.py   # Layout3D, AlliumLayout, FunnelLayout, LayoutNode, LayoutEdge
│       │   ├── layout.py     # Layout engine + Fibonacci point distributions
│       │   ├── organic.py    # Space-colonization trees + POV-Ray export (viz3d-render extra for meshes)
│       │   └── qt.py         # Qt render lifecycle: worker thread, session, cast path (viz3d-qt extra)
│       └── analysis/
│           ├── __init__.py   # load_scores, available_metrics, ScoreSet
│           └── scores.py     # Read persisted centrality out of SQLite
└── tests/                    # One test module per source module; conftest.py
                              # forces Qt onto the offscreen platform

Development

Requires Python 3.12 or 3.13 (requires-python = ">=3.12,<3.14"); CI builds on 3.12.

git clone https://github.com/Flux-Frontiers/KG_utils.git
cd KG_utils
poetry env use python3.12
poetry install --with dev --extras "semantic" --extras "synthesis" --extras "viz" \
  --extras "lancedb" --extras "viz3d-render" --extras "viz3d-qt"

The extras are not optional for testing. The core install is deliberately zero-dependency (dependencies = []), so poetry install --with dev on its own installs no runtime packages at all — pytest then aborts during collection on missing numpy and httpx and runs nothing. The six extras above are what CI installs: the test job omits lancedb (the vector-backend tests no longer need it), but the type-check job requires it so ty can resolve the legacy backend's imports — and pre-commit runs ty, so install it locally.

Run the fast test suite (no model downloads) — 618 passed, 1 skipped (the skip is a test that only runs when PyVista is absent):

poetry run pytest -m "not integration"

The Qt suite runs on the offscreen platform automatically (tests/conftest.py sets QT_QPA_PLATFORM=offscreen), so no windows appear during a test run. To see real widgets while debugging one, override it: QT_QPA_PLATFORM=cocoa poetry run pytest tests/test_viz3d_qt.py.

Run everything, including the integration marker — these download embedding models on first use and are excluded from CI:

poetry run pytest

TEIEmbedder's live tests are skipped unless a Text Embeddings Inference server is reachable; point them at one to include them:

KG_EMBED_ENDPOINT=http://localhost:8080 poetry run pytest -m integration

Citation

If you use kgmodule-utils in research or a project, please cite it:

APA

Suchanek, E. G. (2026). kgmodule-utils: Shared SDK for the KGModule Knowledge-Graph Ecosystem (Version 0.16.0) [Software]. Flux-Frontiers. https://doi.org/10.5281/zenodo.21284866

BibTeX

@software{suchanek_kgmodule_utils,
  author    = {Suchanek, Eric G.},
  title     = {{kgmodule-utils}: Shared SDK for the KGModule Knowledge-Graph Ecosystem},
  version   = {0.16.0},
  year      = {2026},
  publisher = {Flux-Frontiers},
  url       = {https://github.com/Flux-Frontiers/KG_utils},
  doi       = {10.5281/zenodo.21284866},
}

Citation metadata is also available in CITATION.cff.

License

Elastic License 2.0 — see LICENSE.

Free to use, modify, and distribute. You may not offer the software as a hosted or managed service to third parties. Commercial use internally is permitted.