kgmodule-utils

kgmodule-utils — Shared graph store, semantic index, pipeline base, and snapshot infrastructure for the KGModule SDK.

Author: Eric G. Suchanek, PhD

Flux-Frontiers, Liberty TWP, OH

Overview

kgmodule-utils is the shared SDK layer for the Flux-Frontiers knowledge-graph ecosystem. It provides everything a domain KG module needs — from type abstractions and SQLite graph storage through pluggable vector indexing (sqlite-vec; a deprecated LanceDB backend remains only for un-migrated stores) and a full build/query/pack pipeline — so domain authors implement only what is specific to their source domain.

Fleet modules use it in two ways. PyCodeKG, TypeScriptKG, and FTreeKG subclass KGModule from here and implement exactly three methods: make_extractor(), kind(), and analyze(). The rest — DocKG, MemoryKG, MetaboKG, and others — keep their own pipelines and pull in the shared pieces they need: embedders, vector backends, snapshots, synthesis.

Latest News

• 0.17.0 (2026-08-18) — New kg_utils.ingest sub-package behind an ingest extra: IngestPipeline turns a folder of mixed-format documents (PDF, Word, PowerPoint, Excel, EPUB, and more) into a staged Markdown corpus any builder can consume. Conversion is anydoc; a 40-page text PDF converts in about 20 ms. Every file examined gets a manifest record — including the skips and failures — so a corpus explains its own gaps. The CLI surface is kgrag ingest in kg-rag; this package supplies the library. See Document ingestion.
• 0.16.0 (2026-08-16) — cast_scene_to_looking_glass() now returns a CastResult (path, error, elapsed, status-bar message) instead of a (path, error) tuple, and defaults its quilt spec sensibly. Breaking for callers of the tuple form.
• 0.15.0 (2026-08-16) — New viz3d-qt extra: PovRenderSession and PovRenderWorker keep POV-Ray off the GUI thread and shut down safely on window close. Tag pushes now publish to PyPI through trusted publishing.

Older changes are in the CHANGELOG.

Features

The SDK covers the full life of a knowledge graph: core contracts and SQLite storage (specs, extractor, store, pipeline), semantic indexing over pluggable vector backends (semantic, vector_backend, embed, embedder), document ingestion (ingest), temporal snapshots (snapshots), text + image synthesis (synthesis), centrality analysis (analysis), and a visualization stack that runs from interactive HTML (viz) through 3-D layouts and organic trees (viz3d) to Looking Glass light-field casting (viz3d.qt).

The module-by-module feature list lives in docs/features.md.

Installation

Requirements: Python ≥ 3.12, < 3.14

Core only (stdlib, no optional deps)

pip install kgmodule-utils

With semantic search (sqlite-vec + sentence-transformers)

pip install 'kgmodule-utils[semantic]'

With document ingestion (PDF, Word, PowerPoint, Excel, EPUB, …)

Only needed for non-textual sources — .md, .txt and .rst ingest with no extra dependency at all.

pip install 'kgmodule-utils[ingest]'

With deprecated LanceDB support (only for an un-migrated LanceDB store)

LanceDB is deprecated — sqlite-vec is the backend for all new stores. As of 0.10.0 lancedb is no longer part of [semantic]. Install it explicitly only if you have a pre-existing, un-migrated LanceDB store on disk; migrate when you can.

pip install 'kgmodule-utils[semantic,lancedb]'

With text + image synthesis (oMLX / Ollama / OpenAI / mflux-serve)

pip install 'kgmodule-utils[synthesis]'

With local mflux image generation (Apple Silicon, includes synthesis)

pip install 'kgmodule-utils[synthesis-mflux]'

With interactive graph rendering (pyvis / vis-network)

pip install 'kgmodule-utils[viz]'

With 3-D graph layout (numpy)

pip install 'kgmodule-utils[viz3d]'

With 3-D rendering (numpy + pyvista)

The layouts above return coordinates and draw nothing. To build meshes from them — smooth_paths, tree_mesh, leaf_glyphs — you need a renderer too:

pip install 'kgmodule-utils[viz3d-render]'

With the Qt render lifecycle (PyQt5 + quiltwright)

To ray-trace and cast to a Looking Glass display from a Qt viewer — the kg_utils.viz3d.qt worker thread, session lifecycle, and cast path:

pip install 'kgmodule-utils[viz3d-qt]'

In a Poetry project

[tool.poetry.dependencies]
kgmodule-utils = { version = ">=0.4.0", extras = ["semantic", "synthesis"] }

Quick Start

Build a domain KG module

from collections.abc import Iterator
from pathlib import Path

from kg_utils.extractor import KGExtractor
from kg_utils.pipeline import KGModule
from kg_utils.specs import EdgeSpec, NodeSpec


class MyExtractor(KGExtractor):
    def node_kinds(self) -> list[str]:
        return ["document", "section"]

    def edge_kinds(self) -> list[str]:
        return ["CONTAINS"]

    def meaningful_node_kinds(self) -> list[str]:
        return ["section"]

    def extract(self) -> Iterator[NodeSpec | EdgeSpec]:
        for doc in self.repo_path.glob("**/*.md"):
            doc_id = f"document:{doc}"
            yield NodeSpec(node_id=doc_id, kind="document",
                           name=doc.stem, qualname=doc.stem,
                           source_path=str(doc))
            # … yield sections and CONTAINS edges


class MyKG(KGModule):
    _default_dir = ".mykg"

    def make_extractor(self) -> KGExtractor:
        return MyExtractor(self.repo_root)

    def kind(self) -> str:
        return "my"

    def analyze(self) -> str:
        s = self.stats()
        return f"# MyKG\nnodes={s['total_nodes']}"


# Build and query
kg = MyKG("/path/to/repo")
kg.build(wipe=True)

result = kg.query("authentication flow", k=8, hop=1)
pack   = kg.pack("error handling", max_nodes=10)
print(pack.to_markdown())

Track metrics over time

from kg_utils.snapshots import SnapshotManager

mgr = SnapshotManager(".mykg/snapshots", package_name="my-kg")

snapshot = mgr.capture(
    version="1.0.0",
    branch="main",
    graph_stats_dict=kg.stats(),
)
mgr.save_snapshot(snapshot)

snaps = mgr.list_snapshots(limit=5)
delta = mgr.diff_snapshots(snaps[-1]["key"], snaps[0]["key"])

API Reference

kg_utils.specs

kg_utils.extractor

kg_utils.store

kg_utils.semantic

kg_utils.pipeline

kg_utils.snapshots

kg_utils.synthesis

Full reference: docs/synthesis.md

kg_utils.viz

Requires the viz extra.

kg_utils.viz3d

Requires the viz3d extra for layouts, or viz3d-render to build meshes.

A layout maps nodes and edges onto {node_id: [x, y, z]} and draws nothing, so the same layout feeds a PyVista desktop viewer, an off-screen light-field renderer, or a plain scatter plot. Which kind is a root, which relation means containment, and which kind sits on which Z level are all constructor arguments — one engine, every domain.

Organic trees — kg_utils.viz3d.organic

Lattice layouts place nodes; this half grows a skeleton toward them, so a corpus reads as wood rather than as a scatter plot. Leaf-level positions become attraction points and the branching comes from space colonization, so every limb is a real structural path and the canopy's shape is the graph's shape.

The engine takes crown attractors and a root — the hierarchy is yours to choose. A document corpus grows document → section → chunk; a diary grows trunk → one limb per year → entry cluster → leaves. Because the pipe model sizes a limb by what it carries, a prolific year grows visibly heavier wood.

The geometry above is NumPy-only and needs just viz3d. The three that return PyVista objects — smooth_paths, tree_mesh, leaf_glyphs — import it lazily and raise a ModuleNotFoundError naming the install if it is absent, so reach for viz3d-render when you intend to build meshes.

from kg_utils.viz3d import grow_tree, tree_mesh

skeleton = grow_tree(chunk_positions, root=[0, 0, 0], key="pepys")
wood = tree_mesh(skeleton)          # needs pyvista; see below

The viz3d extra installs NumPy only. The geometry above is pure NumPy; only smooth_paths, tree_mesh and leaf_glyphs need PyVista, which they import lazily. Install pyvista alongside if you want meshes — everything else works without it.

from kg_utils.viz3d import FunnelLayout, LayoutEdge, LayoutNode

layout = FunnelLayout(
    layer_gap=12.0,
    zlevels={"document": 0, "section": 1, "chunk": 2},
    level_sizes={0: 1.8, 1: 0.9, 2: 0.28},
)
positions = layout.compute(nodes, edges)   # {node_id: np.array([x, y, z])}

Qt render lifecycle — kg_utils.viz3d.qt

Requires the viz3d-qt extra. Not re-exported from kg_utils.viz3d: these classes subclass QThread/QDialog/QObject, so importing them requires PyQt5 at class-definition time, and importing a layout must not.

The machinery a Qt viewer needs to ray-trace a scene and cast it to a Looking Glass display, factored out of pycode_kg and gutenberg_kg. Which node becomes a trunk stays per-repo; the session takes its progress bar and status callback as constructor arguments and makes no assumptions about the host window.

kg_utils.analysis

Project Structure

KG_utils/
├── pyproject.toml
├── docs/
│   ├── features.md           # Module-by-module feature list
│   ├── synthesis.md          # Synthesis sub-package reference
│   ├── encode-batch-memory-postmortem.md
│   └── viz-bootstrap-selfcontainment.md
├── src/
│   └── kg_utils/
│       ├── __init__.py
│       ├── specs.py          # NodeSpec, EdgeSpec, BuildStats, QueryResult, SnippetPack
│       ├── extractor.py      # KGExtractor ABC
│       ├── store.py          # GraphStore (SQLite)
│       ├── semantic.py       # SemanticIndex, SentenceTransformerEmbedder, SeedHit
│       ├── vector_backend.py # VectorBackend protocol, SqliteVecBackend (default), LanceDBBackend (deprecated)
│       ├── pipeline.py       # KGModule concrete base class
│       ├── module.py         # Re-export shim
│       ├── embed.py          # Embedder protocol, model registry
│       ├── embedder.py       # SentenceTransformerEmbedder factory functions
│       ├── corpus_embedder.py # CorpusEmbedder / EmbeddingCache: multi-worker corpus embedding
│       ├── ingest/          # Documents → staged Markdown corpus (ingest extra)
│       │   ├── __init__.py
│       │   ├── converters.py # Converter protocol, PassthroughConverter, AnydocConverter
│       │   ├── manifest.py   # IngestRecord, IngestManifest, IngestStats — per-file provenance
│       │   └── pipeline.py   # IngestPipeline: walk, convert, stage, dedup by content digest
│       ├── retrieval/        # Serialize + enrich KG hits (hit_to_dict, attach_content_by_sqlite)
│       │   ├── __init__.py
│       │   └── hits.py
│       ├── worker/           # RunPod /runsync client (WorkerClient, decode/handle helpers)
│       │   ├── __init__.py
│       │   ├── client.py
│       │   └── ops.py
│       ├── snapshots/
│       │   ├── __init__.py
│       │   ├── models.py     # Snapshot, SnapshotManifest, PruneResult
│       │   └── manager.py    # SnapshotManager
│       ├── synthesis/
│       │   ├── __init__.py   # Public API + factory functions
│       │   ├── _config.py    # TextBackend, ImageBackend, TextConfig, ImageConfig, env factories
│       │   ├── _text.py      # TextSynthesizer
│       │   ├── _image.py     # ImageSynthesizer
│       │   └── factory.py    # Per-request backend override helpers
│       ├── viz/              # Shared graph rendering (viz extra)
│       │   ├── __init__.py   # build_graph_html, select_nodes, GraphTheme, TooltipSpec
│       │   ├── graph_html.py # Interactive HTML renderer (vis-network inlined)
│       │   ├── theme.py      # GraphTheme, KindStyle, with_alpha
│       │   └── tooltip.py    # TooltipSpec, TooltipRow
│       ├── viz3d/            # Shared 3-D graph layout (viz3d extra)
│       │   ├── __init__.py   # Layout3D, AlliumLayout, FunnelLayout, LayoutNode, LayoutEdge
│       │   ├── layout.py     # Layout engine + Fibonacci point distributions
│       │   ├── organic.py    # Space-colonization trees + POV-Ray export (viz3d-render extra for meshes)
│       │   └── qt.py         # Qt render lifecycle: worker thread, session, cast path (viz3d-qt extra)
│       └── analysis/
│           ├── __init__.py   # load_scores, available_metrics, ScoreSet
│           └── scores.py     # Read persisted centrality out of SQLite
└── tests/                    # One test module per source module; conftest.py
                              # forces Qt onto the offscreen platform

Document ingestion

Turn a folder of mixed-format documents into a corpus any KGModule builder can consume:

from kg_utils.ingest import IngestPipeline

pipeline = IngestPipeline(staging_root="corpora/specs")
stats = pipeline.run(["~/Documents/specs", "handbook.docx"])

print(f"{stats.ingested} staged, {stats.skipped} skipped, {stats.failed} failed")

Then build over the staged corpus as usual — dockg build --repo corpora/specs.

This module is the library layer; the command-line surface lives one level up in kg-rag, whose kgrag ingest stages a source tree, registers the corpus, and runs dockg build over it in one step. kgmodule-utils itself ships no CLI.

A run rebuilds the staging corpus from nothing by default — the same contract as dockg build and pycodekg build. The corpus therefore reflects exactly the sources given: a document removed upstream does not linger as a phantom, and a converter upgrade is picked up with no special flag.

Pass update=True for the incremental path (dockg build --update, pycodekg update), which keeps what is already staged and converts only what is new. The trade is that a source deleted upstream keeps its staged copy.

Either way, sources are deduplicated by the SHA-256 of their bytes rather than their filename, so the same document arriving twice under different names is ingested once.

Every file examined is accounted for, including the ones that did not make it — anydoc performs no OCR, so scanned PDFs are skipped rather than converted, and a corpus should say so rather than simply lack them:

for record in pipeline.manifest().problems():
    print(f"{record.source_path}: {record.status} — {record.reason}")

The same records live in <staging_root>/.ingest/manifest.json, with the converter and version that produced each staged file.

Development

Requires Python 3.12 or 3.13 (requires-python = ">=3.12,<3.14"); CI builds on 3.12.

git clone https://github.com/Flux-Frontiers/KG_utils.git
cd KG_utils
poetry env use python3.12
poetry install --with dev --extras "semantic" --extras "synthesis" --extras "viz" \
  --extras "lancedb" --extras "viz3d-render" --extras "viz3d-qt"

The extras are not optional for testing. The core install is deliberately zero-dependency (dependencies = []), so poetry install --with dev on its own installs no runtime packages at all — pytest then aborts during collection on missing numpy and httpx and runs nothing. The six extras above are what CI installs: the test job omits lancedb (the vector-backend tests no longer need it), but the type-check job requires it so ty can resolve the legacy backend's imports — and pre-commit runs ty, so install it locally.

Run the fast test suite (no model downloads) — 650 passed, 1 skipped (the skip is a test that only runs when PyVista is absent):

poetry run pytest -m "not integration"

The Qt suite runs on the offscreen platform automatically (tests/conftest.py sets QT_QPA_PLATFORM=offscreen), so no windows appear during a test run. To see real widgets while debugging one, override it: QT_QPA_PLATFORM=cocoa poetry run pytest tests/test_viz3d_qt.py.

Run everything, including the integration marker — these download embedding models on first use and are excluded from CI:

poetry run pytest

TEIEmbedder's live tests are skipped unless a Text Embeddings Inference server is reachable; point them at one to include them:

KG_EMBED_ENDPOINT=http://localhost:8080 poetry run pytest -m integration

Citation

If you use kgmodule-utils in research or a project, please cite it:

APA

Suchanek, E. G. (2026). kgmodule-utils: Shared SDK for the KGModule Knowledge-Graph Ecosystem (Version 0.17.0) [Software]. Flux-Frontiers. https://doi.org/10.5281/zenodo.21284866

BibTeX

@software{suchanek_kgmodule_utils,
  author    = {Suchanek, Eric G.},
  title     = {{kgmodule-utils}: Shared SDK for the KGModule Knowledge-Graph Ecosystem},
  version   = {0.17.0},
  year      = {2026},
  publisher = {Flux-Frontiers},
  url       = {https://github.com/Flux-Frontiers/KG_utils},
  doi       = {10.5281/zenodo.21284866},
}

Citation metadata is also available in CITATION.cff.

License

Elastic License 2.0 — see LICENSE.

Free to use, modify, and distribute. You may not offer the software as a hosted or managed service to third parties. Commercial use internally is permitted.