kgmodule-utils

kgmodule-utils — Shared graph store, semantic index, pipeline base, and snapshot infrastructure for the KGModule SDK.

Author: Eric G. Suchanek, PhD

Flux-Frontiers, Liberty TWP, OH

Overview

kgmodule-utils is the shared SDK layer for the Flux-Frontiers knowledge-graph ecosystem. It provides everything a domain KG module needs — from type abstractions and SQLite graph storage through pluggable vector indexing (sqlite-vec; a deprecated LanceDB backend remains only for un-migrated stores) and a full build/query/pack pipeline — so domain authors implement only what is specific to their source domain.

Fleet modules use it in two ways. PyCodeKG, TypeScriptKG, and FTreeKG subclass KGModule from here and implement exactly three methods: make_extractor(), kind(), and analyze(). The rest — DocKG, MemoryKG, MetaboKG, and others — keep their own pipelines and pull in the shared pieces they need: embedders, vector backends, snapshots, synthesis.

Latest News

• 0.26.0 (2026-09-26) -- Leaves hang as the Knowledge Press web forest hangs them. New hang_leaves() puts each leaf's stalk on its nearest twig, its blade out and lifted, its face to the sky, and returns the frame so any renderer can stamp a flat species leaf in it. leaf_frames() is unchanged.

• 0.25.1 (2026-09-26) -- Two species presets fixed. The blackthorn's limbs coiled into helices and the fir grew a branch back down to its lowest whorl, both because too short an influence radius let growth random-walk. A new test bounds how far any species' limbs may wander, and checks the 0.25.0 values fail it.

• 0.25.0 (2026-09-26) -- Organic trees grow as species. New kg_utils.viz3d.species: a Habit sets the crown's envelope (dome, cone, umbrella, spindle, ...) and how the wood grows toward it -- tropism, reach, taper, a central leader, gravity droop -- and SPECIES holds nine tuned presets, oak to willow; grow_tree(..., habit=) grows as one. The data still sets the height, the sections and one crown point per chunk. bark_sweep()/bark_mesh() sweep continuous bark with texture coordinates, so trees can carry bark images. bounded_int() and require_query() now reject loosely typed input ("8", True, 3.7, a non-string query) -- a behaviour change -- and frame_tree(fov=...) fits a rounded crown tightly. Run against all 15 consumers before tagging.

• 0.24.0 (2026-09-22) -- Wiping the graph now drops the vector index it invalidates. KGModule.build_graph(wipe=True) used to leave the previous build's vectors in place, so a graph-only rebuild followed by query() could return notes the new graph no longer held; it now calls the new KGModule.drop_index(), and a query without an index says so. The 2-D renderer gains build_graph_html(..., edge_labels=False), which keeps relations on hover instead of printing them across a dense graph, and leaf_glyphs() takes one size per leaf, so leaf size can carry data as tint carries colour. Both came out of building vault_kg.

• 0.23.0 (2026-09-20) -- KGModule.query() and pack() now check their arguments before touching the index: an empty query, k outside 1-100, hop outside 0-5 or max_nodes outside 1-500 raises ValueError naming the parameter. The bounds are class attributes (max_k, max_hop, max_max_nodes, max_query_len), so a module that needs different ones sets those rather than overriding query(). They were lifted from genealogy_kg, where three repos had copied the same check by hand. KGModule.__enter__ and GraphStore.__enter__ also return Self, so with MyKG(...) as kg: narrows kg to MyKG under ty.

• 0.22.0 (2026-09-17) -- cast_scene_to_looking_glass() caps the camera sweep at 35 degrees instead of sweeping the preset's full cone. It rendered with no view_cone, so a cast swept whatever the preset carries -- 50 degrees for 16-landscape, the cone the panel can display rather than the cone that reliably fuses -- while the same repo's CLI, rendering the same scene through quiltwright, swept 35. The difference was visible only on the glass. The cap is not redeclared here: the cast path calls quiltwright's resolve_view_cone(), which caps rather than overrides, so a spec already under 35 keeps its own cone. A new view_cone argument is honored as given, including past the cap, and a narrowed cone is named in the progress message. The viz3d-qt extra now needs quiltwright>=0.14.1.

• 0.21.0 (2026-09-10) — Two fixes to state that reaches disk. SnapshotManager.save_snapshot()'s dedup branch refreshed only four fields of the manifest entry it reused, so re-saving a snapshot with a corrected subject — or any changed metrics_ignore key — updated the file and left the manifest carrying the old value, with a zero exit. snapshot list reads the manifest and snapshot show reads the file, so the two could disagree indefinitely. Both branches now build the entry from one place. SqliteVecBackend raises the new VectorStoreNotFoundError when a store is read before anything built it, instead of a bare sqlite3.OperationalError("unable to open database file") that named neither the file nor what creates it — and no longer leaves an empty database behind when the path was simply wrong.

• 0.20.0 (2026-09-08) — Six extension points on SnapshotManager, each replacing an override that two or more KG modules were carrying: package_name, dict_metric_deltas, metrics_ignore, and capture_aliases as class attributes, a _domain_metrics(stats) capture hook, and timestamp plus issues_delta in the diff_snapshots() result. Every default reproduces 0.19.1 behaviour; nothing on disk changes.

• 0.19.1 (2026-09-06) — SnapshotManager.load_snapshot() backfills a missing delta through _compute_delta_from_metrics, the extension point every KG module overrides, instead of building a two-key nodes/edges dict inline. A subclass's domain fields — coverage_delta, kinetic_params_delta, files_delta — were absent from a backfilled delta and present from list_snapshots() and diff_snapshots(), so the same snapshot reported different deltas depending on which method you asked. Nothing on disk changes and no migration is needed.

• 0.19.0 (2026-09-05) — Snapshot.key is now a caller-supplied value instead of a git tree hash read before git add staged the snapshot, which resolved for only 63 of 605 fleet snapshot keys. SnapshotManager. capture() takes key=: pass a release tag for a repo snapshot, or omit it for a corpus (no tag) to get a UTC timestamp. tree_hash stays on the record as provenance. New subject, tool, and tool_version fields on Snapshot separate what was measured from the version of the measuring tool. The manifest loader accepts all three key shapes (commit, tree_hash, key), so existing snapshots stay addressable.

• 0.18.1 (2026-08-25) — resolve_symbols() no longer guesses across classes when it doesn't have to. A dotted call stub's RESOLVES_TO edge used to match by trailing name alone, so any first-party definition sharing a method name could pick up a fabricated edge regardless of receiver. When a caller records a receiver_class in the stub's metadata (e.g. a visitor that traced a call's receiver back to a type annotation), matching now scopes to that class, and a typed stub with no match in it stays unresolved instead of falling back to the untyped guess.

• 0.18.0 (2026-08-22) — New kg_utils.temporal: a shared temporal contract so time can become a federation axis instead of a per-module convention. Modules write the same occurred_start / occurred_end / recorded_at metadata keys, and a federated query can filter and order dated nodes across all of them. Precision is preserved — "1876" stays a year and overlaps any query touching 1876 — and occurred is kept distinct from recorded, so a diary entry written tonight about last Tuesday lands on Tuesday. Also fixes NodeSpec.metadata, which the store had always dropped on write; existing databases are migrated on open.

• 0.17.0 (2026-08-18) — New kg_utils.ingest sub-package behind an ingest extra: IngestPipeline turns a folder of mixed-format documents (PDF, Word, PowerPoint, Excel, EPUB, and more) into a staged Markdown corpus any builder can consume. Conversion is anydoc; a 40-page text PDF converts in about 20 ms. Every file examined gets a manifest record — including the skips and failures — so a corpus explains its own gaps. The CLI surface is kgrag ingest in kg-rag; this package supplies the library. See Document ingestion.

• 0.16.0 (2026-08-16) — cast_scene_to_looking_glass() now returns a CastResult (path, error, elapsed, status-bar message) instead of a (path, error) tuple, and defaults its quilt spec sensibly. Breaking for callers of the tuple form.

• 0.15.0 (2026-08-16) — New viz3d-qt extra: PovRenderSession and PovRenderWorker keep POV-Ray off the GUI thread and shut down safely on window close. Tag pushes now publish to PyPI through trusted publishing.

Older changes are in the CHANGELOG.

Features

The SDK covers the full life of a knowledge graph: core contracts and SQLite storage (specs, extractor, store, pipeline), semantic indexing over pluggable vector backends (semantic, vector_backend, embed, embedder), document ingestion (ingest), a shared temporal contract for dated nodes (temporal), temporal snapshots (snapshots), text + image synthesis (synthesis), centrality analysis (analysis), and a visualization stack that runs from interactive HTML (viz) through 3-D layouts and organic trees grown as species (viz3d, viz3d.species) to Looking Glass light-field casting (viz3d.qt).

The module-by-module feature list lives in docs/features.md.

Installation

Requirements: Python ≥ 3.12, < 3.14

Core only (stdlib, no optional deps)

pip install kgmodule-utils

With semantic search (sqlite-vec + sentence-transformers)

pip install 'kgmodule-utils[semantic]'

With document ingestion (PDF, Word, PowerPoint, Excel, EPUB, …)

Only needed for non-textual sources — .md, .txt and .rst ingest with no extra dependency at all.

pip install 'kgmodule-utils[ingest]'

With deprecated LanceDB support (only for an un-migrated LanceDB store)

LanceDB is deprecated — sqlite-vec is the backend for all new stores. As of 0.10.0 lancedb is no longer part of [semantic]. Install it explicitly only if you have a pre-existing, un-migrated LanceDB store on disk; migrate when you can.

pip install 'kgmodule-utils[semantic,lancedb]'

With text + image synthesis (oMLX / Ollama / OpenAI / mflux-serve)

pip install 'kgmodule-utils[synthesis]'

With local mflux image generation (Apple Silicon, includes synthesis)

pip install 'kgmodule-utils[synthesis-mflux]'

With interactive graph rendering (pyvis / vis-network)

pip install 'kgmodule-utils[viz]'

With 3-D graph layout (numpy)

pip install 'kgmodule-utils[viz3d]'

With 3-D rendering (numpy + pyvista)

The layouts above return coordinates and draw nothing. To build meshes from them — smooth_paths, tree_mesh, leaf_glyphs — you need a renderer too:

pip install 'kgmodule-utils[viz3d-render]'

With the Qt render lifecycle (PyQt5 + quiltwright)

To ray-trace and cast to a Looking Glass display from a Qt viewer — the kg_utils.viz3d.qt worker thread, session lifecycle, and cast path:

pip install 'kgmodule-utils[viz3d-qt]'

In a Poetry project

[tool.poetry.dependencies]
kgmodule-utils = { version = ">=0.26.0", extras = ["semantic", "synthesis"] }

Quick Start

Build a domain KG module

from collections.abc import Iterator
from pathlib import Path

from kg_utils.extractor import KGExtractor
from kg_utils.pipeline import KGModule
from kg_utils.specs import EdgeSpec, NodeSpec


class MyExtractor(KGExtractor):
    def node_kinds(self) -> list[str]:
        return ["document", "section"]

    def edge_kinds(self) -> list[str]:
        return ["CONTAINS"]

    def meaningful_node_kinds(self) -> list[str]:
        return ["section"]

    def extract(self) -> Iterator[NodeSpec | EdgeSpec]:
        for doc in self.repo_path.glob("**/*.md"):
            doc_id = f"document:{doc}"
            yield NodeSpec(node_id=doc_id, kind="document",
                           name=doc.stem, qualname=doc.stem,
                           source_path=str(doc))
            # … yield sections and CONTAINS edges


class MyKG(KGModule):
    _default_dir = ".mykg"

    def make_extractor(self) -> KGExtractor:
        return MyExtractor(self.repo_root)

    def kind(self) -> str:
        return "my"

    def analyze(self) -> str:
        s = self.stats()
        return f"# MyKG\nnodes={s['total_nodes']}"


# Build and query
kg = MyKG("/path/to/repo")
kg.build(wipe=True)

result = kg.query("authentication flow", k=8, hop=1)
pack   = kg.pack("error handling", max_nodes=10)
print(pack.to_markdown())

Track metrics over time

from kg_utils.snapshots import SnapshotManager

mgr = SnapshotManager(".mykg/snapshots", package_name="my-kg")

snapshot = mgr.capture(
    version="1.0.0",
    branch="main",
    graph_stats_dict=kg.stats(),
)
mgr.save_snapshot(snapshot)

snaps = mgr.list_snapshots(limit=5)
delta = mgr.diff_snapshots(snaps[-1]["key"], snaps[0]["key"])

API Reference

kg_utils.specs

kg_utils.extractor

kg_utils.store

kg_utils.semantic

kg_utils.pipeline

kg_utils.snapshots

kg_utils.synthesis

Full reference: docs/synthesis.md

kg_utils.viz

Requires the viz extra.

kg_utils.viz3d

Requires the viz3d extra for layouts, or viz3d-render to build meshes.

A layout maps nodes and edges onto {node_id: [x, y, z]} and draws nothing, so the same layout feeds a PyVista desktop viewer, an off-screen light-field renderer, or a plain scatter plot. Which kind is a root, which relation means containment, and which kind sits on which Z level are all constructor arguments — one engine, every domain.

Organic trees — kg_utils.viz3d.organic

Lattice layouts place nodes; this half grows a skeleton toward them, so a corpus reads as wood rather than as a scatter plot. Leaf-level positions become attraction points and the branching comes from space colonization, so every limb is a real structural path and the canopy's shape is the graph's shape.

The engine takes crown attractors and a root — the hierarchy is yours to choose. A document corpus grows document → section → chunk; a diary grows trunk → one limb per year → entry cluster → leaves. Because the pipe model sizes a limb by what it carries, a prolific year grows visibly heavier wood.

The geometry above is NumPy-only and needs just viz3d. The four that return PyVista objects — smooth_paths, tree_mesh, bark_mesh, leaf_glyphs — import it lazily and raise a ModuleNotFoundError naming the install if it is absent, so reach for viz3d-render when you intend to build meshes.

from kg_utils.viz3d import grow_tree, tree_mesh

skeleton = grow_tree(chunk_positions, root=[0, 0, 0], key="pepys")
wood = tree_mesh(skeleton)          # needs pyvista; see below

Species. kg_utils.viz3d.species makes those trees look like species. A Habit sets where the crown sits — an envelope (dome, cone, umbrella, spindle, ...), width, clear bole, whorls — and how the wood grows toward it: tropism, reach, step, jitter, taper, a central leader, and gravity droop. SPECIES holds nine tuned presets (oak, chestnut, fir, plane, blackthorn, pine, birch, willow, poplar). The data still sets the height, the sections and one crown point per chunk; the habit only decides where they sit and how the wood reaches them.

from kg_utils.viz3d import SPECIES, bark_mesh, crown_sections, grow_tree, section_cluster, vary_habit

habit = vary_habit(SPECIES["willow"], key="pepys")  # vary once, use for both
tips = crown_sections(n_sections, trunk_height, branch_length, habit)
crown = np.vstack([section_cluster(n, tip, [0, 0, tip[2]], 1.5, habit) for n, tip in zip(chunks_per_section, tips)])
skeleton = grow_tree(crown, root=[0, 0, 0], key="pepys", habit=habit)
wood = bark_mesh(skeleton, aspect=2.0)  # texture coordinates for a 1:2 bark image

The viz3d extra installs NumPy only. The geometry above is pure NumPy; only smooth_paths, tree_mesh, bark_mesh and leaf_glyphs need PyVista, which they import lazily. Install pyvista alongside if you want meshes — everything else works without it.

from kg_utils.viz3d import FunnelLayout, LayoutEdge, LayoutNode

layout = FunnelLayout(
    layer_gap=12.0,
    zlevels={"document": 0, "section": 1, "chunk": 2},
    level_sizes={0: 1.8, 1: 0.9, 2: 0.28},
)
positions = layout.compute(nodes, edges)   # {node_id: np.array([x, y, z])}

Qt render lifecycle — kg_utils.viz3d.qt

Requires the viz3d-qt extra. Not re-exported from kg_utils.viz3d: these classes subclass QThread/QDialog/QObject, so importing them requires PyQt5 at class-definition time, and importing a layout must not.

The machinery a Qt viewer needs to ray-trace a scene and cast it to a Looking Glass display, factored out of pycode_kg and gutenberg_kg. Which node becomes a trunk stays per-repo; the session takes its progress bar and status callback as constructor arguments and makes no assumptions about the host window.

kg_utils.analysis

Project Structure

KG_utils/
├── pyproject.toml
├── docs/
│   ├── features.md           # Module-by-module feature list
│   ├── synthesis.md          # Synthesis sub-package reference
│   ├── encode-batch-memory-postmortem.md
│   └── viz-bootstrap-selfcontainment.md
├── src/
│   └── kg_utils/
│       ├── __init__.py
│       ├── specs.py          # NodeSpec, EdgeSpec, BuildStats, QueryResult, SnippetPack
│       ├── extractor.py      # KGExtractor ABC
│       ├── store.py          # GraphStore (SQLite)
│       ├── semantic.py       # SemanticIndex, SentenceTransformerEmbedder, SeedHit
│       ├── vector_backend.py # VectorBackend protocol, SqliteVecBackend (default), LanceDBBackend (deprecated)
│       ├── pipeline.py       # KGModule concrete base class
│       ├── module.py         # Re-export shim
│       ├── embed.py          # Embedder protocol, model registry
│       ├── embedder.py       # SentenceTransformerEmbedder factory functions
│       ├── corpus_embedder.py # CorpusEmbedder / EmbeddingCache: multi-worker corpus embedding
│       ├── ingest/          # Documents → staged Markdown corpus (ingest extra)
│       │   ├── __init__.py
│       │   ├── converters.py # Converter protocol, PassthroughConverter, AnydocConverter
│       │   ├── manifest.py   # IngestRecord, IngestManifest, IngestStats — per-file provenance
│       │   └── pipeline.py   # IngestPipeline: walk, convert, stage, dedup by content digest
│       ├── retrieval/        # Serialize + enrich KG hits (hit_to_dict, attach_content_by_sqlite)
│       │   ├── __init__.py
│       │   └── hits.py
│       ├── worker/           # RunPod /runsync client (WorkerClient, decode/handle helpers)
│       │   ├── __init__.py
│       │   ├── client.py
│       │   └── ops.py
│       ├── snapshots/
│       │   ├── __init__.py
│       │   ├── models.py     # Snapshot, SnapshotManifest, PruneResult
│       │   └── manager.py    # SnapshotManager
│       ├── synthesis/
│       │   ├── __init__.py   # Public API + factory functions
│       │   ├── _config.py    # TextBackend, ImageBackend, TextConfig, ImageConfig, env factories
│       │   ├── _text.py      # TextSynthesizer
│       │   ├── _image.py     # ImageSynthesizer
│       │   └── factory.py    # Per-request backend override helpers
│       ├── viz/              # Shared graph rendering (viz extra)
│       │   ├── __init__.py   # build_graph_html, select_nodes, GraphTheme, TooltipSpec
│       │   ├── graph_html.py # Interactive HTML renderer (vis-network inlined)
│       │   ├── theme.py      # GraphTheme, KindStyle, with_alpha
│       │   └── tooltip.py    # TooltipSpec, TooltipRow
│       ├── viz3d/            # Shared 3-D graph layout (viz3d extra)
│       │   ├── __init__.py   # Layout3D, AlliumLayout, FunnelLayout, LayoutNode, LayoutEdge
│       │   ├── layout.py     # Layout engine + Fibonacci point distributions
│       │   ├── organic.py    # Space-colonization trees + POV-Ray export (viz3d-render extra for meshes)
│       │   └── qt.py         # Qt render lifecycle: worker thread, session, cast path (viz3d-qt extra)
│       └── analysis/
│           ├── __init__.py   # load_scores, available_metrics, ScoreSet
│           └── scores.py     # Read persisted centrality out of SQLite
└── tests/                    # One test module per source module; conftest.py
                              # forces Qt onto the offscreen platform

Document ingestion

Turn a folder of mixed-format documents into a corpus any KGModule builder can consume:

from kg_utils.ingest import IngestPipeline

pipeline = IngestPipeline(staging_root="corpora/specs")
stats = pipeline.run(["~/Documents/specs", "handbook.docx"])

print(f"{stats.ingested} staged, {stats.skipped} skipped, {stats.failed} failed")

Then build over the staged corpus as usual — dockg build --repo corpora/specs.

This module is the library layer; the command-line surface lives one level up in kg-rag, whose kgrag ingest stages a source tree, registers the corpus, and runs dockg build over it in one step. kgmodule-utils itself ships no CLI.

A run rebuilds the staging corpus from nothing by default — the same contract as dockg build and pycodekg build. The corpus therefore reflects exactly the sources given: a document removed upstream does not linger as a phantom, and a converter upgrade is picked up with no special flag.

Pass update=True for the incremental path (dockg build --update, pycodekg update), which keeps what is already staged and converts only what is new. The trade is that a source deleted upstream keeps its staged copy.

Either way, sources are deduplicated by the SHA-256 of their bytes rather than their filename, so the same document arriving twice under different names is ingested once.

Every file examined is accounted for, including the ones that did not make it — anydoc performs no OCR, so scanned PDFs are skipped rather than converted, and a corpus should say so rather than simply lack them:

for record in pipeline.manifest().problems():
    print(f"{record.source_path}: {record.status} — {record.reason}")

The same records live in <staging_root>/.ingest/manifest.json, with the converter and version that produced each staged file.

For the end-to-end pipeline — converters, staging layout, manifest schema, re-run semantics, and the kgrag ingest command that drives build and registration — see KGRAG's ingestion guide.

Development

Requires Python 3.12 or 3.13 (requires-python = ">=3.12,<3.14"); CI builds on 3.12.

git clone https://github.com/Flux-Frontiers/KG_utils.git
cd KG_utils
poetry env use python3.12
poetry install --with dev --extras "semantic" --extras "synthesis" --extras "viz" \
  --extras "lancedb" --extras "viz3d-render" --extras "viz3d-qt"

The extras are not optional for testing. The core install is deliberately zero-dependency (dependencies = []), so poetry install --with dev on its own installs no runtime packages at all — pytest then aborts during collection on missing numpy and httpx and runs nothing. The six extras above are what CI installs: the test job omits lancedb (the vector-backend tests no longer need it), but the type-check job requires it so ty can resolve the legacy backend's imports — and pre-commit runs ty, so install it locally.

Run the fast test suite (no model downloads) — 750 passed, 1 skipped (the skip is a test that needs doc_kg, which the SDK cannot depend on):

poetry run pytest -m "not integration"

The Qt suite runs on the offscreen platform automatically (tests/conftest.py sets QT_QPA_PLATFORM=offscreen), so no windows appear during a test run. To see real widgets while debugging one, override it: QT_QPA_PLATFORM=cocoa poetry run pytest tests/test_viz3d_qt.py.

Run everything, including the integration marker — these download embedding models on first use and are excluded from CI:

poetry run pytest

TEIEmbedder's live tests are skipped unless a Text Embeddings Inference server is reachable; point them at one to include them:

KG_EMBED_ENDPOINT=http://localhost:8080 poetry run pytest -m integration

Citation

If you use kgmodule-utils in research or a project, please cite it:

APA

Suchanek, E. G. (2026). kgmodule-utils: Shared SDK for the KGModule Knowledge-Graph Ecosystem (Version 0.26.0) [Software]. Flux-Frontiers. https://doi.org/10.5281/zenodo.21284866

BibTeX

@software{suchanek_kgmodule_utils,
  author    = {Suchanek, Eric G.},
  title     = {{kgmodule-utils}: Shared SDK for the KGModule Knowledge-Graph Ecosystem},
  version   = {0.26.0},
  year      = {2026},
  publisher = {Flux-Frontiers},
  url       = {https://github.com/Flux-Frontiers/KG_utils},
  doi       = {10.5281/zenodo.21284866},
}

Citation metadata is also available in CITATION.cff.

License

Elastic License 2.0 — see LICENSE.

Free to use, modify, and distribute. You may not offer the software as a hosted or managed service to third parties. Commercial use internally is permitted.