GraphWarehouse

Local DuckDB warehouse for entity–relationship graphs: ingest GraphIngestor Parquet, query bounded subgraphs, join stored node and edge measurements, convert selected graphs to iGraph, write results back, export to Gephi, and prepare toolbox input CSVs.

It is a general-purpose storage, query, analytics, iGraph, and export library. Typical work includes neighborhood and ego analysis, shared-resource structures, temporal slices, type-scoped score policies, constrained traversal, and Gephi publication.

Analytics results are named node or edge measurements produced outside GraphWarehouse. They may be anomaly scores, centrality, classifications, communities, model outputs, business metrics, or other user-defined signals. GraphWarehouse stores, discovers, filters, joins, traverses, exports, and writes those results back. It does not run external scoring algorithms.

Safety: keep work bounded. Seeds, type allowlists, hop counts, and windows limit the result. They do not make warehouse scans free. Full-warehouse iGraph conversion is in-memory and not out-of-core. Do not convert an unfiltered 50–100M-edge warehouse under a typical analyst process. Windows and 50–100M calibration remain environmental validation, not certified results.

Capabilities

Area	What you can do
Ingest	Merge GraphIngestor Parquet runs into one .duckdb with surrogate node_id / edge_id and null-aware property merge
Query	Type filters, bipartite views, ego N-hop, temporal windows, CSV / Arrow / Gephi / toolbox CSVs
Analytics	Load, list, summarize, and delete named runs; seed, filter, and constrain traversal by stored scores
iGraph	Convert a bounded GraphSlice (optional extra); write vectors back; export the in-memory graph to Gephi

Analysts: start with the cheat sheet, then the workflow guide. Building a Parquet graph from SQL is GraphETL (a separate package); GraphQuery and GraphSlice stay in this package and are not part of GraphETL. Jupyter recipes: graph query cheat sheet. Technical recipes: graph query cookbook. Lookup: API reference. Guides: analytics, iGraph. Known unfinished CLI/seed behavior: KNOWN_NOT_FROZEN.md.

Install

pip install mini-atlas-graph-warehouse

Optional in-memory graphs (igraph>=1.0,<2 and numpy>=1.26):

pip install 'mini-atlas-graph-warehouse[igraph]'

Contributor/editable installation from a clone is a secondary development path:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e /path/to/mini-atlas-graph-warehouse
python -m pip install -e "/path/to/mini-atlas-graph-warehouse[dev]"
python -m pip install -e "/path/to/mini-atlas-graph-warehouse[igraph]"

From the repository root, the development extra is:

python -m pip install -e ".[dev]"

The igraph extra is not a core dependency and is not in package __all__.

Example

from graph_warehouse import GraphQuery, GraphWarehouse

with GraphWarehouse("graph.duckdb") as wh:
    wh.ingest_runs(["data/run_a", "data/run_b"])
    print(wh.counts())

with GraphQuery("graph.duckdb") as gq:  # read-only; does not create a missing file
    with gq.ego(uids=["user:1"], hops=2).build() as slice_:
        print(slice_.counts())
        slice_.to_csv("out/ego")

GraphWarehouse owns ingest and analytics writes. GraphQuery is read-only by default. Finish and close writers before opening the same file with a differently configured connection. Prefer with for both the query session and each slice so temporary relations are dropped.

Ingest

GraphIngestor export_parquet (local-graph-ingestor-parquet version 1):

data/run_a/metadata.json
data/run_a/nodes/*.parquet    # uid, type, properties, event_ts
data/run_a/edges/*.parquet    # from_id, to_id, type, properties, event_ts
graph.duckdb

Parquet columns stay string identities. The warehouse assigns node_id / edge_id. File stem is used as type when a parquet file has no type column. properties may be STRUCT, MAP, JSON, or JSON-encoded VARCHAR.

metadata.json	Behavior
Missing, or object with no / empty format	Legacy glob of nodes/*.parquet and edges/*.parquet
format local-graph-ingestor-parquet, integer version 1	Strict v1 adapter; manifest is authoritative
That format with any other version, or any other nonempty format	Fail the run
Present but not a JSON object	Fail the run

v1 paths must be relative POSIX nodes/<file>.parquet or edges/<file>.parquet. Unlisted parquet files are skipped (WARNING skip stray parquet, stray_files in stats). Jobs run in sorted path order. Valid v1 input is assumed unique on (uid, type) and (from_id, type, to_id) inside the run; collisions with already stored rows still use NOT EXISTS and unique constraints. Legacy input keeps defensive identity GROUP BY. parquet_batch_size controls Arrow read size only.

nodes_upserted and edges_upserted count rows read from Parquet, not rows newly inserted or changed in DuckDB. Duplicate identities in a file, logical upserts, and skip_if_ingested=False reingestion of the same identities still report the Parquet row counts. They are not a net-delta of warehouse rows.

A uid cannot be ingested under two conflicting concrete node types. Conflicts raise ValueError and roll back the complete ingest. _untyped endpoint placeholders may still upgrade to one concrete type. The same uid with the same type remains idempotent. Warehouse identity is not being redesigned.

Each ingest_run is one transaction. Metadata, manifest, path, timestamp, or SQL failure rolls that run back. Prior successful runs stay.

Skip is allowed only when the stored run_id matches, the resolved export path is the same, and graph_id is compatible (equal when both sides have it; missing on both sides is compatible; present-vs-absent or two different values conflict). A successful skip is reported with skipped: true plus a reason. That skip is an identity match, not proof that file bytes are unchanged. Rewriting an export in place still skips unless you pass skip_if_ingested=False or CLI --force. Reusing a run_id for a different directory or conflicting graph_id raises ValueError and the CLI exits nonzero. --force / skip_if_ingested=False means upsert/reingest. It does not delete rows previously contributed by that run that are absent from the rewritten export; Warehouse does not have safe row-level run provenance for deletion. There is no content hash and no export_id.

Offset-aware ISO-8601 event_ts (including Z) is stored as naive UTC DuckDB TIMESTAMP. Incoming null does not wipe a stored timestamp. Naive or malformed timestamps fail the run. There is no schema migrator; re-ingest with --force to fill timestamps on older files.

Property merge: later non-null wins; JSON null is not SQL NULL (json_type(value) <> 'NULL').

Recent query performance work did not change the persistent schema. Current surrogate-key warehouses open with GraphQuery with no migration. Very old pre-surrogate databases still need a rebuild from parquet.

CLI: python -m graph_warehouse.cli or graph-warehouse after install.

python -m graph_warehouse.cli ingest --db graph.duckdb --run data/run_a --run data/run_b
python -m graph_warehouse.cli ingest --db graph.duckdb --data-dir data/
python -m graph_warehouse.cli counts --db graph.duckdb

Tables and merge details: package README. Method signatures: API reference.

Query, analytics, and export

Builder, slice lifecycle, windows, seeds, and exports: query README and API reference. CSV/Arrow/Gephi columns: export README.

Callers select an exact analytics run_id. Do not use runs[0]. Analytics observation windows (params["_provenance"]["window"]) are not warehouse edges.event_ts. Final filters and traversal constraints produce different reachable graphs. Type-scoped edge analytics can require scores on listed types while other selected types pass through.

Analytics CSV headers with supported unusual quoting (embedded ' / ") are escaped in generated SQL. A CSV or analytics-directory load that matches zero warehouse identities does not register an empty successful algorithm run. Partial matches still register and store the matching rows. Unmatched identities in CSV loads are skipped; iGraph writeback unmatched identities remain a hard error.

configure_logging configures the graph_warehouse package logger (not the root logger) and sets propagate=False so records do not also go to ancestor loggers. Library code does not configure logging by itself.

python -m graph_warehouse.cli query --db graph.duckdb --out out/ego \
  --seed-uid user:1 --hops 2 --format csv
python -m graph_warehouse.cli load-results --db graph.duckdb \
  --csv edge_results.csv --entity edge --run-id relationship_scores_2026w01 --algo external_anomaly_model

Tests and diagnostics

pytest -q
python -m pip install -e ".[igraph]"   # required for tests/test_igraph_slice.py
pytest -m e2e
GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1 pytest tests/test_ingestor_parquet_contract.py
GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1 pytest tests/e2e/test_ingestor_boundary.py

GraphIngestor is not a core or development dependency. Ordinary pytest skips those contract tests when it is missing unless GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1.

Opt-in (not CI, not Windows certification): bench/README.md, scripts/analyst_windows_smoke.md. Historical architecture notes are not the public entry point: GRAPH_WAREHOUSE_REFACTOR_PLAN.md and related plan files.