Skip to main content

GraphBench

CI PyPI Python 3.10+ License: MIT HuggingFace Demo

GraphBench is an open-source Python library that benchmarks two graph-based RAG pipelines head-to-head on multi-hop question answering.

Pipeline Approach
GraphRAG Louvain community detection + Mistral-7B
GNN-RAG 3-layer GAT (PyTorch Geometric) + Mistral-7B

Dataset: 500 HotpotQA distractor questions (250 bridge, 250 comparison) Knowledge Graph: 50k REBEL triples in Neo4j AuraDB


Installation

pip install graphbench-kg

Or from source:

git clone https://github.com/Rohanjain2312/graphbench.git
cd graphbench
poetry install

Quick Start

from graphbench.utils.neo4j_client import Neo4jClient
from graphbench.utils.faiss_client import FAISSClient
from graphbench.utils.llm_client import LLMClient
from graphbench.pipelines.graphrag_pipeline import GraphRAGPipeline
from graphbench.pipelines.gnnrag_pipeline import GNNRAGPipeline
from graphbench.gnn.model import GATModel
from graphbench.utils.checkpoint import load_checkpoint
from pathlib import Path

# Init shared clients
neo4j = Neo4jClient()          # reads NEO4J_URI / NEO4J_USERNAME / NEO4J_PASSWORD from .env
faiss = FAISSClient.load()     # reads FAISS_INDEX_PATH from .env
llm   = LLMClient(backend="hf")

# GraphRAG pipeline
graphrag = GraphRAGPipeline(neo4j_client=neo4j, faiss_client=faiss, llm_client=llm)
print(graphrag.answer("Where was Marie Curie born?").predicted_answer)

# GNN-RAG pipeline (requires trained checkpoint + entity embeddings)
ckpt = load_checkpoint(Path("checkpoints/gat_best.pt"), map_location="cuda")
model = GATModel()
model.load_state_dict(ckpt["model_state_dict"])
gnnrag = GNNRAGPipeline(neo4j_client=neo4j, faiss_client=faiss, llm_client=llm,
                         gat_model=model, entity_embeddings=embedding_dict)
print(gnnrag.answer("Where was Marie Curie born?").predicted_answer)

Architecture

Architecture Diagram

See docs/architecture.md for full details.


Benchmark Results

500 HotpotQA distractor questions · seed=42 · Mistral-7B-Instruct-v0.2 (fp16) · 60k REBEL triples

Metric GraphRAG GNN-RAG
Exact Match (EM) 3.2% 5.0%
Token F1 10.5% 12.8%
Latency p50 (ms) 5,555 5,344
Latency p95 (ms) 6,388 6,199

GNN-RAG outperforms GraphRAG on all four metrics. The gap is largest on bridge questions (7.2% vs 3.6% EM), where multi-hop graph traversal benefits most from learned edge scoring.

Full results and per-type breakdown: docs/benchmark_results.md


Project Structure

graphbench/
├── graphbench/          # Core library
│   ├── ingestion/       # REBEL loading, triple extraction, FAISS/Neo4j writers
│   ├── pipelines/       # GraphRAG and GNN-RAG pipelines
│   ├── gnn/             # GAT model, dataset, trainer
│   ├── community/       # Louvain community detection and summarization
│   ├── benchmark/       # HotpotQA loader, evaluator, metrics
│   └── utils/           # Neo4j client, FAISS client, LLM client, config
├── notebooks/           # End-to-end Colab notebook
├── demo/                # Gradio HuggingFace Spaces demo
├── tests/               # Pytest test suite
└── docs/                # Documentation

Requirements

  • Python 3.10+
  • Neo4j AuraDB Free (or local Neo4j)
  • GPU recommended for GNN training (Colab Pro)
  • See .env.example for required environment variables

Contributing

See CONTRIBUTING.md.


License

MIT © 2025 Rohan Jain

Metadata

Release files for graphbench-kg 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for graphbench-kg 0.2.0
File Size Uploaded Provenance Actions
graphbench_kg-0.2.0.tar.gz 44.6 kB Without provenance Details for graphbench_kg-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for graphbench-kg 0.2.0
File Interpreter ABI Platform Actions
graphbench_kg-0.2.0-py3-none-any.whl Python 3 none any Details for graphbench_kg-0.2.0-py3-none-any.whl

Total release size: 103.0 kB

Release files / graphbench_kg-0.2.0.tar.gz

Download URL graphbench_kg-0.2.0.tar.gz
Size 44.6 kB
Tags Source
SHA-256 checksum
How to use checksums
319deca845d8463050ac998f998e08fcc8e2b12865fc6cac4d907a61f58b70f4
BLAKE2b-256 checksum
How to use checksums
0b2e1ba6d54066cb7823776ee7f5474ae312106d254088dcd27383a86d78822e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.3.2 CPython/3.12.5 Darwin/25.3.0

Release files / graphbench_kg-0.2.0-py3-none-any.whl

Download URL graphbench_kg-0.2.0-py3-none-any.whl
Size 58.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
76cd5fc206ed687e210b366339cb0cda457d40ec46c2516f59d2e2c1bfbc60b1
BLAKE2b-256 checksum
How to use checksums
36637b60ab53af9abbcfe5b6d51150a6bb5b1b4a4f69b74153ba8f45ddf3b13d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.3.2 CPython/3.12.5 Darwin/25.3.0

Security for graphbench-kg 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page