Skip to main content

A local file-native graph layer for relationship-aware text retrieval.

Project description

FileGraphDB

FileGraphDB builds a local relationship graph over ordinary text files so an LLM can retrieve only the most relevant files instead of reading an entire folder.

Setup

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .

Build A File Graph

filegraph --folder . build

Files longer than 2,000 words keep their file node and are also split into overlapping chunk nodes:

big_report.txt
big_report.txt#chunk-0001
big_report.txt#chunk-0002

The graph can contain:

File --CONTAINS--> Chunk
File --SEMANTICALLY_SIMILAR--> File
Chunk --SEMANTICALLY_SIMILAR--> Chunk
File/Chunk --SHARES_ENTITY--> File/Chunk
File/Chunk --SHARES_TOPIC--> File/Chunk

You can tune this:

filegraph --folder ./docs build --chunk-threshold 2000 --chunk-words 800 --chunk-overlap 120

Or disable chunking:

filegraph --folder ./docs --chunk-threshold 0 build

Show strongest relationships:

filegraph --folder . edges

Find files related to one file:

filegraph --folder . related research/file_native_graph_database.md

Retrieve likely files for an LLM query:

filegraph --folder . search "How can file relationships reduce LLM token cost?"

Print LLM-ready context:

filegraph --folder . context "How can file relationships reduce LLM token cost?"

Python Library

from filegraphdb import FileGraphDB

graph = FileGraphDB("./docs")
graph.build()

for result in graph.retrieve("What caused the project delay?", limit=4):
    print(result.document.rel_path, result.score)

The first build is the expensive step. After that, the SQLite graph lives at:

.filegraphdb.sqlite

Optional Open-Source Embedding Model

By default, FileGraphDB uses local TF-IDF + LSA semantic vectors from scikit-learn. To use a stronger open-source embedding model:

pip install -e ".[models]"
filegraph --folder ./docs --use-model build

The default model is:

sentence-transformers/all-MiniLM-L6-v2

Small LLM Demo

The earlier local LLM demo is still available:

python small_llm.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filegraphdb-0.1.1.tar.gz (19.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

filegraphdb-0.1.1-py3-none-any.whl (21.0 kB view details)

Uploaded Python 3

File details

Details for the file filegraphdb-0.1.1.tar.gz.

File metadata

  • Download URL: filegraphdb-0.1.1.tar.gz
  • Upload date:
  • Size: 19.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.9.12

File hashes

Hashes for filegraphdb-0.1.1.tar.gz
Algorithm Hash digest
SHA256 ffb629bc3897fd4bc4c0f9ef8de1e612bf4872c71fe7f2d048f032c5a40a4bea
MD5 cc5f814093ec8c09c67a7e3e0b149a6d
BLAKE2b-256 b4a33bb7e19d5ae5a1376bcafdebd34530be88bfda90f4c69735120cf7eefbee

See more details on using hashes here.

File details

Details for the file filegraphdb-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: filegraphdb-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 21.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.9.12

File hashes

Hashes for filegraphdb-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 8011812a14d312ae204508a00becc3248f3b23c5f667281b0956313ba164d52c
MD5 922016e7e1457b641c399bb23a4f10ca
BLAKE2b-256 8f7425151c3b27a39895861b1fd45bb532b99cde3e807c6416ccf28af04e76e1

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page