Django Graph Search

Semantic vector search for Django — searches across FK, M2M, and reverse relations by traversing your model graph. Pluggable backends: ChromaDB, FAISS, Qdrant, pgvector.

python -m pip install "django-graph-search[chromadb]"

Why Django Graph Search?

Most Django search solutions (Haystack, Elasticsearch, full-text) treat each model in isolation. Django Graph Search builds rich search context by traversing the ORM relation graph before indexing:

• A Product becomes searchable by its category__name, tags__name, brand__description, etc. — automatically
• Uses sentence-transformers embeddings for multilingual semantic similarity
• Delta indexing — only re-index what changed
• Admin UI — semantic search inside /admin/ out of the box
• REST API — ready-to-use search endpoint

Want to check relation traversal without downloading an embedding model? The lightweight contributor test setup includes a test that creates a product with a category and tags, then verifies that their text is included in the document to be indexed.

Installation

Requires Python 3.10+ and Django 4.2+. Use a virtual environment:

python -m venv .venv
source .venv/bin/activate

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 instead. Choose the extra for your vector store; installing the base package alone does not install a vector store. Quote extras so shells such as zsh do not interpret the brackets as filename patterns.

# ChromaDB backend (recommended for local/dev)
python -m pip install "django-graph-search[chromadb]"

# FAISS backend (fast CPU similarity, no server needed)
python -m pip install "django-graph-search[faiss]"

# Qdrant backend (production, scalable)
python -m pip install "django-graph-search[qdrant]"

# pgvector (PostgreSQL extension)
python -m pip install "django-graph-search[pgvector]"

# ChromaDB with OpenAI / Cohere embeddings (configure the provider separately)
python -m pip install "django-graph-search[chromadb,openai]"
python -m pip install "django-graph-search[chromadb,cohere]"

# All backends + LangGraph
python -m pip install "django-graph-search[all]"

Download and service requirements: sentence-transformers is a base dependency, so all normal installs (including cloud extras) install its dependencies, including PyTorch. The default embedding backend downloads its model on first use and needs network access then; download time and disk usage depend on the model and platform. OpenAI/Cohere configurations avoid loading a local embedding model, but still install the base dependencies and require provider credentials, network access, and may incur usage charges. Extras add dependencies; they do not select VECTOR_STORE or EMBEDDINGS in your settings.

What's new in 0.3.5

This maintenance release improves installation instructions, distribution contents, and package validation, including dependency checks and ChromaDB/FAISS CI. See the 0.3.5 release notes.

python -m pip install "django-graph-search[chromadb]==0.3.5"

Previous release: 0.3.4

Reliability & security hardening of the stable 0.3 line. Install with:

python -m pip install "django-graph-search[chromadb]==0.3.4"

Highlights vs 0.3.3 (full detail in CHANGELOG.md and RELEASE_NOTES_0.3.4.md):

Quick Start

This walkthrough uses ChromaDB and local embeddings. Install the [chromadb] extra above first. It assumes an existing Django project with a migrated shop.Product model and some saved products, including name, description, category (FK), and tags (M2M). Replace the model label and field paths with your own; the package does not create a shop app or sample records.

1. Add to INSTALLED_APPS

INSTALLED_APPS = [
    ...,
    "django_graph_search",
]

2. Configure GRAPH_SEARCH

# settings.py
GRAPH_SEARCH = {
    "MODELS": [
        {
            "model": "shop.Product",
            # Index local fields + traverse relations with __ notation
            "fields": ["name", "description", "category__name", "tags__name"],
            "follow_relations": True,
            "relation_depth": 2,
        },
        # Or index all concrete fields (weight_fields still apply by field name):
        # {"model": "shop.Review", "fields": "__all__",
        #  "weight_fields": {"title": 2.0, "body": 1.0, "internal_note": 0.0}},
    ],
    "VECTOR_STORE": {
        "BACKEND": "django_graph_search.backends.ChromaDBBackend",
        "OPTIONS": {
            "persist_directory": "vector_db",
            "collection_name": "django_search",
        },
    },
    "EMBEDDINGS": {
        "default": {
            "BACKEND": "django_graph_search.embeddings.SentenceTransformerBackend",
            # Multilingual model — works with Russian, English, etc.
            "MODEL_NAME": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
        },
        "fast": {
            "BACKEND": "django_graph_search.embeddings.SentenceTransformerBackend",
            "MODEL_NAME": "sentence-transformers/all-MiniLM-L6-v2",
        },
    },
    "DEFAULT_EMBEDDING": "default",
    "DEFAULT_RESULTS_LIMIT": 20,
    "DELTA_INDEXING": True,
    "CACHE": {
        "BACKEND": "file",   # Options: file | redis | db
        "OPTIONS": {"path": "graph_search_cache"},
        "TTL": 86400,
    },
    # Optional REST hardening — permissions / throttling (see "Securing the REST API"):
    # "API": { ... },
}

To restrict access to the main search, streaming, and conversational HTTP endpoints, add an "API" block as described below.

Search relevance (semantic noise). Vector search scores the full string built for indexing (all configured fields plus related rows when follow_relations is true). If results feel noisy or scores look flat, narrow fields to the attributes users actually query (e.g. username, email), set follow_relations / relation_depth lower, then rebuild the index. Admin Graph Search shows a text preview of indexed text per hit and supports optional min_score (same semantics as the REST API).

3. Add URLs

# urls.py
from django.urls import path, include

urlpatterns = [
    ...,
    path("api/search/", include("django_graph_search.urls")),
]

4. Build the index

python manage.py build_search_index

This embeds the configured models' existing records. With the default local embedding backend, the first non-empty index build also downloads the model.

5. Search

Start your local Django development server:

python manage.py runserver

In another terminal:

# REST API
curl "http://127.0.0.1:8000/api/search/?q=wireless+headphones&models=shop.Product&limit=5"

# Find similar items (replace 42 with an existing product's primary key)
curl "http://127.0.0.1:8000/api/search/similar/shop.Product/42/?limit=5"

The search endpoint returns JSON with a results list. The matches depend on your records and embedding model; an empty index returns no matches. Add &min_score=0.75 only if you want to filter out lower-scoring matches. The REST endpoints are open by default: configure API access controls before exposing them outside local development.

How It Works

Django ORM Model Graph
        │
        ▼
  Relation Traversal    <- FK, M2M, reverse relations up to depth N
        │
        ▼
  Text Concatenation    <- fields + related fields merged into one document
        │
        ▼
  Sentence Transformer  <- embeddings (dimensions depend on the model)
        │
        ▼
  Vector Store          <- ChromaDB / FAISS / Qdrant
        │
        ▼
  Semantic Search       <- cosine similarity, top-K results

Python API

from django_graph_search import search, index, get_similar

# Semantic search across models
results = search("red smartphone", models=["shop.Product"], limit=5)

# Index a single instance (e.g. in a signal)
index(product_instance)

# Find similar objects
similar = get_similar(product_instance, limit=5)

REST API

Each result object includes model, pk, score (0.0–1.0 similarity), and text (the indexed document string). When min_score is used, the response also contains min_score_applied.

Query parameters (limit)

The limit parameter controls how many results are returned (where supported):

Rules:

• Must be a positive integer in the range 1–1000. Values greater than 1000 are clamped to 1000 and a warning is logged.
• Invalid values (non-numeric strings, negative numbers, booleans, etc.) produce HTTP 400 with JSON {"error": "'limit' must be a positive integer."}.
• If min_score is set on /api/search/, only results with score >= min_score are returned. The JSON body includes min_score_applied with the threshold used. Invalid values return HTTP 400.

Optional embedding backends (OpenAI / Cohere)

Instead of downloading a sentence-transformers model, you can point EMBEDDINGS at django_graph_search.embeddings.OpenAIEmbeddingBackend or django_graph_search.embeddings.CohereEmbeddingBackend (extras [openai] / [cohere]). Cohere uses asymmetric input_type: indexing uses document mode and search uses query mode (embed_batch(..., is_query=False) vs embed(..., is_query=True)).

Async indexing from signals (optional)

When AUTO_INDEX is on, saves can block on large graphs. Enable ASYNC_INDEXING to offload work:

"ASYNC_INDEXING": {
    "ENABLED": True,
    "BACKEND": "celery",  # or "thread" | "django_q"
    "CELERY_QUEUE": "search_indexing",
    "CELERY_TASK_PATH": "django_graph_search.tasks.index_instance_task",
    "CELERY_DELETE_TASK_PATH": "django_graph_search.tasks.delete_instance_task",
},

With thread, indexing runs in a bounded daemon thread pool sized by THREAD_POOL_SIZE (no retries). With celery, install Celery and register tasks; if Celery is missing, the task module falls back to synchronous execution with a warning.

Production: zero impact on unrelated requests

AUTO_INDEX hooks every post_save for models listed in MODELS. A login that updates auth.User.last_login, or any frequent save on an indexed model, can load a local sentence-transformers model and block the request thread for seconds.

Recommended for production web workers:

Signal-driven indexing and deletion run via Django's transaction.on_commit: background tasks re-read objects from the database, so they only run after the transaction commits (no lost indexing on rollback, no "object not found" races).

Example minimal fix for login latency:

GRAPH_SEARCH = {
    "AUTO_INDEX": False,
    # or keep AUTO_INDEX and offload:
    # "ASYNC_INDEXING": {"ENABLED": True, "BACKEND": "thread"},
    # "AUTO_INDEX_SKIP_UPDATE_FIELDS": ["last_login"],
    "MODELS": [
        # avoid auth.User unless required
        {"model": "shop.Product", "fields": ["name", "description"]},
    ],
}

Heavy components (vector store client, embedding backend, graph resolver) are cached once per worker process after the first search or index operation. Restart workers after changing GRAPH_SEARCH backends or embedding models — or call django_graph_search.settings.reload_settings() to re-read the configuration at runtime (useful in tests and long-lived shells).

For local sentence-transformers, run indexing in a dedicated Celery worker if web workers must stay lean.

Securing the REST API (optional)

Scope: Settings under GRAPH_SEARCH["API"] apply to all search endpoints: GET /api/search/, /api/search/stream/, POST and DELETE /api/search/conversation/, and /api/search/similar/.../.

By default the search endpoints remain public (backward compatible). Configure GRAPH_SEARCH["API"] to add authentication, permissions, and throttling:

GRAPH_SEARCH = {
    # ... existing keys ...
    "API": {
        "REQUIRE_AUTHENTICATION": True,
        "PERMISSION_CLASSES": [
            # "rest_framework.permissions.IsAuthenticated",  # if DRF is installed
            # or a dotted path to a callable(request) -> bool
        ],
        "THROTTLE_CLASSES": [
            "django_graph_search.permissions.SimpleScopedRateThrottle",
        ],
        "THROTTLE_RATES": {
            "search": "60/minute",
            "search_authenticated": "300/minute",
        },
    },
}

SimpleScopedRateThrottle applies in-process limits (per Gunicorn worker). For accurate global limits across workers, use DRF cache-backed throttles or a reverse-proxy rate limit.

Management Commands

python manage.py build_search_index                  # Index all configured models
python manage.py build_search_index --model shop.Product  # Index one model
python manage.py clear_search_index                  # Remove all vectors
python manage.py search_index_status                 # Show index statistics
python manage.py purge_search_cache                  # Remove expired file delta cache (CACHE.BACKEND=file)
python manage.py purge_search_cache --dry-run        # Count expired entries without deleting

Admin UI

With django.contrib.admin installed, the app adds a Django Graph Search section on the admin index (/admin/) with Поиск and Статус индексации entries. The legacy URL /admin/graph-search/ still works for bookmarks and docs.

Disable the admin section and custom URLs with:

GRAPH_SEARCH = {
    # ...
    "ADMIN_SEARCH_ENABLED": False,
}

Supported Backends

FAISS persistence: by default the FAISS index lives in process memory and is lost on restart. Pass VECTOR_STORE.OPTIONS: {"persist_path": "vector_db/faiss.pkl"} to save it to disk after every mutation and reload it on startup. The file is loaded with pickle — use only a trusted local path that untrusted users cannot overwrite (a malicious pickle is remote code execution).

Re-indexing semantics: all backends use upsert semantics — re-saving an object overwrites its previous document (no duplicates, no DuplicateIDError).

Install: python -m pip install "django-graph-search[pgvector]". Table is created automatically on first use (see backend docstring for VECTOR_STORE.OPTIONS).

Delta Indexing & Cache

Enable DELTA_INDEXING: True to skip objects that haven’t changed since last index run. Choose a cache backend:

With CACHE.BACKEND: "file", each delta entry stores an expires_at timestamp derived from CACHE.TTL. Expired entries are removed lazily when read; the directory can still grow if keys are never re-read — run python manage.py purge_search_cache periodically (or via cron), or use --dry-run to count stale files without deleting. Redis/db backends use Django’s cache TTL and do not require this command; purge_search_cache only affects the file backend.

LangGraph-powered search pipeline (optional)

Starting with this version, django-graph-search ships with an optional orchestration layer built on top of LangGraph. It is disabled by default; the public API (Searcher.search, Searcher.find_similar, REST endpoints) is fully backwards-compatible.

When enabled, the pipeline runs as a small graph:

analyze_query → [expand_query] → vector_search → [rerank] → postprocess

Steps in [brackets] are toggled via settings, and each one degrades gracefully: if the LLM backend fails or is not configured, the pipeline keeps working using the deterministic vector search.

GRAPH_SEARCH = {
    # ... your existing config ...
    "LANGGRAPH": {
        "ENABLED": True,                # Master switch.
        "QUERY_EXPANSION": True,        # Generate semantic reformulations.
        "RERANKING": True,              # Rerank top-K candidates.
        "MAX_EXPANDED_QUERIES": 3,
        "RERANK_TOP_K": 20,
        "TIMEOUT_SECONDS": 15,
        "MAX_QUERY_LENGTH": 1024,
        "FALLBACK_ON_ERROR": True,      # Fall back to legacy search on graph errors.
        "USE_FOR_SIMILAR": False,       # Route find_similar through the graph.
        "LLM": {
            # Leave BACKEND=None to use the deterministic dummy backend.
            "BACKEND": None,
            "MODEL": None,
            "OPTIONS": {},
        },
    },
}

Bring your own LLM backend

Implement django_graph_search.llm.BaseLLMBackend and point LANGGRAPH.LLM.BACKEND at the dotted path. The contract is intentionally tiny — expand_query(query, models, max_variants) and rerank(query, candidates, top_k) — so you can wrap any provider (OpenAI, Ollama, vLLM, your in-house service) in a few lines.

Why optional?

The library refuses to add hard dependencies on langgraph or any LLM SDK. If langgraph is not installed, the pipeline transparently uses an in-tree sequential runner with the same node structure, so behaviour and tests stay identical.

Conversational search (optional)

For session-aware semantic search (follow-ups like "more", "only products", "similar") enable the conversational endpoint. It is a thin search-first shell on top of Searcher and never invents user intent: ambiguous follow-ups are surfaced as a structured clarification_needed flag instead of a hallucinated query.

GRAPH_SEARCH = {
    # ... existing config ...
    "CONVERSATIONAL": {
        "ENABLED": True,
        "MEMORY_BACKEND": "redis",
        "MEMORY_OPTIONS": {
            "alias": "default",        # Django CACHES alias
            "key_prefix": "dgs_conv",
            "ttl": 3600,
        },
        "MAX_HISTORY_ITEMS": 10,
        "ALLOW_CLARIFICATIONS": True,
    },
}

For local development and tests, MEMORY_BACKEND: "inmemory" is fine. With DEBUG=False (typical production), the library emits a RuntimeWarning if in-memory mode is left enabled — switch to redis (Django cache → Redis) so every Gunicorn worker shares the same conversation state.

Endpoint: POST /api/search/conversation/

// Request
{
  "query": "only products",
  "conversation_id": "abc-123",
  "models": ["shop.Product"],
  "limit": 5
}

// Response
{
  "conversation_id": "abc-123",
  "query": "only products",
  "interpreted_query": "red phone",
  "clarification_needed": false,
  "results": [...],
  "total": 5
}

Use DELETE /api/search/conversation/?conversation_id=abc-123 to clear a conversation.

Built-in memory backends:

Bring your own by subclassing BaseMemoryBackend and pointing MEMORY_BACKEND at the dotted path.

Smart indexing (optional)

The classic indexer joins selected fields with whitespace. That works, but the embedding model loses the role of each value: a category name and a body paragraph become indistinguishable tokens. The optional SmartIndexer builds structured documents with labelled sections so the embedder sees something closer to:

Title: Pixel 8
Description:
Camera-first Android phone with Tensor G3.
Category: Phones

Enable it from settings — your existing index, signals, and management command keep working because the resolver and get_indexer() factory pick the new implementation transparently:

GRAPH_SEARCH = {
    # ... your existing config ...
    "SMART_INDEXING": {
        "ENABLED": True,
        # Optional per-model templates; the indexer falls back to a heuristic
        # template based on your MODELS config when one is missing.
        "TEMPLATES": {
            "shop.Product": {
                "title_field": "name",
                "sections": [
                    {"label": "Description", "field": "description", "multiline": True},
                    {"label": "Category", "field": "category__name"},
                ],
            }
        },
    },
}

The original deterministic text is always appended as a safety net so smart indexing never produces less searchable content than the legacy pipeline. Disable the flag to fall back instantly — no reindex required to switch back.

Streaming search endpoint (optional)

Long-running pipelines (query expansion, vector search, reranking) can stream lifecycle events to the client so users see progress instead of staring at a spinner. Two transports are supported:

• ndjson (default): one JSON object per line, ideal for fetch + ReadableStream and CLI tools like jq.
• sse: Server-Sent Events for EventSource clients.

Enable from settings:

GRAPH_SEARCH = {
    # ... your existing config ...
    "STREAMING": {
        "ENABLED": True,
        "FORMAT": "ndjson",  # or "sse"
        "INCLUDE_INTERNAL_EVENTS": True,
    },
}

The endpoint is registered at /<API_URL_PREFIX>/stream/ (default /api/search/stream/) and returns HTTP 404 when disabled, so it is safe to leave the URL config untouched.

Quick test:

curl -N "http://localhost:8000/api/search/stream/?q=phone"

Example event sequence (NDJSON):

{"type": "query_received", "query": "phone"}
{"type": "vector_search_completed", "candidate_count": 12}
{"type": "completed", "total": 5}
{"type": "results", "results": [...], "total": 5}
{"type": "end"}

Under the hood the view subscribes a queue.Queue to a per-request EventHub, runs the search in a worker thread, and yields events as soon as the nodes publish them. The hub also powers structured logging and any custom subscribers you register from your own apps.

Comparison

Contributing

Pull requests are welcome! Please open an issue first to discuss significant changes.

Fork and clone the repository, then run these commands from its root in a virtual environment (activation instructions are in Installation):

git checkout -b feature/my-feature
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

# Lightweight core suite: install test requirements, not the ML dependency tree.
python -m pip install "django>=4.2" "pytest>=9.0.0" "pytest-django>=4.0" "numpy>=1.26"
python -m pip install --no-deps -e .
python -m pytest tests/ -q -ra

--no-deps is intentional only for this test environment, not a supported end-user installation. The suite uses SQLite, dummy embeddings, mocked cloud clients, and the in-tree LangGraph fallback: no API keys, model downloads, or external services are needed. NumPy is required by the mocked FAISS tests. Tests needing real ChromaDB, FAISS, or PostgreSQL skip when unavailable; skipped tests are not integration coverage.

For a small, model-free check of the relation traversal shown above:

python -m pytest tests/test_resolver.py::GraphResolverTests::test_build_searchable_text_includes_relations -q

To also run the existing local ChromaDB/FAISS integration tests, install "chromadb>=0.5.0" and "faiss-cpu>=1.7.4" with python -m pip install, then rerun the suite. This still does not exercise real embedding models or the PostgreSQL integration test.

Check both distribution formats and their metadata before submitting a PR:

python -m pip install build twine
python -m build
python -m twine check --strict dist/*

CI runs the lightweight Python/Django matrix, a separate ChromaDB/FAISS job with a normal full-dependency install and pip check, and a wheel-install smoke test for imports, metadata, templates, and static assets. Commit your changes and open a PR with the test results and any skips.

License

MIT — see LICENSE

Author

Alexander Valenchits — GitHub

Links

• 📦 PyPI Package
• 🐛 Issues
• 🤖 sentence-transformers
• 🕷️ ChromaDB