django-ragtime

Chunked, pgvector-backed embeddings for Django — built for the retrieval half of RAG. Text is split into chunks, each chunk is embedded, and the vectors live in a separate EmbeddingChunk table so your models' tables stay untouched.

Requirements

• PostgreSQL with the pgvector extension

Installation

pip install django-ragtime

Add the app (and contenttypes, which chunks depend on) to your settings and run its migrations. The first migration also enables the pgvector extension, so the database user running migrate needs permission to CREATE EXTENSION (otherwise, enable it manually first).

INSTALLED_APPS = [
    # ...
    "django.contrib.contenttypes",
    "django_ragtime",
]

python manage.py migrate django_ragtime

Usage

Add EmbeddingMixin to any of your existing models:

from django.db import models
from django_ragtime.models import EmbeddingMixin


class Article(EmbeddingMixin, models.Model):
    title = models.CharField(max_length=200)

The mixin adds no database columns, so your models need no new migrations. Embeddings live in EmbeddingChunk rows attached to your instances via a generic foreign key; they are deleted automatically when the parent instance is deleted.

To build (or rebuild) an instance's embeddings, pass update_chunks() a callable that embeds a batch of texts:

def embed_texts(texts: list[str]) -> list[list[float]]:
    """One batched call to your embedding API."""
    response = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [item.embedding for item in response.data]


article.update_chunks(embed_texts, model_name="text-embedding-3-small")

update_chunks() splits the instance's text into chunks (paragraph-aware, controlled by the chunk_size and chunk_overlap class attributes, in characters), embeds them with your callable, and atomically replaces the instance's existing chunks. article.embedding_chunks returns them.

Models you don't own

You don't have to inherit the mixin. The same storage layer works with any saved instance — User, a model from another package, etc.:

from django.contrib.auth.models import User
from django_ragtime import register, update_chunks_for

# In your AppConfig.ready(), so chunks are deleted with the parent:
register(User, fields=["first_name", "last_name"])

update_chunks_for(user, embed_texts, model_name="text-embedding-3-small")

Pass fields (or text=) whenever you call update_chunks_for if you haven't registered defaults. Auto-introspection skips emails, slugs, URLs, and names like password, but listing fields is the safe choice for third-party models.

update_chunks_for(user, embed_texts, fields=["first_name", "last_name"])
update_chunks_for(user, embed_texts, text=user.get_full_name())

chunks_for(user) and delete_chunks_for(user) are the function equivalents of instance.embedding_chunks and deleting those rows.

Similarity search

Search chunks directly, or get the best chunk per parent object:

from django_ragtime.models import EmbeddingChunk

# All matching chunks, nearest first, annotated with .distance
EmbeddingChunk.objects.search(query_vector)

# Restrict to one or more parent model classes
EmbeddingChunk.objects.search_models(query_vector, Article)
EmbeddingChunk.objects.search_models(query_vector, Article, Note)

# Same idea via chaining
EmbeddingChunk.objects.for_models(Article, Note).search(query_vector)

# At most one (best) chunk per parent object; parent via chunk.parent
for chunk in EmbeddingChunk.objects.for_model(Article).nearest_parents(query_vector, limit=10):
    print(chunk.parent, chunk.distance, chunk.text)

Getting the text to embed

get_embedding_text() returns the text an instance is embedded from. By default it concatenates every free-text field on the model (CharField/TextField), skipping slugs, URLs, emails, file paths, and choice fields. To control which fields are used (and their order), declare embedding_fields — dotted paths across relations and zero-argument methods/properties are supported:

class Note(EmbeddingMixin, models.Model):
    embedding_fields = ["subject", "body", "article.title"]

For full control (templates, related-object chunks, etc.), override the method:

def get_embedding_text(self) -> str:
    return f"{self.title}\n{self.render_summary()}"

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

The tests need Postgres with pgvector; a ready-made container is included:

docker compose up -d
pytest

The database connection can be overridden with the POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, and POSTGRES_PORT environment variables (defaults match docker-compose.yml, port 5434).

License

MIT