django typesense

What is it?

Faster Django Admin powered by Typesense

Quick Start Guide

Installation

pip install django-typesense

or install directly from github to test the most recent version

pip install git+https://github.com/Siege-Software/django-typesense.git

Configuration

Update your settings to include the following

• Add django_typesense to the list of installed apps.

...
INSTALLED_APPS = [
    ...
    "django_typesense"
]

• Add TYPESENSE connection details

...
TYPESENSE = {
    "api_key": "xyz",
    "nodes": [{"host": "0.0.0.0", "port": "8108", "protocol": "http"}],
    "connection_timeout_seconds": 2
}

Follow this guide to install and run typesense

Create Collections

Throughout this guide, we’ll refer to the following models, which comprise a song catalogue application:

from django.db import models


class Genre(models.Model):
    name = models.CharField(max_length=100)

    def __str__(self):
        return self.name


class Artist(models.Model):
    name = models.CharField(max_length=200)

    def __str__(self):
        return self.name


class Song(models.Model):
    title = models.CharField(max_length=100)
    genre = models.ForeignKey(Genre, on_delete=models.CASCADE)
    release_date = models.DateField(blank=True, null=True)
    artists = models.ManyToManyField(Artist)
    number_of_comments = models.IntegerField(default=0)
    number_of_views = models.IntegerField(default=0)
    duration = models.DurationField()
    description = models.TextField()

    def __str__(self):
        return self.title
     
    @property
    def release_date_timestamp(self):
        # read https://typesense.org/docs/0.25.0/api/collections.html#indexing-dates
        return self.release_date.timestamp() if self.release_date else self.release_date
      
    def artist_names(self):
        return list(self.artists.all().values_list('name', flat=True))
        

For such an application, you might be interested in improving the search and load times on the song records list view.

from django_typesense.collections import TypesenseCollection
from django_typesense import fields


class SongCollection(TypesenseCollection):
    # At least one of the indexed fields has to be provided as one of the `query_by_fields`. Must be a CharField
    query_by_fields = 'title,artist_names,genre_name'
    
    title = fields.TypesenseCharField()
    genre_name = fields.TypesenseCharField(value='genre.name')
    genre_id = fields.TypesenseSmallIntegerField()
    release_date = fields.TypesenseDateField(value='release_date_timestamp', optional=True)
    artist_names = fields.TypesenseArrayField(base_field=fields.TypesenseCharField(), value='artist_names')
    number_of_comments = fields.SmallIntegerField(index=False, optional=True)
    number_of_views = fields.SmallIntegerField(index=False, optional=True)
    duration = fields.DurationField()

It's okay to store fields that you don't intend to search but to display on the admin. Such fields should be marked as un-indexed e.g:

number_of_views = fields.SmallIntegerField(index=False, optional=True)

Update the song model as follows:

from django_typesense.mixins import TypesenseModelMixin

class Song(TypesenseModelMixin):
    ...
    collection_class = SongCollection
    ...

The TypesenseModelMixin provides a Manager that overrides the update and delete methods of the Queryset. If your model has a custom manager, make sure the custom manager inherits django_typesense.mixin.TypesenseManager

How the value of a field is retrieved from a model instance:

1. The collection field name is called as a property of the model instance
2. If value is provided, it will be called as a property or method of the model instance

Where the collections live is totally dependent on you but we recommend having a collections.py file in the django app where the model you are creating a collection for is.

Instead of this in the admin:

@admin.display('Genre')
def genre_name(self, obj):
    return obj.genre.name

Do this:

@admin.display('Genre')
def genre_name(self, obj):
    # genre_name is field in the Collection. You can also store the object url as html
    return obj.genre_name

Search Collections

Using Typesense for search

from django_typesense.utils import typesense_search

from .models import Song


def search_songs(request):
    search_term = request.GET.get("q", None)
    songs = Song.objects.all()

    if search_term:
        data = {
            "q": search_term,
            "query_by": Song.collection_class.query_by_fields,
            # Include other search parameters here
            # https://typesense.org/docs/27.1/api/search.html#search-parameters
        }
        res = typesense_search(Song.collection_class.schema_name, **data)
        ids = [result["document"]["id"] for result in res["hits"]]
        songs = songs.filter(id__in=ids)

    ...

Update Collection Schema

To add or remove fields to a collection's schema in place, update your collection then run: python manage.py updatecollections. Consider adding this to your CI/CD pipeline.

This also updates the synonyms

How updates are made to Typesense

1. Signals - django-typesense listens to signal events (post_save, pre_delete, m2m_changed) to update typesense records. If update_fields were provided in the save method, only these fields will be updated in typesense.

2. Update query - django-typesense overrides Django's QuerySet.update to make updates to typesense on the specified fields

3. Manual - You can also update typesense records manually e.g after doing a bulk_create

objs = Song.objects.bulk_create(
    [
      Song(title="Watch What I Do"),
      Song(title="Midnight City"),
   ]
)
collection = SongCollection(objs, many=True)
collection.update()

Admin Integration

To make a model admin display and search from the model's Typesense collection, the admin class should inherit TypesenseSearchAdminMixin. This also adds Live Search to your admin changelist view.

from django_typesense.admin import TypesenseSearchAdminMixin

@admin.register(Song)
class SongAdmin(TypesenseSearchAdminMixin):
    ...
    list_display = ['title', 'genre_name', 'release_date', 'number_of_views', 'duration']
    
    @admin.display(description='Genre')
    def genre_name(self, obj):
        return obj.genre.name
    ...

Indexing

For the initial setup, you will need to index in bulk. Bulk updating is multi-threaded. Depending on your system specs, you should set the batch_size keyword argument.

from django_typesense.utils import bulk_delete_typsense_records, bulk_update_typsense_records

model_qs = Song.objects.all().order_by('id')  # querysets should be ordered
bulk_update_typesense_records(model_qs, batch_size=1024)

Custom Admin Filters

To make use of custom admin filters, define a filter_by property in the filter definition. Define boolean typesense field has_views that gets it's value from a model property. This is example is not necessarily practical but for demo purposes.

# models.py
class Song(models.Model):
    ...
    @property
    def has_views(self):
        return self.number_of_views > 0
    ...

# collections.py
class SongCollection(TypesenseCollection):
    ...
    has_views = fields.TypesenseBooleanField()
    ...

class HasViewsFilter(admin.SimpleListFilter):
    title = _('Has Views')
    parameter_name = 'has_views'

    def lookups(self, request, model_admin):
        return (
            ('all', 'All'),
            ('True', 'Yes'),
            ('False', 'No')
        )

    def queryset(self, request, queryset):
        # This is used by the default django admin
        if self.value() == 'True':
            return queryset.filter(number_of_views__gt=0)
        elif self.value() == 'False':
            return queryset.filter(number_of_views=0)
            
        return queryset

    @property
    def filter_by(self):
        # This is used by typesense
        if self.value() == 'True':
            return {"has_views": "=true"}
        elif self.value() == 'False':
            return {"has_views": "!=false"}

        return {}

Note that simple lookups like the one above are done by default (hence no need to define filter_by) if the parameter_name is a field in the collection

Synonyms

The synonyms feature allows you to define search terms that should be considered equivalent. Synonyms should be defined with classes that inherit from Synonym

from django_typesense.collections import Synonym

# say you need users searching the genre hip-hop to get results if they use the search term rap

class HipHopSynonym(Synonym):
    name = 'hip-hop-synonyms'
    synonyms = ['hip-hop', 'rap']
 
# Update the collection to include the synonym
class SongCollection(TypesenseCollection):
    ...
    synonyms = [HipHopSynonym]
    ...
    

To update the collection with any changes made to synonyms run python manage.py updatecollections