Skip to main content

Inverted Index using efficient Redis set

Project description

Redis-index: Inverted Index using efficient Redis set

Redis-index helps to delegate part of the work from database to cache. It is useful for highload projects, with complex serach logic underneath the hood.

Build Status codecov License Code style: black PyPI

Introduction

Suppose you have to implement a service that will fetch data for a given set of filters.

GET /api/companies?region=US&currency=USD&search_ids=233,816,266,...

Filters may require a significant costs for the database: each of them involves joining multiple tables. By writing a solution on raw SQL, we have a risk of stumbling into database performance.

Such "heavy" queries can be precalculated, and put into redis SET. We can intersect the resulting SETs with each other, thereby greatly simplifying our SQL.

search_ids = {233, 816, 266, ...}
us_companies_ids = {266, 112, 643, ...}
usd_companies_ids = {816, 54, 8395, ...}

filtered_ids = search_ids & us_companies_ids & usd_companies_ids  # intersection
...
"SELECT * from companies whrere id in {filtered_ids}"

But getting such precalculated SETS from Redis to Python memory could be another bottleneck: filters can be really large, and we don't want to transfer a lot of data between servers.

The solution is intersect these SETs directly in redis. This is exactly what redis-index library does.

Installation

Use pip to install redis-index.

pip install redis-index

Usage

  1. Declare your filters. They must inherit BaseFilter class.
from redis_index import BaseFilter

class RegionFilter(BaseFilter):

    def get_ids(self, region, **kwargs) -> List[int]:
        """
        get_ids should return a precalculated list of ints.
        """
        with psycopg2.connect(...) as conn:
            with conn.cursor() as cursor:
                cursor.execute('SELECT id FROM companies WREHE region = %s', (region, ))
                return cursor.fetchall()

class CurrencyFilter(BaseFilter):

    def get_ids(self, currency, **kwargs):
        with psycopg2.connect(...) as conn:
            with conn.cursor() as cursor:
                cursor.execute('SELECT id FROM companies WREHE currency = %s', (currency, ))
                return cursor.fetchall()
  1. Initialize Filtering object
from redis_index import RedisFiltering
from hot_redis import HotClient

redis_clent = HotClient(host="localhost", port=6379)
filtering = RedisFiltering(redis_clent)
  1. Now you can use filtering as a singleton in your project. Simply call filter() method with specific filters, and your search_ids
company_ids = request.GET["company_ids"]  # input list
result = filtering.filter(search_ids, [RegionFilter("US"), CurrencyFilter("USD")])

The result will be a list, that contains only ids, that are both satisfying RegionFilter and CurrencyFilter.

How to warm the cache?

You can warm up the cache in various ways, for example, using the cron command

*/5  *   *   *   *   python warm_filters

Inside such a command, you can use specific method warm_filters

result = filtering.filter(search_ids, [RegionFilter("US"), CurrencyFilter("USD")])

Or directly RedisIndex class

for _filter in [RegionFilter("US"), CurrencyFilter("USD")]:
    filter_index = RedisIndex(_filter, redis_client)
    filter_index.warm()

Statsd integration

Redis-index optionally supports statsd-integration.

Redis-Index performance

Redis-Index by filters

Code of Conduct

Everyone interacting in the project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.

History

[0.1.11] - 2019-11-08

Added

  • Added code for initial release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

redis_index-0.8.0.tar.gz (5.5 kB view details)

Uploaded Source

Built Distribution

redis_index-0.8.0-py3-none-any.whl (6.7 kB view details)

Uploaded Python 3

File details

Details for the file redis_index-0.8.0.tar.gz.

File metadata

  • Download URL: redis_index-0.8.0.tar.gz
  • Upload date:
  • Size: 5.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.1 CPython/3.10.5 Darwin/23.2.0

File hashes

Hashes for redis_index-0.8.0.tar.gz
Algorithm Hash digest
SHA256 ac5354dca5b86775ee84058bfaab493f582dca83bc642bc8ee2087c4e3201cb8
MD5 ae0404fe61035debeb609c5b6e47b985
BLAKE2b-256 e19d73bb9e907eb22a71497ae50f903995cc68337ba4b32f0352875b5c5be7b1

See more details on using hashes here.

File details

Details for the file redis_index-0.8.0-py3-none-any.whl.

File metadata

  • Download URL: redis_index-0.8.0-py3-none-any.whl
  • Upload date:
  • Size: 6.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.1 CPython/3.10.5 Darwin/23.2.0

File hashes

Hashes for redis_index-0.8.0-py3-none-any.whl
Algorithm Hash digest
SHA256 7f2e21a8b68c77c112bf91d0e29aee7bac13216bd898052725c3f0c39f67faed
MD5 1767569289c7adfc8aa03322e0a9f8c7
BLAKE2b-256 d54efe7e159dcbd50bcbca8d09ea3054a7c8e12da980dc49360ba28cdad5114a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page