Inverted Index using efficient Redis set
Project description
Redis-index: Inverted Index using efficient Redis set
Redis-index helps to delegate part of the work from database to cache. It is useful for highload projects, with complex serach logic underneath the hood.
Introduction
Suppose you have to implement a service that will fetch data for a given set of filters.
GET /api/companies?region=US¤cy=USD&search_ids=233,816,266,...
Filters may require a significant costs for the database: each of them involves joining multiple tables. By writing a solution on raw SQL, we have a risk of stumbling into database performance.
Such "heavy" queries can be precalculated, and put into redis SET. We can intersect the resulting SETs with each other, thereby greatly simplifying our SQL.
search_ids = {233, 816, 266, ...}
us_companies_ids = {266, 112, 643, ...}
usd_companies_ids = {816, 54, 8395, ...}
filtered_ids = search_ids & us_companies_ids & usd_companies_ids # intersection
...
"SELECT * from companies whrere id in {filtered_ids}"
But getting such precalculated SETS from Redis to Python memory could be another bottleneck: filters can be really large, and we don't want to transfer a lot of data between servers.
The solution is intersect these SETs directly in redis. This is exactly what redis-index library does.
Installation
Use pip
to install redis-index
.
pip install redis-index
Usage
- Declare your filters. They must inherit BaseFilter class.
from redis_index import BaseFilter
class RegionFilter(BaseFilter):
def get_ids(self, region, **kwargs) -> List[int]:
"""
get_ids should return a precalculated list of ints.
"""
with psycopg2.connect(...) as conn:
with conn.cursor() as cursor:
cursor.execute('SELECT id FROM companies WREHE region = %s', (region, ))
return cursor.fetchall()
class CurrencyFilter(BaseFilter):
def get_ids(self, currency, **kwargs):
with psycopg2.connect(...) as conn:
with conn.cursor() as cursor:
cursor.execute('SELECT id FROM companies WREHE currency = %s', (currency, ))
return cursor.fetchall()
- Initialize Filtering object
from redis_index import RedisFiltering
from hot_redis import HotClient
redis_clent = HotClient(host="localhost", port=6379)
filtering = RedisFiltering(redis_clent)
- Now you can use
filtering
as a singleton in your project. Simply callfilter()
method with specific filters, and yoursearch_ids
company_ids = request.GET["company_ids"] # input list
result = filtering.filter(search_ids, [RegionFilter("US"), CurrencyFilter("USD")])
The result will be a list, that contains only ids, that are both satisfying RegionFilter and CurrencyFilter.
How to warm the cache?
You can warm up the cache in various ways, for example, using the cron command
*/5 * * * * python warm_filters
Inside such a command, you can use specific method warm_filters
result = filtering.filter(search_ids, [RegionFilter("US"), CurrencyFilter("USD")])
Or directly RedisIndex class
for _filter in [RegionFilter("US"), CurrencyFilter("USD")]:
filter_index = RedisIndex(_filter, redis_client)
filter_index.warm()
Statsd integration
Redis-index optionally supports statsd-integration.
Code of Conduct
Everyone interacting in the project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.
History
[0.1.11] - 2019-11-08
Added
- Added code for initial release
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file redis_index-0.8.0.tar.gz
.
File metadata
- Download URL: redis_index-0.8.0.tar.gz
- Upload date:
- Size: 5.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.7.1 CPython/3.10.5 Darwin/23.2.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac5354dca5b86775ee84058bfaab493f582dca83bc642bc8ee2087c4e3201cb8 |
|
MD5 | ae0404fe61035debeb609c5b6e47b985 |
|
BLAKE2b-256 | e19d73bb9e907eb22a71497ae50f903995cc68337ba4b32f0352875b5c5be7b1 |
File details
Details for the file redis_index-0.8.0-py3-none-any.whl
.
File metadata
- Download URL: redis_index-0.8.0-py3-none-any.whl
- Upload date:
- Size: 6.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.7.1 CPython/3.10.5 Darwin/23.2.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f2e21a8b68c77c112bf91d0e29aee7bac13216bd898052725c3f0c39f67faed |
|
MD5 | 1767569289c7adfc8aa03322e0a9f8c7 |
|
BLAKE2b-256 | d54efe7e159dcbd50bcbca8d09ea3054a7c8e12da980dc49360ba28cdad5114a |