Chunk large QuerySets into small chunks, and iterate over them without killing your RAM.
Project description
Chunk large QuerySets into small chunks, and iterate over them without killing your RAM.
Tested with all the combinations of:
- Python: 3.5, 3.6, 3.7, 3.8
- Django: 2, 2.1, 2.2, 3.0, master
Note
Django 3.0 is incompatible with Python 3.5, see <https://docs.djangoproject.com/en/3.0/releases/3.0/#python-compatibility>
Usage
from chunkator import chunkator for item in chunkator(LargeModel.objects.all(), 200): do_something(item)
This tool is intended to work on Django querysets.
Your model must define a pk field (this is done by default, but sometimes it can be overridden) and this pk has to be unique. django- chunkator has been tested with PostgreSQL and SQLite, using regular PKs and UUIDs as primary keys.
You can also use values():
from chunkator import chunkator for item in chunkator(LargeModel.objects.values('pk', 'name'), 200): do_something(item)
Important
If you’re using values() you have to add at least your “pk” field to the values, otherwise, the chunkator will throw a MissingPkFieldException.
Warning
This will not accelerate your process. Instead of having one BIG query, you’ll have several small queries. This will save your RAM instead, because you’ll not load a huge queryset result before looping on it.
If you want to manipulate the pages directly, you can use chunkator_page:
from chunkator import chunkator_page queryset = LargeModel.objects.all().values('pk') for page in chunkator_page(queryset, 200): launch_some_task([item['pk'] for item in page])
License
MIT License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size django_chunkator-2.0.0-py3-none-any.whl (3.3 kB) | File type Wheel | Python version py3 | Upload date | Hashes View |
Filename, size django-chunkator-2.0.0.tar.gz (3.6 kB) | File type Source | Python version None | Upload date | Hashes View |
Hashes for django_chunkator-2.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 210713faf05d68035d067de9b015868b3d902d5405865401384a3c134f64c43a |
|
MD5 | 4eb33cdff127c777d6d7f080e97d0904 |
|
BLAKE2-256 | 1e20c3b02663d901ddbdbc89a7b98de4cdae768214b0931921a14953ef058eb0 |