Skip to main content

Process long running django imports and exports in celery

Project description

https://img.shields.io/pypi/v/django-import-export-celery.svg

django-import-export-celery: process slow django imports and exports in celery

django-import-export-celery helps you process long running imports and exports in celery.

Basic installation

  1. Set up celery to work with your project.

  2. Add 'import_export_celery' to your INSTALLED_APPS settings variable

  3. Add 'author.middlewares.AuthorDefaultBackendMiddleware' to your MIDDLEWARE_CLASSES

  4. Configure the location of your celery module setup

    IMPORT_EXPORT_CELERY_INIT_MODULE = "projectname.celery"

Setting up imports with celery

A fully configured example project can be found in the example directory of this repository.

  1. Perform the basic setup procedure described above.

  2. Configure the IMPORT_EXPORT_CELERY_MODELS variable.

    def resource():  # Optional
        from myapp.models import WinnerResource
        return WinnerResource
    
    
    IMPORT_EXPORT_CELERY_MODELS = {
        "Winner": {
            'app_label': 'winners',
            'model_name': 'Winner',
            'resource': resource,  # Optional
        }
    }

    The available parameters are app_label, model_name, and resource. ‘resource’ should be a function which returns a django-import-export Resource.

  3. Done

By default a dry run of the import is initiated when the import object is created. To instead import the file immediately without a dry-run set the IMPORT_DRY_RUN_FIRST_TIME to False

IMPORT_DRY_RUN_FIRST_TIME = False

Performing an import

You will find an example django application that uses django-import-export-celery for importing data. There are instructions for running the example application in the example directory’s README file. Once you have it running, you can perform an import with the following steps.

  1. Navigate to the example applications admin page:

    screenshots/admin.png
  2. Navigate to the ImportJobs table:

    screenshots/import_jobs.png
  3. Create a new import job. There is an example import CSV file in the example/example-data directory. Select that file. Select csv as the file format. We’ll be importing to the Winner’s model table.

    screenshots/new_import_job.png
  4. Select “Save and continue editing” to save the import job and refresh until you see that a “Summary of changes made by this import” file has been created.

    screenshots/summary.png
  5. You can view the summary if you want. Your import has NOT BEEN PERFORMED YET!

    screenshots/view-summary.png
  6. Return to the import-jobs table, select the import job we just created, and select the “Perform import” action from the actions drop down.

    screenshots/perform-import.png
  7. In a short time, your imported Winner object should show up in your Winners table.

    screenshots/new-winner.png

Setting up exports

As with imports, a fully configured example project can be found in the example directory.

  1. Add a export_resource_classes classmethod to the model you want to export.
    @classmethod
    def export_resource_classes(cls):
        return {
            'winners': ('Winners resource', WinnersResource),
            'winners_all_caps': ('Winners with all caps column resource', WinnersWithAllCapsResource),
        }

    This should return a dictionary of tuples. The keys should be unique unchanging strings, the tuples should consist of a resource and a human friendly description of that resource.

  2. Add the create_export_job_action to the model’s ModelAdmin.
    from django.contrib import admin
    from import_export_celery.admin_actions import create_export_job_action
    
    from . import models
    
    
    @admin.register(models.Winner)
    class WinnerAdmin(admin.ModelAdmin):
        list_display = (
            'name',
        )
    
        actions = (
            create_export_job_action,
        )
  3. To customise export queryset you need to add get_export_queryset to the ModelResource.
    class WinnersResource(ModelResource):
        class Meta:
            model = Winner
    
        def get_export_queryset(self):
            """To customise the queryset of the model resource with annotation override"""
            return self.Meta.model.objects.annotate(device_type=Subquery(FCMDevice.objects.filter(
                    user=OuterRef("pk")).values("type")[:1])
  4. Done!

Performing exports with celery

  1. Perform the basic setup procedure described in the first section.

  2. Open up the object list for your model in django admin, select the objects you wish to export, and select the Export with celery admin action.

  3. Select the file format and resource you want to use to export the data.

  4. Save the model

  5. You will receive an email when the export is done, click on the link in the email

  6. Click on the link near the bottom of the page titled Exported file.

Excluding export file formats in the admin site

All available file formats to export are taken from the Tablib project.

To exclude or disable file formats from the admin site, configure IMPORT_EXPORT_CELERY_EXCLUDED_FORMATS django settings variable. This variable is a list of format strings written in lower case.

IMPORT_EXPORT_CELERY_EXCLUDED_FORMATS = ["csv", "xls"]

Customizing File Storage Backend

If you are using the new Django 4.2 STORAGES:

By default, import_export_celery uses Django default storage. To use your own storage, use the the IMPORT_EXPORT_CELERY_STORAGE_ALIAS variable in your Django settings and adding the STORAGES definition. For instance:

STORAGES = {
    "import_export_celery": {
        "BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
    },
}
IMPORT_EXPORT_CELERY_STORAGE_ALIAS = 'import_export_celery'

DEPRECATED: If you are using old style storages:

Define a custom storage backend by adding the IMPORT_EXPORT_CELERY_STORAGE to your Django settings. For instance:

IMPORT_EXPORT_CELERY_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"

Customizing Task Time Limits

By default, there is no time limit on celery import/export tasks. This can be customized by setting the following variables in your Django settings file.

# set import time limits (in seconds)
IMPORT_EXPORT_CELERY_IMPORT_SOFT_TIME_LIMIT = 300  # 5 minutes
IMPORT_EXPORT_CELERY_IMPORT_HARD_TIME_LIMIT = 360  # 6 minutes

# set export time limits (in seconds)
IMPORT_EXPORT_CELERY_EXPORT_SOFT_TIME_LIMIT = 300  # 5 minutes
IMPORT_EXPORT_CELERY_EXPORT_HARD_TIME_LIMIT = 360  # 6 minutes

Customizing email template for export job completion email

By default this is the subject and template used to send the email

Subject: 'Django: Export job completed'
Email template: 'email/export_job_completion.html'

The default email template can be found here

The default email subject and template can be customized by overriding these values from django settings:-

EXPORT_JOB_COMPLETION_MAIL_SUBJECT="Your custom subject"
EXPORT_JOB_COMPLETION_MAIL_TEMPLATE="path_to_folder/your_custom_template.html"

The email template will get some context variables that you can use to customize your template.

{
    export_job: The current instance of ExportJob model
    app_label: export_job.app_label
    model: export_job.model
    link: A link to go to the export_job instance on django admin
}

For developers of this library

You can enter a preconfigured dev environment by first running make and then launching ./develop.sh to get into a docker compose environment packed with redis, celery, postgres and everything you need to run and test django-import-export-celery.

Before submitting a PR please run flake8 and (in the examples directory) python3 manange.py test.

Please note, that you need to restart celery for changes to propogate to the workers. Do this with docker-compose down celery, docker-compose up celery.

Commercial support

Commercial support is provided by gradesta s.r.o.

Credits

django-import-export-celery was developed by the Czech non-profit auto*mat z.s..

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django_import_export_celery-1.7.1.tar.gz (18.9 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file django_import_export_celery-1.7.1.tar.gz.

File metadata

File hashes

Hashes for django_import_export_celery-1.7.1.tar.gz
Algorithm Hash digest
SHA256 a4cc086ae24384e4640bdcd262d5933a162775d272546194840d98c2af6b3500
MD5 8539b5c5e205c748baff798937c24465
BLAKE2b-256 48986b723cd93c75a569912a98d22ba258c7a4aae1e0244925a29a45f5516917

See more details on using hashes here.

File details

Details for the file django_import_export_celery-1.7.1-py3-none-any.whl.

File metadata

File hashes

Hashes for django_import_export_celery-1.7.1-py3-none-any.whl
Algorithm Hash digest
SHA256 4897ff63a2fc5ca85b4f2fb4e01a3937af9ad3881f2b9106411def72dad77232
MD5 a299b1a3672fc1704f31fd007df44e68
BLAKE2b-256 9c05067b6de07fbfac38b25c3eada96dc2079ffc9332db631c2166183a20d99c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page