Run django-import-export processes using django-stomp
Project description
django-import-export-stomp: process slow django imports and exports using django-stomp
django-import-export-stomp helps you process long running imports and exports using django-stomp.
Basic installation
-
Set up django stomp <github.com/juntossomosmais/django-stomp>
__ to work with your project. -
Add
'import_export_stomp'
to yourINSTALLED_APPS
settings variable -
Add
'author.middlewares.AuthorDefaultBackendMiddleware'
to yourMIDDLEWARE_CLASSES
How to run Example app
docker-compose up example
Setting up imports with stomp
A fully configured example project can be found in the example directory of this repository.
-
Perform the basic setup procedure described above.
-
Configure the IMPORT_EXPORT_STOMP_MODELS variable.
::
def resource(): # Optional from myapp.models import WinnerResource return WinnerResource IMPORT_EXPORT_MODELS = { "Winner": { 'app_label': 'winners', 'model_name': 'Winner', 'resource': resource, # Optional } }
The available parameters are
app_label
,model_name
, andresource
. 'resource' should be a function which returns a django-import-exportResource <https://django-import-export.readthedocs.io/en/latest/api_resources.html>
__. -
Done
By default a dry run of the import is initiated when the import object is created. To instead import the file immediately without a dry-run set the IMPORT_DRY_RUN_FIRST_TIME
to False
::
IMPORT_DRY_RUN_FIRST_TIME = False
Preforming an import
You will find an example django application that uses django-import-export-stomp for importing data. There are instructions for running the example application in the example directory's README file. Once you have it running, you can perform an import with the following steps.
-
Navigate to the example applications admin page:
.. image:: screenshots/admin.png
-
Navigate to the ImportJobs table:
.. image:: screenshots/import_jobs.png
-
Create a new import job. There is an example import CSV file in the example/example-data directory. Select that file. Select csv as the file format. We'll be importing to the Winner's model table.
.. image:: screenshots/new_import_job.png
-
Select "Save and continue editing" to save the import job and refresh until you see that a "Summary of changes made by this import" file has been created.
.. image:: screenshots/summary.png
-
You can view the summary if you want. Your import has NOT BEEN PERFORMED YET!
.. image:: screenshots/view-summary.png
-
Return to the import-jobs table, select the import job we just created, and select the "Perform import" action from the actions drop down.
.. image:: screenshots/perform-import.png
-
In a short time, your imported Winner object should show up in your Winners table.
.. image:: screenshots/new-winner.png
Setting up exports
As with imports, a fully configured example project can be found in the example
directory.
-
Add a
export_resource_classes
classmethod to the model you want to export. ::@classmethod def export_resource_classes(cls): return { 'winners': ('Winners resource', WinnersResource), 'winners_all_caps': ('Winners with all caps column resource', WinnersWithAllCapsResource), }
This should return a dictionary of tuples. The keys should be unique unchanging strings, the tuples should consist of a
resource <https://django-import-export.readthedocs.io/en/latest/getting_started.html#creating-import-export-resource>
__ and a human friendly description of that resource. -
Add the
create_export_job_action
to the model'sModelAdmin
. ::from django.contrib import admin from import_export.admin_actions import create_export_job_action from . import models @admin.register(models.Winner) class WinnerAdmin(admin.ModelAdmin): list_display = ( 'name', ) actions = ( create_export_job_action, )
-
To customise export queryset you need to add
get_export_queryset
to theModelResource
. ::class WinnersResource(ModelResource): class Meta: model = Winner def get_export_queryset(self): """To customise the queryset of the model resource with annotation override""" return self.Meta.model.objects.annotate(device_type=Subquery(FCMDevice.objects.filter( user=OuterRef("pk")).values("type")[:1])
-
Done!
Performing exports with stomp
-
Perform the basic setup procedure described in the first section.
-
Open up the object list for your model in django admin, select the objects you wish to export, and select the
Export with stomp
admin action. -
Select the file format and resource you want to use to export the data.
-
Save the model
-
You will receive an email when the export is done, click on the link in the email
-
Click on the link near the bottom of the page titled
Exported file
.
Excluding export file formats in the admin site
All available file formats to export are taken from the Tablib project <https://pypi.org/project/tablib/>
__.
To exclude or disable file formats from the admin site, configure IMPORT_EXPORT_STOMP_EXCLUDED_FORMATS
django settings variable. This variable is a list of format strings written in lower case.
::
IMPORT_EXPORT_STOMP_EXCLUDED_FORMATS = ["csv", "xls"]
Customizing File Storage Backend
Define a custom storage backend by adding the IMPORT_EXPORT_STOMP_STORAGE
to your Django settings. For instance:
::
IMPORT_EXPORT_STOMP_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
For developers of this library
TO BE INCLUDED
Credits
django-import-export-stomp
was based on django-import-export-celery developed by the Czech non-profit auto*mat z.s. <https://auto-mat.cz>
_.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for django_import_export_stomp-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70528f236bab43c3834a85785043c7fdceeb809c5161e3d98ca534c24a8a6423 |
|
MD5 | 2c5c032855ebbe924a061260983f2ff8 |
|
BLAKE2b-256 | 47ffe0f0fb5e35eb627b0b33e80bffe8a75b28cef08ecee0d3cf4a56a2f61ba0 |
Hashes for django_import_export_stomp-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2df0845a4b240e8b4a839ee34b98f44d2ada0012a9b7cd7fe428e254804d440 |
|
MD5 | 1c1caabe7fbf69a4f756adde0bfbaee9 |
|
BLAKE2b-256 | d18bcd92acfb2a204fcc46e0d2736290d1f64d2f022f9727c77aacc4622ba202 |