A Django/Django-Storages threaded S3 chunk uploader
Project description
A Django file handler to manage piping uploaded files directly to S3 without passing through the server’s file system. The uploader uses multiple threads to speed up the upload of larger files. This package relies on Django and Django-Storages, allowing the use of the S3 storages FileField but changes the upload behaviour to bypass local file system.
Quick start
Install the package:
pip install s3chunkuploader
Set the Django FILE_UPLOAD_HANDLERS setting:
FILE_UPLOAD_HANDLERS = (‘s3chunkuploader.file_handler.S3FileUploadHandler’,)
How it works
The File Handler intercepts the file upload multipart request at the door, and as chunks of the file are received from the browser, they are collectd into an internal queue within custom ThreadPoolWorker. When the queue surpasses a configurable size (by default 5MB which is the minimum Part size for S3 multipart upload), it is submitted to the Thread Pool as a Future which will then resolve. Once all the chunks are uploaded and all the futures are resolved the upload is complete. By default 10 threads are used which means a 100MB file upload can be potentially sent as 20 5MB parts to S3.
The FileHandler ultimately returns a ‘dummy’ django-storages S3Boto3StorageFile which is compatible with the storages S3 File Field, but was not actually used to upload a full file. The file is also enhanced with two additional attributes:
original_name: The original file name uploaded file_size: The actual full file size uploaded
It is recommended to bypass csrf checks on the upload file view as the csrf check will read the POST params before the handler is used. A replacement file field S3FileField is provided in fields.py and is satisfied with the S3 object key
Settings
The following settings are expected in your Django application (only 2 are required)
Setting |
Description |
---|---|
AWS_ACCESS_KEY_ID |
Required. Your AWS access key |
AWS_SECRET_ACCESS_KEY |
Required. The AWS secret |
AWS_REGION |
Optional. Region of S3 bucket |
S3_DOCUMENT_ROOT_DIRECTORY |
Optional. Document root for all uploads (prefix) |
S3_APPEND_DATETIME_ON_UPLOAD |
Optional [True]. Appent the current datetime sring to the uploaded file name |
S3_PREFIX_QUERY_PARAM_NAME |
Optional [__prefix]. A query param key name which provides additional prefix for the object key on S3 |
S3_MIN_PART_SIZE |
Optional [5MB]. The part size in bytes to upload to S3 |
MAX_UPLOAD_SIZE |
Optional [None]. The maximum file size in bytes for an individual file. |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for s3chunkuploader-0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7e74f0a72f154815f8f807bf1b28b5cdcb49a407eb4535617292d97fb4007aa |
|
MD5 | 5ef7ef89a9a8ad7ff4fbd0e86b3b1ac4 |
|
BLAKE2b-256 | 1e4ecdd53ee29a01c2e7ca8b79f082c1de5ed753fde53b4fe7fdaf210a285397 |