django-chunk-upload-handlers 0.0.17

Chunking Django file handlers for S3 and ClamAV service uploads

This library provides the following Django file upload handlers:

• allow_list provides functionality for preventing arbitrary file uploads.

• s3 provides chunk uploading to an AWS S3 bucket and is heavily based on https://pypi.org/project/s3chunkuploader/

• clam_av provides chunk uploading to a ClamAV rest endpoint provided by - https://github.com/uktrade/dit-clamav-rest/ although it could be used with other compatible projects.

If used together, the results of the anti virus check are written to the object uploaded to S3.

Installation

pip install django-chunk-upload-handlers

Usage

Using in a Django logging configuration:

INSTALLED_APPS = [
    ...
    "django_chunk_upload_handlers",
]

FILE_UPLOAD_HANDLERS = (
    "django_chunk_upload_handlers.allow_list.AllowListFileUploadHandler",
    "django_chunk_upload_handlers.clam_av.ClamAVFileUploadHandler",
    "django_chunk_upload_handlers.s3.S3FileUploadHandler",
)  # Order is important

Dependencies

This project is a Django app and depends on the Django package.

The s3 file handler depends on boto3 and django-storages

settings.DEFAULT_FILE_STORAGE must be set to "storages.backends.s3boto3.S3Boto3Storage" or a class that derives from it.

Settings

Allow list

To allow file uploads, decorate the target views using the @allow_uploads decorator.

The fields list should contain field names of fields that you want to allow file uploads from.

Examples:

For function-based views, use:

@allow_uploads(fields=['field', 'names', 'here'])

For class-based views, use:

@method_decorator(allow_uploads(fields=['field', 'names', 'here']), name='dispatch')

For views imported from external packages, add a dictionary named UPLOAD_ALLOW_DICT to your Django settings with the path to the endpoint as the key and the value as an array of field names.

UPLOAD_ALLOW_DICT = {
    "/people/.+?/edit/role/": ["upload"], # Regex can be used for the path
    "/forecast/upload-budgets": ["file"],
}

S3

AWS_ACCESS_KEY_ID CHUNK_UPLOADER_AWS_ACCESS_KEY_ID

Provide either for the AWS access key optional. CHUNK_UPLOADER_AWS_ACCESS_KEY_ID is preferred if both are set.

AWS_SECRET_ACCESS_KEY CHUNK_UPLOADER_AWS_SECRET_ACCESS_KEY

Provide either for the AWS access secret key optional. CHUNK_UPLOADER_AWS_SECRET_ACCESS_KEY is preferred if both are set.

AWS_STORAGE_BUCKET_NAME CHUNK_UPLOADER_AWS_STORAGE_BUCKET_NAME

The S3 bucket to use for uploads. CHUNK_UPLOADER_AWS_STORAGE_BUCKET_NAME is preferred if both are set.

AWS_REGION CHUNK_UPLOADER_AWS_REGION

The AWS region to use. CHUNK_UPLOADER_AWS_REGION is preferred if both are set.

S3_ROOT_DIRECTORY CHUNK_UPLOADER_S3_ROOT_DIRECTORY

The directory path to use as root for uploads. CHUNK_UPLOADER_S3_ROOT_DIRECTORY is preferred if both are set.

CHUNK_UPLOADER_RAISE_EXCEPTION_ON_VIRUS_FOUND Defines whether or not to throw an exception if a virus is found. Defaults to False.

ClamAV

CLAM_AV_USERNAME The ClamAV service username.

CLAM_AV_PASSWORD The ClamAV service password.

CLAM_AV_DOMAIN The domain to use for the ClamAV service. Note, this is domain only so test.com rather than https://test.com

CLAM_PATH The path to the ClamAV service (used with the domain defined in the setting above). Defaults to /v2/scan-chunked

CLAM_AV_IGNORE_EXTENSIONS A list of file extensions to not process with ClamAV. Defaults to an empty list.

CLAM_USE_HTTP Use http rather than https. Should not be used in production environments. Defaults to False.

Usage with file fields

The package provides a validator for use with form and model fields.

The CHUNK_UPLOADER_RAISE_EXCEPTION_ON_VIRUS_FOUND should not be set to True when using this validator.

from django import forms
from django_chunk_upload_handlers.clam_av import validate_virus_check_result


class ExampleForm(forms.Form):
    example_form_field = forms.FileField(
        validators=[validate_virus_check_result, ]
    )

from django.db import models

class ExampleModel(models.Model):
    example_model_field = models.FileField(
        max_length=10,
        validators=[validate_virus_check_result, ],
    )

The validation message will display ‘A virus was found’ if a virus is detected. This message is a translation string.

Tests

$ pip install -r requirements.txt
$ tox

Publishing to PyPI

1. Acquire API token from Passman - Request access from the SRE team.

2. Add token to your keyring: keyring set https://upload.pypi.org/legacy/ __token__

3. Ensure build and twine are installed: pip install --upgrade build twine

4. Update the version field in setup.py

5. Build the package: python -m build

6. Upload the package: twine upload --repository pypi dist/*