Django-Filer ("Finder" branch)

The "Finder" branch of django-filer is a complete rewrite of the original django-filer project.

A rewrite was necessary because the original codebase was not maintainable anymore. Please read this discussion for details. Apart from the maintainability issues, using the File and Folder models was not possible in a multi-tenant environment. Extending the file model with a more specialized implementation was possible in theory, but so complicated that apart from the ImageModel no other implementations ever have been created.

Less third-party dependencies

The "Finder" branch of django-filer has less third-party dependencies. It does not depend on django-polymorphic, django-mptt and easy-thumbnails anymore.

For large datasets django-cte is reccomended, in order to improve the speed of tree travesals, which is important while searching.

Since each FileModel contains a JSONField to store arbitrary data, django-entangled is reccomended, in order to give users the opportunity to edit the contents of that field.

The client part of the new admin user interface has no runtime dependencies. It is compiled into two JavaScript files, which are included by the corresponding admin views. One of them is used for the django-admin interface, the other one for the frontend. The frontend part is also used for the new file selection widget.

Extensibility

The "Finder" branch of django-filer is designed to be extensible for every MIME-type one desires. Therefore, those extensions might need their own dependencies in order to extract or visualize a certain type of file. A minimal installation can be configured to only distinguish between files and folders.

Multiple Ambits to hold Folder Roots

The "Finder" branch of django-filer allows multiple ambits, each able to hold multiple root folders. Such an ambit must be created using the management command manage.py finder add-ambit …. Each ambit has a slug and a name. The slug is used in the URL to access that ambit, while the name is used for display purposes. All ambits are displayed in the left sidebar of the Django admin. For each ambit, a different Django storage can be configured.

New Model Features

The "Finder" branch introduces a new model named InodeModel. This is the base class for all other models, such as the FileModel, the FolderModel and all other specialized implementations. By using a UUID instead of a numerical primary key, it is possible to use different database tables while preserving the possibility to perform queries across all tables. This allows us to register models inheriting from the AbstractFileModel without the need of django-polymorphic, and thus a join between two or more tables for each query.

Multiple ambits are allowed, each containg one root folder, one trash folder per user and a list of allowed tags. Each ambit must explictly be created using a management command. All ambits are displayed in the left sidebar of the Django admin. For each ambit, a different Django storage can be configured. It also is possible to restrict an ambit to a Site and/or to an AdminSite.

The Admin interface has also been completely rewritten. For instance, there is no more list view for the FolderModel and the FileModel (or any specialized implementation). Instead, there are only details views for each of those models. This is much nearer to what a user would expect from a file system. Therefore, if a user wants to access the list view of a folder, he is redirected immediately to the detail view of the root folder of the given ambit. From there, he can traverse the folder tree in the same manner he's used to from his operating system.

New User Interface

The user interface has also been completely rewritten. It offers drag and drop functionality for moving files and folders. It now is possible to select multiple files and folders at once by a rectangular selection with the mouse. Double-clicking on a folder opens it, by loading the detail view for that FolderModel. Selected files or folders can be dragged into a target folder. They can also be cutted or copied to a clipboard and pasted into another folder.

Trash Folder

Each user now has its own private trash folder. Whenever he deletes a file or folder, it is moved to that trash folder. From there, he can restore the file or folder, or delete it permanently. The trash folder can be emptied automatically after a given time period.

Permission System

The permission system of django-filer is using the concept of Access Control Lists (ACLs) similar to Posix or NTFS ACLs. This allows granting fine-grained permissions to everybody, individual users and/or groups for each file and folder.

Permissions are controlled through the model named AccessControlEntry. This model has a foreign key onto InodeModel and a so-called "principal". A principal is either a user, a group or for everyone. They are mutually exclusive and implemented as a nullable foreign key onto User, or a foreign key onto Group. Either of them can be set, but not both. If neither is set, the permission is considered for everyone.

By using a separate model AccessControlEntry, django-filer can now compute the permissions using just one database (sub-)query while filtering all inodes. Until version 3, the permissions had to be computed traversing all ancestors starting from the current folder up to the root of the folder tree. This was a time-consuming operation and made django-filer slow for large datasets.

Each AccessControlEntry has a bitmap field used to represent:

• write set for a folder: Allows the user to upload a file, to change the name of that folder, to reorder files in that folder, to move files out of that filer.
• write set for a file: Allows the user to replace a file, to edit the file's meta-data, and to move the file to another folder, if the source and target folders have write permissions.
• read set for a folder: Allows the user to open that folder and view its content.
• read set for a file: Allows the user to view a thumbnail of that file inside its containing folder and use that file, including copying.
• admin: Allows the user to change the permissions of that folder or file.
• A generic foreign key pointing onto the InodeModel. This creates a one-to-many relation between different file types and folders on one side and the access control list on the other.

In addition to AccessControlEntry the permission system also provides a model named DefaultAccessControlEntry. This acts as a template for newely files or folders created inside, or move to another folder. The DefaultAccessControlEntry is implemented as a separate model with a foreign key onto the FolderModel, because it is not possible to use the AccessControlEntry for both purposes, since the default permissions must be inherited by all children of a folder, while the regular permissions only apply to the file or folder they are attached to.

A foreign key onto the folder model to set a permission template. Read below for details.

If a folder has write but no read permission, the user can upload files into that folder, but doesn't see files from other users. This is named the "Dropbox" functionality.

Each file and folder has a foreign key named owner, pointing onto the User model. The owner of a file or folder can change its permissions if he has the global permission to do so. When creating a new file or folder, the currently loggedin user is set as the owner of that file or folder.

Only a superuser and the owner of a file or folder can change its permissions. The superuser can change the permissions of any file or folder. Staff users can change the permissions only of files they own themselves.

Only a superuser can change the owner of a file or folder.

In addition to the file and folder permissions, each folder requires a template of permissions on how to inherit them to files and folders created as children of that specific folder. This can be achieved with a separate foreign key in model AccessControlEntry pointing onto the FolderModel.

Microsoft gives a good explanation on the implementation of ACLs in their Data-Lake implementation.

Alternative Views

The user interface offers four alternative views for the folder tree.

Tiles is a "flat" view, which shows all files and folders as a bunch of tiles.

Mosaic is very similar to the "Tiles" view, but shows a much smaller version of those file, folder and image icons.

List is a view with detailed information about every file or folder. Here every item requires a separate row.

Column is a view which shows columns for the currently selected folder and each of its ancestors. This allows to easily move files between those folders.

Multiple Favrourite Folders

Each user can have multiple favourite folders. This allows him to quickly access those folders from the navigation bar. It is also possible to drag a file from the current view into one of the tabs for of the favorite folders.

Implementation Details

The new user interface is based on the React framework with the DnD-Kit as its main additional dependency.

Installation

The new version of django-filer is not yet available on PyPI. Therefore, you have to install it from GitHub:

git clone https://github.com/django-cms/django-filer.git
cd django-filer
git switch finder
pip install --no-deps -e .

The new version of django-filer requires Django-5.2 or later.

The new version of django-filer currently has only been tested with SQLite and Postgres, but should also work on MariaDB and MySQL.

In settings.py of your project, add these extra dependencies or those you really need:

    INSTALLED_APPS = [
        ...
        'finder',
        'finder.contrib.archive',  # supports zip, tar, tar.gz, tar.bz2, tar.xz
        'finder.contrib.audio',  # supports mp3, ogg, flac, wav
        'finder.contrib.common',  # supports PDF, spreadsheets
        'finder.contrib.image.pil',  # supports bmp, gif, jpeg, png, tiff
        'finder.contrib.image.svg',  # supports svg
        'finder.contrib.video',  # supports mp4, webm, ogv
        ...
    ]

If you use:

• finder.contrib.audio or finder.contrib.video, assure that ffmpeg-python is installed.
• finder.contrib.image.pil, assure that Pillow is installed.
• finder.contrib.image.svg needs no extra dependency; it reads an SVG's size from the document and thumbnails it by rewriting the root element. Documents that state their size only in relative units (width="100%" without a viewBox) cannot be measured that way and are shown with a generic icon.
• Postgres as a database, install psycopg2 or psycopg2-binary if available for your platform.

Each root folder requires two storage backends. One for the public files and one for their thumbnails and/or samples. Add these storage backends to the STORAGES setting in your settings.py:

STORAGES = {
    'default': {
        'BACKEND': 'django.core.files.storage.FileSystemStorage',
    },
    'staticfiles': {
        'BACKEND': 'django.contrib.staticfiles.storage.StaticFilesStorage',
    },
    'finder_public': {
        'BACKEND': 'finder.storage.FinderSystemStorage',
        'OPTIONS': {
            'location': '/path/to/your/media/filer_public',
            'base_url': '/media/filer_public/',
            'allow_overwrite': True,
        },
    },
    'finder_public_samples': {
        'BACKEND': 'finder.storage.FinderSystemStorage',
        'OPTIONS': {
            'location': '/path/to/your/media/filer_public_thumbnails',
            'base_url': '/media/filer_public_thumbnails/',
            'allow_overwrite': True,
        },
    },
}

If instead of using the local file system you want to use another storage backend, such as MinIO, Amazon S3 or Google Cloud Storage, configure the corresponding storage backend in the STORAGES setting as:

STORAGES = {
    ...
    'finder_public': {
        'BACKEND': 'storages.backends.s3.S3Storage',
        'OPTIONS': {...},
    },
    'finder_public_samples': {
        'BACKEND': 'storages.backends.s3.S3Storage',
        'OPTIONS': {...},
    },
}

Note that multiple root folders can share the same storage location or bucket.

If you declare neither finder_public nor finder_public_samples, finder derives them from your default storage on startup: same filesystem root and URL prefix, but a subdirectory per alias, the FinderSystemStorage backend and allow_overwrite turned on. With Django's defaults that gives you

MEDIA_ROOT/finder_public/          served from  MEDIA_URL + finder_public/
MEDIA_ROOT/finder_public_samples/  served from  MEDIA_URL + finder_public_samples/

which is enough to get started. Declare them yourself once you want the files somewhere else — on S3, on another volume, or shared between projects.

Run the migrations for app finder:

python manage.py migrate finder

This creates a first ambit for you, so that the admin and the form widgets work right away. It is named by FINDER_DEFAULT_AMBIT, which is "public" — the slug the FinderFileFields and FinderFolderFields of your models refer to when they declare no ambit of their own. Override it if you want a different name:

FINDER_DEFAULT_AMBIT = 'assets'

Its root folder grants read/write to everybody, which is the same thing manage.py finder add-ambit does. Restrict it through the permission editor in the admin, or set up the ambits yourself instead:

FINDER_CREATE_DEFAULT_AMBIT = False

The setting is only read while finder.0001_initial is applied, so it has to be in place before you migrate for the first time.

Create further ambits with the configuration above:

python manage.py finder add-ambit private --values name="Private Folder" storage=finder_private sample_storage=finder_private_samples

You can create multiple root folders, each with their own unique slug and name. You may configure a storage backend for each root folder or share them between multiple root folders.

If you already have django-filer installed and that database is filled, you can migrate the meta-data of those files and folders into the new database tables. The physical files on disk are not affected by this migration. Remember to leave the filer app in INSTALLED_APPS.

python manage.py filer_to_finder

This does not affect the original database tables. You can still use the original django-filer codebase in parallel to the new "Finder" branch. They both share the same underlying file system, usually media/filer_public but store their meta information in completely independent database tables. The only caveat is that you should not erase files in the trash folder, because then they are also gone on disk.

The client part of the new admin user must be compiled before it can be used. This requires a modern version (18 or later) of Node.js and npm, which usually are installed anyway. The following commands installs the requirements and compiles the code:

cd django-filer
npm install --include=dev
npm run buildall

Usage

After restarting the development server, you can access the new admin user interface at the URL http://localhost:8000/admin/finder/foldermodel/ . Clicking on that link will redirect you to the root folder of the current site.

Admin Backend

The admin backend is implemented as a web interface mimicking the features of the Finder app of your operating system's graphical user interface. It is possible to …

• Download, upload files and to view them as thumbnailed images.
• Navigate through the folder tree by double-clicking on a folder.
• Move files and folders by drag and drop.
• Select multiple files and folders by a rectangular selection with the mouse.
• Cut or copy files and folders to a clipboard and paste them into another folder.
• Delete files and folders, which are moved to a trash folder.
• Restore files and folders from the trash folder or to delete them permanently.
• Create new folders and to upload files.
• Rename files and folders by slowly double-clicking on their name.
• Create multiple tabs for favourite folders quick access. They also can be used to drag files from the current view into one of the tabs.
• Search for files and folders.
• Tag files and filter them by their tags.

Interface to other Django projects

django-filer (Finder branch) ships with a new file selection field. This field can be used in any Django model. It allows to select a file from the files stored inside django-filer. It also allows to upload a local file and use it immediatly.

Example:

from django.db import models
from finder.models.fields import FinderFileField

class MyModel(models.Model):
    my_file = FinderFileField(
        null=True,
        blank=True,
        ambit='my-ambit',  # this must be the slug of an existing ambit
    )

Forms generated from this model will have a file selection widget for the my_file field. When rendered as HTML, this widget becomes the webcomponent <finder-file-select …></finder-file-select> with a few additional attributes. The JavaScript part of the widget must be included using the script tag <script src="{% static 'finder/js/finder-select.js' %}"></script>.

The demonstration app provides an example how this field can be used in a form. Just vist the URL http://localhost:8000/demoapp/ and click on the blank area to either select an existing file or upload a new one.

Thumbnailing

In addition to the x- and y-coordinates of the focal point, the ImageModel now also stores the main area of interest as a square. This information then is used to create thumbnails using Art Direction.

The method finder.contrib.image.pil.models.ImageModel.crop() creates a thumbnail of the given image taking the focal point and the resolution into account. The following rules apply when cropping and thumbnailing an image:

• The thumbnailed image always contains at least the main area of interrest, except for this situation: If the thumbnail is narrower than the main area of interest, the thumbnail will be cropped to the center of that area.
• The resolution of the area to be thumbnailed is always at least the same as the resolution of the original image.

will take the resolution of the corresponsding image into consideration. This will allow to create different versions of the same canonical image, depending on the width of the device the image is displayed.

By extending the focal point to three degrees of freedom, it is possible to create a thumbnail taking the resolution into account. This allows us to create thumbnails taking art direction into consideration. For instance, if a user uploads a portrait image, the focal point can be set to the head of the person in that image.

Validating Uploads

Every uploaded payload is validated before it is written to storage. A validator is a callable

def my_validator(file_name, file, owner, mime_type):
    ...

that raises finder.exceptions.FileValidationError to reject the upload. It is handed the uploaded file rewound and opened for writing, so it may also rewrite the content in place rather than reject it — that is how the bundled SVG sanitizer works. Rejecting a payload leaves nothing behind: validation runs before the file reaches the storage backend.

This is deliberately the same contract django-filer uses for FILER_ADD_FILE_VALIDATORS, so validators written for filer — including third-party ones — work here unchanged. A validator signalling rejection with a plain ValueError is accepted too.

By default finder rejects the formats a browser executes in the media origin, and sanitizes SVG uploads (which needs py-svg-hush, part of the svg extra):

MIME-type	Validator
text/html	finder.validators.deny_html
application/xhtml+xml, application/xml, text/xml, application/xslt+xml	finder.validators.deny
image/svg+xml	finder.contrib.image.svg.validators.sanitize_svg

Unlike django-filer, finder does not deny application/octet-stream: FileModel is the documented fallback for everything that matches no other model, and browsers report that MIME-type for plenty of harmless files. Add {'application/octet-stream': ['finder.validators.deny']} for filer's behaviour.

Add your own validators with FINDER_PAYLOAD_VALIDATORS, a dict mapping a MIME-type to a list of validators. The MIME-type may be exact, a subtype wildcard (image/*) or */*; more general matches run first. Values are dotted paths, callables or classes.

FINDER_PAYLOAD_VALIDATORS = {
    'image/*': ['myapp.validators.strip_exif'],
}

Drop a default with FINDER_REMOVE_PAYLOAD_VALIDATORS, a list of MIME-types:

FINDER_REMOVE_PAYLOAD_VALIDATORS = ['image/svg+xml']

Further Steps

• A quota system, which allows to limit the amount of disk space a user can use.

Further Readings

https://developer.mozilla.org/en-US/docs/Web/HTML/Guides/Responsive_images https://www.smashingmagazine.com/2014/05/responsive-images-done-right-guide-picture-srcset/ https://www.smashingmagazine.com/2016/01/responsive-image-breakpoints-generation/ https://www.smashingmagazine.com/2016/09/automating-art-direction-with-the-responsive-image-breakpoints-generator/ https://cloudfour.com/thinks/responsive-images-101-part-9-image-breakpoints/ https://cloudfour.com/thinks/sensible-jumps-in-responsive-image-file-sizes/

https://cloudfour.com/thinks/the-real-conflict-behind-picture-and-srcset/

License

This part of the code will be licensed under the terms of the MIT License.