Skip to main content

Deduplicate your uploaded file

If making sure your file uploads are never duplicated is more important than organising your files into neat folders, you might want to try this package.

Usage

You can use the storage backend on a global level by adding the following to your django settings:

DEFAULT_FILE_STORAGE = 'dedupebackend.storage.DedupedStorage'

If you want to use the other features offered by dedupebackend, you need to add dedupebackend to INSTALLED_APPS like this:

INSTALLED_APPS = [
    'dedupebackend',  # does not matter what spot
    ...
]

Admin integration

Adding dedupebackend to INSTALLED_APPS gives you an admin page where you can check your uploaded files. I allready let you know dedupebackend just throws verything in a large folder, but that does not mean you can not add structure to the storage. Just not on a filesystem level. You should add structure by adding relations to other models. It is easy enough to add categories or something:

class FileCategory(models.Model):
    files = models.ManyToManyField('dedupebackend.UniqueFile')
    name = models.TextField()

If you want to add a filter to the dedupebackend admin, try something like this:

from dedupebackend.admin import UniqueFileAdmin
from dedupebackend.models import UniqueFile

admin.site.unregister(UniqueFile)

class CategoryUniqueFileAdmin(UniqueFileAdmin):
    list_filter = UniqueFileAdmin.list_filter + ('filecategory__name',)

admin.site.register(UniqueFile, CategoryUniqueFileAdmin)

that might need some work, I never tested it :p

fields

There are some fields in dedupebackend you can use instead of the django FileField and ImageField. You get a picker added to that, you can use to select a file from the existing uploaded files.

Use something like this:

from dedupebackend.fields import *

class KoeHenkModel(model.Model):
    name = models.TextField()
    file = UniqueFileField("A normal file, nothing special")
    image = UniqueImageField("an image")

How does it work?

Well, for each uploaded file, dedupebackend creates a file on disk named after the hash of the file. Mostly the same as git does (I actually tried to use libgit2 for this, but git is bad with deletions). Next to that file, a table holds a record with some information about the file. The primary key of this table is the hash value of the file. So it is really impossible to add duplicates (but but, hash collisions).

The fields actually render a file form field on a foreign key model field. The storage backend returns the hash value as the file name. And it can return file objects when given such a hash value.

Metadata

Release files for django-dedupe-backend 0.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for django-dedupe-backend 0.0.2
File Size Uploaded Provenance
django-dedupe-backend-0.0.2.tar.gz 20.8 kB Without provenance Details

Release files / django-dedupe-backend-0.0.2.tar.gz

Download URL django-dedupe-backend-0.0.2.tar.gz
Size 20.8 kB
Tags Source
SHA-256 checksum
How to use checksums
9ace4a0836a5c3433656b9c2f43e30b232f9e9fc8e6a8fba0005dc04f4fce444
BLAKE2b-256 checksum
How to use checksums
983de63b310aed473b0ececebe826699755b4fa008b87cc0406bc3fe3041f133
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for django-dedupe-backend 0.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

Release history Release notifications | RSS feed

This release

0.0.2 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page