django-fsspec

A Django app that provides a file system interface via fsspec, backed by Django ORM.

Features

• fsspec compatible — use standard fsspec.filesystem("django") API
• Multi-database — works through Django ORM on supported relational databases
• Configurable block size — tune storage granularity per deployment
• Optimistic locking — safe concurrent writes with conflict detection
• Append-safe API — append mode uses the same database-backed append operation as the public API
• Namespace partitioning — separate path spaces via integer namespace; authorization remains the host app's responsibility
• Path validation — blacklist rules + Unicode NFC normalization
• Implicit directories — no directory records, derived from file paths
• Management commands — fsspec_gc, fsspec_fsck, fsspec_repair, fsspec_rechunk, fsspec_stats

Quick Start

pip install django-fsspec

Add to INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    "django_fsspec",
]

Run migrations:

python manage.py migrate

Use it:

import fsspec

fs = fsspec.filesystem("django", namespace_id=1)

# Write
with fs.open("/hello.txt", "wb") as f:
    f.write(b"Hello World")

# Read
data = fs.cat("/hello.txt")  # b"Hello World"

# List
fs.ls("/")  # ["/hello.txt"]

# Delete
fs.rm("/hello.txt")

Configuration

Add to your Django settings.py:

# Block size in bytes (default: 32KB)
DJANGO_FSSPEC_BLOCK_SIZE = 32 * 1024

# Maximum file size in bytes (default: 2MB)
DJANGO_FSSPEC_MAX_FILE_SIZE = 2 * 1024 * 1024

Supported File Modes

Performance

Benchmarked on GitHub Actions (ubuntu-latest), using the historical 256KB block size. The table below uses CI run 28412676243 on commit 2236341 with --scale ci --seed 1. Format: average latency (throughput).

Full benchmark results, including concurrency and manually triggered seeded runs, are documented in Benchmarks and available as GitHub Actions artifacts.

Documentation

• Getting Started
• Configuration
• Usage Guide
• Architecture
• Management Commands
• Benchmarks
• Block Size Operations
• Exceptions

中文文档 | Chinese Documentation

License

MIT — see LICENSE.