django-fsspec

A Django app that provides a file system interface via fsspec, backed by Django ORM.

Features

• fsspec compatible — use standard fsspec.filesystem("django") API
• Multi-database — works through Django ORM on supported relational databases
• Configurable block size — tune storage granularity per deployment
• Optimistic locking — safe concurrent writes with conflict detection
• Append-safe API — append mode uses the same database-backed append operation as the public API
• Namespace partitioning — separate path spaces via integer namespace; authorization remains the host app's responsibility
• Path validation — blacklist rules + Unicode NFC normalization
• Implicit directories — no directory records, derived from file paths
• Management commands — fsspec_gc, fsspec_fsck, fsspec_stats

Quick Start

pip install django-fsspec

Add to INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    "django_fsspec",
]

Run migrations:

python manage.py migrate

Use it:

import fsspec

fs = fsspec.filesystem("django", namespace_id=1)

# Write
with fs.open("/hello.txt", "wb") as f:
    f.write(b"Hello World")

# Read
data = fs.cat("/hello.txt")  # b"Hello World"

# List
fs.ls("/")  # ["/hello.txt"]

# Delete
fs.rm("/hello.txt")

Configuration

Add to your Django settings.py:

# Block size in bytes (default: 256KB)
DJANGO_FSSPEC_BLOCK_SIZE = 64 * 1024

# Maximum file size in bytes (default: 2MB)
DJANGO_FSSPEC_MAX_FILE_SIZE = 2 * 1024 * 1024

Supported File Modes

Performance

Benchmarked on GitHub Actions (ubuntu-latest), default 256KB block size. Push/PR CI runs the bounded benchmark scale (--scale ci --seed 1) and uploads JSON artifacts that include database, backend, scale, and seed metadata.

Full benchmark results (including concurrency tests) are collected by CI on every push and available as GitHub Actions artifacts.

For larger seeded datasets, run the manual GitHub Actions workflow “Large Benchmark”. It accepts database (sqlite, mysql, postgres, oracle), scale (medium, large), seed, and optional scenario inputs. These runs include seeded ls, exists, info, and find scenarios and upload JSON artifacts named with the database, scale, and seed.

Documentation

• Getting Started
• Configuration
• Usage Guide
• Architecture
• Management Commands
• Benchmarks
• Block Size Migration
• Exceptions

中文文档 | Chinese Documentation

License

MIT — see LICENSE.