muleline 0.1.0

Universal file sync engine. Chunked resumable uploads, SHA-256 dedup, storage backend connectors.

Muleline

Self-hosted file sync and photo management. One container, SQLite, no nonsense.

---

What is Muleline?

Muleline is a self-hosted file sync and photo management engine built for developers who want to own their infrastructure without the operational overhead of Immich, Nextcloud, or PhotoPrism.

Immich requires 4 containers, PostgreSQL, Redis, and a machine learning service before you can upload a single photo. Muleline requires Python and a directory. It ships as a pip install, mounts as a FastAPI router in your existing app, or runs standalone. All state lives in SQLite. No Redis, no Postgres, no Docker Compose sprawl.

It also goes where Immich cannot: store files on S3, Cloudflare R2, Google Drive, Dropbox, SFTP, or WebDAV — or implement your own backend in five methods.

---

---

Key Features

• Photo timeline with EXIF metadata — dates, GPS coordinates, dimensions, and MIME type extracted automatically on upload; query and sort by any field
• Chunked resumable uploads — TUS-inspired protocol, 5MB chunks; disconnect mid-upload and resume exactly where you left off
• SHA-256 deduplication — hash-checked before upload initiates; duplicate files are detected in one round-trip without transferring bytes
• Automatic thumbnails — 400px JPEG thumbnails generated and cached on first access; served independently from the source file
• Multi-device sync with manifest diff — client sends a list of file hashes, server returns exactly what to upload and what to download; no redundant transfers
• Device management — register phones, tablets, desktops; track last sync time and platform per device
• Sync event log — every upload, download, and error recorded for auditing and debugging
• 6 storage backends — local filesystem, S3/R2/MinIO, WebDAV, SFTP, Dropbox, Google Drive (details below)
• Pluggable custom backend — 5-method abstract interface; point Muleline at any storage system in under 50 lines
• Mountable FastAPI router — 3 lines to add Muleline sync endpoints to an existing app; no separate service, no new port
• Works without FastAPI — the engine is a plain Python class; use it directly in scripts, tests, or other frameworks
• Single SQLite database, zero external dependencies — WAL mode, no connection pooling setup, no migration tools required
• MIT licensed — use it, fork it, ship it in commercial products

---

Muleline vs. the Alternatives

	Muleline	Immich	Nextcloud	PhotoPrism
Containers required	1 (or 0)	4+	1	1
Database	SQLite	PostgreSQL	MySQL/Postgres	SQLite/MySQL
Message queue	none	Redis	none	none
Install	pip install	Docker Compose	Docker Compose	Docker Compose
Mountable in your app	yes	no	no	no
Storage backends	6 + custom	local only	local + plugins	local only
Language	Python/FastAPI	TypeScript/NestJS	PHP	Go
License	MIT	AGPL-3.0	AGPL-3.0	AGPL-3.0

---

Quick Start

pip install muleline[server]

# server.py
from fastapi import FastAPI
from muleline.engine import SyncEngine
from muleline.storage import LocalStorage
from muleline.db import run_migrations
import muleline.router as sync_router

run_migrations()
engine = SyncEngine(storage=LocalStorage("./data/uploads"))
sync_router.init(engine)

app = FastAPI()
app.include_router(sync_router.router)

uvicorn server:app --port 8000

All sync endpoints are now live at http://localhost:8000/sync/. Visit /docs for the auto-generated API reference.

---

Storage Backends

Backend	Use case	Install
LocalStorage	Local filesystem, NAS mount, any path	included
S3Storage	AWS S3, Cloudflare R2, MinIO, Backblaze B2	pip install muleline[s3]
WebDAVStorage	Synology NAS, Nextcloud, any WebDAV server	pip install muleline[webdav]
SFTPStorage	Any SSH server	pip install muleline[sftp]
DropboxStorage	Dropbox Business, personal	pip install muleline[dropbox]
GoogleDriveStorage	Google Drive, Shared Drives	pip install muleline[gdrive]
StorageBackend (base)	Anything else — implement 5 methods	included

Install everything at once:

pip install muleline[all]

S3 / R2 / MinIO

from muleline.storage import S3Storage

storage = S3Storage(
    bucket="my-bucket",
    endpoint_url="https://abc.r2.cloudflarestorage.com",  # omit for AWS
    access_key="...",
    secret_key="...",
)

Custom Backend

from muleline.storage.base import StorageBackend

class MyNASStorage(StorageBackend):
    def save(self, key: str, content: bytes, mime_type: str = "") -> None: ...
    def read(self, key: str) -> bytes | None: ...
    def delete(self, key: str) -> None: ...
    def exists(self, key: str) -> bool: ...
    def url(self, key: str, expires: int = 3600) -> str: ...

---

API Reference

All endpoints are mounted under /sync/. Authentication is handled by your middleware — Muleline reads request.state.user (dict with id or sub) or request.state.owner_id.

Upload

Method	Path	Description
POST	/sync/upload/init	Start a chunked upload session. Returns upload_id, chunk_size, total_chunks. Dedup check runs here — if the hash already exists, no upload is needed.
POST	/sync/upload/{id}/chunk	Upload one chunk. Body is multipart/form-data with chunk_index and chunk file.
GET	/sync/upload/{id}/status	Check progress. Returns received_chunks, total_chunks, list of received indexes.
POST	/sync/upload/{id}/complete	Assemble chunks, verify SHA-256, extract EXIF, persist to storage.
DELETE	/sync/upload/{id}	Abort and clean up chunk temp files.

Files

Method	Path	Description
GET	/sync/files	List files. Query params: sort (newest/oldest/name), mime (filter prefix), limit, offset.
GET	/sync/files/{id}	Get file metadata including EXIF, GPS, dimensions.
GET	/sync/files/{id}/serve	Serve the raw file with correct MIME type.
GET	/sync/files/{id}/thumb	Serve 400px JPEG thumbnail (generated and cached on first access).
DELETE	/sync/files/{id}	Delete file from database and storage backend.

Devices & Sync

Method	Path	Description
POST	/sync/devices/register	Register or update a device (upsert on device_id).
GET	/sync/devices	List all devices for the current user.
POST	/sync/manifest	Send client file hashes, get back to_upload and to_download arrays.
POST	/sync/log	Record a sync event (upload, download, error) with byte count.
GET	/sync/status	Sync overview: devices, recent activity, total files and bytes.
POST	/sync/cleanup	Remove expired pending uploads and their temp files.
GET	/sync/server/info	Server version, uptime, total files, storage backend type.

---

Configuration

Muleline reads from environment variables at startup.

Variable	Default	Description
MULELINE_DB_PATH	./data/muleline.db	SQLite database file path
MULELINE_CHUNK_DIR	./data/chunks	Temporary directory for upload chunks
MULELINE_THUMB_DIR	./data/thumbs	Thumbnail cache directory
MULELINE_CHUNK_SIZE	5242880 (5MB)	Chunk size in bytes
MULELINE_MAX_UPLOAD_SIZE	524288000 (500MB)	Maximum file size per upload
MULELINE_PENDING_TTL	86400 (24h)	Seconds before incomplete uploads expire
MULELINE_MAX_PENDING	10	Max concurrent uploads per user

---

Architecture

Sync flow, from client to storage:

1. Client hashes all local files (SHA-256).
2. Client calls POST /sync/manifest with the hash list.
3. Server diffs against its own database and returns two lists: what to upload, what to download.
4. For each file to upload, client calls POST /sync/upload/init. If the hash already exists for that user, the server returns immediately — zero bytes transferred.
5. Client splits the file into 5MB chunks and POSTs each to /sync/upload/{id}/chunk. Order does not matter; any chunk can be retried independently.
6. Client calls POST /sync/upload/{id}/complete. Server assembles chunks in order, verifies the full-file SHA-256, extracts EXIF metadata, and writes to the configured storage backend.
7. Thumbnails are generated lazily on first /thumb request and cached to disk.

State management:

All state lives in one SQLite file (muleline.db) with five tables: files, pending_uploads, devices, sync_log, and migrations. No ORM, no migration framework — raw SQL with WAL mode enabled.

Embedding in your app:

# Your existing FastAPI app
app = FastAPI()

# Add Muleline in 3 lines
run_migrations()
sync_router.init(SyncEngine(storage=LocalStorage("./uploads")))
app.include_router(sync_router.router, prefix="/media")

Muleline does not own the app. It is a module in yours.

---

Roadmap

• Mobile app — iOS and Android clients with background sync
• Desktop sync client — watched folder, automatic upload on change
• Albums and sharing — named collections, public share links with optional expiry
• Web UI — dark-themed browser interface for browsing and managing files
• Face recognition — cluster photos by person using local ML (no cloud)
• Map view — browse photos by GPS location
• End-to-end encryption — client-side encryption before upload; server stores ciphertext only
• Multi-user support — tenant isolation, admin panel
• PyPI release — pip install muleline from the public registry

---

Contributing

Muleline is MIT licensed and open to contributions. The codebase is small by design — 16 files, roughly 1,100 lines. If you can read Python you can understand the whole thing in an afternoon.

To run locally:

git clone https://gitlab.com/ArchonAGI/muleline
cd muleline
pip install -e ".[dev,server]"
python example.py          # runs on port 8510
pytest                     # run the test suite

Open an issue before a large PR. Small focused changes merge fastest.

---

License

MIT. See LICENSE.

Copyright (c) 2026 ArchonAGI