Whitebox Plugin - Transfer Manager

Manages file transfers from external devices (cameras, etc.) with a two-phase workflow: download from device, then process (transcode/stitch/convert). Tracks persistent state, emits real-time progress to the frontend, and handles retries and crash recovery automatically.

Installation

poetry add whitebox-plugin-transfer-manager

Capabilities

Exposed to Other Plugins

Plugin Classes (get_plugin_classes_map)

Accessible via import_whitebox_plugin_class():

Models (model_registry)

Frontend Components (exposed_component_map)

Frontend Slots (slot_component_map)

State Stores (state_store_map)

Daemon

Command: daemon_transfer_manager

Runs as a BaseDaemonCommand that orchestrates all transfers.

Subscribed Events

Periodic Checks (every 5s)

• Stale transfer recovery: Resets transfers stuck in active states with no DB update for 2+ minutes (worker likely dead)
• Dispatch pending: Picks up PENDING transfers respecting concurrency limits

Startup Recovery

• Resets any in-progress transfers (DOWNLOAD_STARTED, DOWNLOAD_COMPLETED, PROCESSING_STARTED) back to PENDING
• Clears retry_after on PENDING transfers that were in backoff when daemon stopped

Emitted Events

All events are sent to the management WebSocket group.

Registering an Adapter

Device plugins expose adapters via their plugin class:

# In device plugin's whitebox plugin class
def get_transfer_adapters(self):
    from .transfer_adapter import MyDeviceTransferAdapter
    return [("my_device_codename", MyDeviceTransferAdapter())]

The daemon discovers these on startup; workers discover lazily on first use.

Process file recording command

Command: process_file_recording [device_codename] [source_path] [destination_path]

This command is used to process a file recording for a transfer manually.

Example:

docker exec backend-dev poetry run python whitebox/manage.py process_file_recording \
  insta360_x4 \
  /opt/whitebox/media/transfers/downloads/VID_001.insv \
  /opt/whitebox/media/transfers/processed/VID_001.m3u8

Running this command will process the specified source path with the transfer adapter defined by the device plugin for the given device_codename, and store it into the destination path, just like the transfer manager would do it.

It can be used for debugging or testing purposes.

TransferManager API

Queueing

from plugin.registry import import_whitebox_plugin_class

transfer_manager = import_whitebox_plugin_class("transfer.TransferManager")
RemoteFile = import_whitebox_plugin_class("transfer.RemoteFile")

# Queue a batch of files (sync, for use in RQ tasks or daemon)
files = [RemoteFile(path="/DCIM/VID_001.insv", size=1_000_000)]
transfers = transfer_manager.queue_batch_sync(
    device_connection=conn,
    files=files,
    download_dir=Path("/media/transfers/downloads"),
    processed_dir=Path("/media/transfers/processed"),
    associate_with=flight_session,
)

Concurrency

Retry Logic

• Transient errors (network, device) retry with exponential backoff: 2s, 4s, 8s, ..., capped at 60s
• Retry window: 7 days from initial queue time
• Safety cap: 500 retries max
• Non-transient errors (disk full, corrupt file) do not retry

Additional Instructions

• Plugin Development Guide
• Plugin Testing Guide
• Contributing Guidelines