whitebox-plugin-transfer-manager 0.1.6

A plugin for whitebox to manage file transfers from devices

Whitebox Plugin - Transfer Manager

Manages file transfers from external devices (cameras, etc.) with a two-phase workflow: download from device, then process (transcode/stitch/convert). Tracks persistent state, emits real-time progress to the frontend, and handles retries and crash recovery automatically.

Installation

poetry add whitebox-plugin-transfer-manager

Capabilities

Provides	Requires
transfer-manager	device

Exposed to Other Plugins

Plugin Classes (get_plugin_classes_map)

Accessible via import_whitebox_plugin_class():

Key	Type	Description
transfer.TransferManager	TransferManager instance	Singleton service for queueing, dispatching, and tracking transfers
transfer.TransferAdapter	Protocol class	Interface that device plugins implement for download/processing
transfer.RemoteFile	dataclass class	Represents a file on a remote device

Models (model_registry)

Key	Model	Description
transfer.Transfer	Transfer	Persistent record for each file transfer

Frontend Components (exposed_component_map)

Category	Key	Component	Description
service-component	transfer-service	TransferServiceComponent	Background service that listens for WebSocket transfer events, updates the Zustand store, and triggers flight session re-fetches on completion
transfer-manager	transfer-panel	TransferPanel	UI panel showing active/completed transfers with progress bars, cancel, and retry actions

Frontend Slots (slot_component_map)

Slot	Component
transfer.panel	TransferPanel

State Stores (state_store_map)

Key	Module	Description
transfer	stores/transfer	Zustand store tracking transfer state (transfers, activeCount, upsertTransfer, removeTransfer, clearCompleted, dismissAll)

Daemon

Command: daemon_transfer_manager

Runs as a BaseDaemonCommand that orchestrates all transfers.

Subscribed Events

Event	Action
observation.flight.end	Dispatch any pending transfers
observation.device.connection_status.update	On reconnect, dispatch pending transfers for that device
command.transfer.cancel	Cancel a transfer (sets Redis flag for in-progress workers)
command.transfer.retry	Retry a failed/cancelled transfer
command.transfer.queue_batch	Create transfer records from a batch of files and dispatch

Periodic Checks (every 5s)

• Stale transfer recovery: Resets transfers stuck in active states with no DB update for 2+ minutes (worker likely dead)
• Dispatch pending: Picks up PENDING transfers respecting concurrency limits

Startup Recovery

• Resets any in-progress transfers (DOWNLOAD_STARTED, DOWNLOAD_COMPLETED, PROCESSING_STARTED) back to PENDING
• Clears retry_after on PENDING transfers that were in backoff when daemon stopped

Emitted Events

All events are sent to the management WebSocket group.

Event	When
observation.transfer.queued	Transfer created or reset to pending
observation.transfer.download.started	Download begins
observation.transfer.download.progress	Download progress (throttled: 1s or 5% delta)
observation.transfer.download.completed	Download finished
observation.transfer.processing.started	Processing begins
observation.transfer.processing.progress	Processing progress (throttled: 1s or 5% delta)
observation.transfer.processing.completed	Processing finished, FlightSessionRecording created
observation.transfer.download.failed	Download failed permanently
observation.transfer.processing.failed	Processing failed permanently
observation.transfer.retry_scheduled	Transfer scheduled for retry with backoff
observation.transfer.cancelled	User cancelled transfer
observation.transfer.paused	User paused transfer

Registering an Adapter

Device plugins expose adapters via their plugin class:

# In device plugin's whitebox plugin class
def get_transfer_adapters(self):
    from .transfer_adapter import MyDeviceTransferAdapter
    return [("my_device_codename", MyDeviceTransferAdapter())]

The daemon discovers these on startup; workers discover lazily on first use.

Process file recording command

Command: process_file_recording [device_codename] [source_path] [destination_path]

This command is used to process a file recording for a transfer manually.

Example:

docker exec backend-dev poetry run python whitebox/manage.py process_file_recording \
  insta360_x4 \
  /opt/whitebox/media/transfers/downloads/VID_001.insv \
  /opt/whitebox/media/transfers/processed/VID_001.m3u8

Running this command will process the specified source path with the transfer adapter defined by the device plugin for the given device_codename, and store it into the destination path, just like the transfer manager would do it.

It can be used for debugging or testing purposes.

TransferManager API

Queueing

from plugin.registry import import_whitebox_plugin_class

transfer_manager = import_whitebox_plugin_class("transfer.TransferManager")
RemoteFile = import_whitebox_plugin_class("transfer.RemoteFile")

# Queue a batch of files (sync, for use in RQ tasks or daemon)
files = [RemoteFile(path="/DCIM/VID_001.insv", size=1_000_000)]
transfers = transfer_manager.queue_batch_sync(
    device_connection=conn,
    files=files,
    download_dir=Path("/media/transfers/downloads"),
    processed_dir=Path("/media/transfers/processed"),
    associate_with=flight_session,
)

Concurrency

Setting	Default	Description
Max concurrent global	4	Total active transfers across all devices
Max concurrent per device	2	Active transfers per device connection

Retry Logic

• Transient errors (network, device) retry with exponential backoff: 2s, 4s, 8s, ..., capped at 60s
• Retry window: 7 days from initial queue time
• Safety cap: 500 retries max
• Non-transient errors (disk full, corrupt file) do not retry

Additional Instructions

• Plugin Development Guide
• Plugin Testing Guide
• Contributing Guidelines