seamless-remote

seamless-remote is the client connectivity layer of the Seamless ecosystem. It provides async HTTP clients for every remote service that a Seamless workflow can talk to — buffer storage, transformation cache, jobserver, and Dask scheduler — together with launch-aware wrappers that start those services on demand via remote-http-launcher. When seamless-config calls init() or set_stage(), it is seamless-remote that opens the actual connections and keeps them alive.

This is an internal infrastructure package. User workflow code never imports seamless-remote directly — it is activated behind the scenes by seamless-config and consumed by seamless-core, seamless-transformer, and seamless-dask. The only user-facing entry points are the two CLI scripts described below.

Core concepts

Client hierarchy

Every remote service is accessed through a pair of classes:

Base clients (BufferClient, DatabaseClient, JobserverClient) perform async HTTP against a known host and port. They inherit from a shared Client base that manages per-thread aiohttp sessions, a retry decorator for transient failures, and a background keepalive thread that healthchecks open connections.

Launched wrappers extend the base clients with auto-launch: they call seamless_config.tools.configure_*() to build a launch dict, pass it to remote_http_launcher.run(), and cache the resulting server address. If the server is already running, the cache returns the existing connection. DaskserverLaunchedHandle follows the same pattern but is fully synchronous — it constructs a distributed.Client and wraps it in a SeamlessDaskClient.

Activation modules

Each service type has an activation module that manages the active set of clients and exposes the async functions consumed by the rest of the Seamless stack:

Each module maintains separate lists of read and write clients (or a single launched handle for the daskserver). activate() is called by seamless-config during stage transitions; it instantiates the appropriate clients from the cluster definition and makes them available to downstream consumers.

Client types: launched vs extern

Clients can be registered in two ways:

• Launched — seamless-remote starts the service itself (via remote-http-launcher). Configuration comes from the cluster definition in seamless.yaml / seamless.profile.yaml.
• Extern — the service is already running and a URL (or local directory, for buffer folders) is provided directly. Useful for shared infrastructure or debugging.

Both are registered through define_launched_client() and define_extern_client() on the activation module and are selected during activate().

Relation to the Seamless ecosystem

    Seamless runtime                   (Buffer/Checksum, direct/delayed, stages)
        │
        │  resolve/write buffers, check/store cached results,
        │  activate backends and delegate jobs to jobserver or daskserver
        ▼
    seamless-remote                    ◄── this package
        │
        │  async HTTP (aiohttp)
        ▼
    ┌──────────────┐  ┌──────────────────┐  ┌─────────────────┐  ┌────────────────────────┐
    │  hashserver   │  │ seamless-database │  │ seamless-jobserver│  │ seamless-dask-wrapper   │
    │  (buffers)    │  │  (result cache)   │  │  (job dispatch)  │  │  (Dask scheduler)       │
    └──────────────┘  └──────────────────┘  └─────────────────┘  └────────────────────────┘
                                                                          │
                                                          Dask workers run seamless-transformer,
                                                          which calls seamless-remote again
                                                          for nested transformations

The Seamless runtime above this layer consists mainly of seamless-core, seamless-transformer, and seamless-config. seamless-core uses seamless-remote for buffer resolution and writes, seamless-transformer uses it for buffer access, cache lookup/store, and job delegation, and seamless-config activates the appropriate backends during stage changes. seamless-remote in turn talks to four remote services: hashserver for buffers, seamless-database for transformation results, seamless-jobserver for lightweight job dispatch, and seamless-dask-wrapper (part of seamless-dask) for Dask-based execution. Inside Dask workers, the same path repeats — seamless-transformer runs again and calls seamless-remote for buffer/cache operations on nested transformations.

seamless-config is the only package that calls activate() / deactivate() directly. All other packages interact with seamless-remote through the module-level async functions (get_buffer, run_transformation, etc.).

Delegation levels

seamless-remote enables the tiered delegation model defined by seamless-config stages:

Moving between levels is a configuration change (seamless.yaml / seamless.profile.yaml), not a code change.

CLI scripts

Installing seamless-remote provides two utilities for working with content-addressed data from the command line:

Both accept a --project and --stage flag to select the storage context, and read remote client configuration from environment variables or config files.

# Resolve a checksum to a file
seamless-resolve abc123...def --output result.bin

# Resolve with project/stage context
seamless-resolve abc123...def --project myproject --stage prod --output result.bin

# Fingertip (resolve with recomputation fallback)
seamless-fingertip abc123...def --output result.bin

Environment variables

Installation

pip install seamless-remote

Requires Python >= 3.10. Dependencies: seamless-core, seamless-config, aiohttp, aiofiles, frozendict.

Optional (activated at runtime when needed): remote-http-launcher (for launched clients), seamless-dask and distributed (for daskserver integration).