Wetlands

Wetlands is a Python library for creating isolated environments with Pixi and running Python functions inside them.

This lets an application use libraries with incompatible dependencies at the same time. For example, Cellpose and StarDist can each run in their own environment while exchanging ordinary Python values and NumPy arrays with the main application.

Wetlands creates these environments when needed, installs their dependencies, keeps worker processes ready for repeated calls, and cleans up their resources automatically. It can be used in desktop applications, servers, and plugin systems.

Wetlands is intended for code you trust. Isolated environments prevent dependency conflicts, but they do not restrict what code can access on your computer.

Appose is an alternative for applications that need interprocess cooperation across Python, Java, or Groovy, including explicit zero-copy tensor sharing between languages. Wetlands is focused on running Python functions and adds automatic NumPy transport, managed worker pools, and post-hoc debugger attachment. See Wetlands and Appose for a short comparison, or visit the Appose documentation.

Wetlands 2 provides:

• side-effect-light manager construction;
• observable and cancellable preparation and provisioning operations;
• reproducible Pixi projects and pixi.lock files;
• managed-environment discovery and safe asynchronous removal;
• warm worker pools;
• validated environment variables for individual worker indices;
• qualified installed-package targets and path targets for local development;
• automatic transport of ordinary Python values and NumPy arrays;
• blocking, callback-based, and asyncio-friendly execution.

The first preparation may download a verified Pixi executable, and the first provisioning of an environment downloads its declared packages. These operations require network access and can take several minutes. Wetlands stores Pixi, managed environments, locks, and runtime state below the manager root you choose.

Installation

pip install wetlands

Install the optional host-side NumPy dependency when arrays cross the execution boundary:

pip install "wetlands[shared-memory]"

Quick start

The manager constructor only validates and stores configuration. Downloading or inspecting Pixi begins when prepare() or provision() is called.

import numpy as np

from wetlands import EnvironmentManager, EnvironmentSpec

manager = EnvironmentManager(root="wetlands")

preparation = manager.prepare()
preparation.listen(lambda event: print(event.stage, event.message))
pixi = preparation.wait_for()

spec = EnvironmentSpec(
    python="3.12.*",
    conda=("numpy>=2",),
)
environment = manager.provision("numpy-example", spec).wait_for()

with environment.start(workers=1) as workers:
    image = np.arange(9, dtype=np.float32).reshape(3, 3)
    task = workers.submit_import(
        "numpy:negative",
        args=(image,),
    )
    result = task.wait_for()

np.testing.assert_array_equal(result, -image)
manager.close()

This example is self-contained: Pixi installs NumPy in the worker environment, and the qualified target imports NumPy inside that environment. Your own worker package exposes ordinary Python functions, is declared in EnvironmentSpec, and is called by its installed module:qualified.callable name in the same way.

Wetlands owns the shared-memory details. Inputs use copy-in semantics and returned arrays are independently owned by the caller. The repository also contains a complete local worker-package example.

Async applications

Preparation, provisioning, and execution objects are awaitable. Their events() methods expose async event streams while the caller retains ownership of its event loop.

import asyncio

from wetlands import EnvironmentManager, EnvironmentSpec


async def main() -> None:
    manager = EnvironmentManager(root="wetlands")
    try:
        preparation = manager.prepare()
        async for event in preparation.events():
            print(event.kind.value, event.message)
        await preparation

        environment = await manager.provision(
            "analysis",
            EnvironmentSpec(python="3.12.*", conda=("numpy",)),
        )

        workers = await asyncio.to_thread(environment.start, workers=2)
        try:
            task = workers.submit_import(
                "numpy:negative",
                args=([1.0, 2.0, 3.0],),
            )
            result = await task
            print(result)
        finally:
            await asyncio.to_thread(workers.close)
    finally:
        await asyncio.to_thread(manager.close)


asyncio.run(main())

Starting, attaching, detaching, and closing worker pools, and closing a manager, are blocking lifecycle calls. Async applications should run those calls with asyncio.to_thread() as shown above.

Cancel an operation or execution task with cancel(). A canceled provisioning operation becomes terminal only after its active process tree has stopped and its incomplete environment has been cleaned up. For a running worker task, Wetlands first requests cooperative cancellation. If the worker does not finish during the configured grace period, Wetlands terminates its process tree and starts a replacement worker.

Pixi projects and lockfiles

EnvironmentSpec is the complete managed recipe:

from pathlib import Path

from wetlands import EnvironmentSpec, LocalPackage, PostInstallCommand

spec = EnvironmentSpec(
    python="3.12.*",
    conda=("numpy>=2", "scikit-image", "pip"),
    pypi=("example-pypi-package==1.2.0",),
    channels=("conda-forge",),
    local=(LocalPackage(Path("../worker-package"), editable=True),),
    post_install=(PostInstallCommand(("python", "-m", "worker_package.prepare_assets")),),
    pixi_lock=Path("pixi.lock"),
)

When pixi_lock is supplied, Wetlands provisions from those exact locked dependencies. If the recipe contains local packages, the supplied lockfile must already resolve those same local sources and editable settings. It must also include Wetlands' exact managed worker-runtime dependencies, including its debugpy pin. Without one, Pixi resolves the generated project and Wetlands preserves the resulting lockfile in the managed environment.

Wetlands owns the managed debugpy version, so applications must not declare it in EnvironmentSpec. The managed runtime pin participates in recipe identity and causes environments to rebuild when it changes.

An environment is ready only after every installation and validation step succeeds and Wetlands atomically publishes its ready metadata. Failed, canceled, or crash-interrupted provisioning is rebuilt on the next attempt rather than resumed. The returned ManagedEnvironment exposes its canonical project and lockfile paths, Pixi executable and version, recipe hash, lockfile hash, and generation ID.

Worker targets

Installed packages use a qualified target:

task = workers.submit_import(
    "package.module:ClassName.method",
    args=(value,),
)

The module is imported inside the isolated worker.

Local development can use an explicit source path:

task = workers.submit_path(
    "worker_code.py",
    "segment",
    kwargs={"image": image},
    cache=False,
)

Path targets are keyed by canonical path and content, so equal filename stems do not collide.

Debug running workers

Wetlands can start a debugger after an application and its workers are already running. No debug flag or debugger call is required in application or worker code.

wetlands workers --root ./wetlands --environment numpy-example
wetlands debug --root ./wetlands --environment numpy-example --worker WORKER_ID --editor vscode --source .

The debug adapter remains available for reconnection until its worker exits. See Debugging running workers and Persistent workers and reconnection.

Supported values

Execution arguments and results may contain:

• None, booleans, integers, floats, strings, and bytes;
• nested lists, tuples, and dictionaries with simple keys;
• NumPy arrays without object dtype.

Unsupported objects fail explicitly at the boundary. Non-contiguous arrays are transported as contiguous arrays. Intermediate task outputs are limited to simple values in Wetlands 2.

Migration from Wetlands 1

Wetlands 2 is a major release with a deliberately smaller public API. Applications should migrate explicitly instead of relying on compatibility shims.

See the Wetlands 2 migration guide.

Development

Install the development environment with:

uv sync --frozen --group dev --extra shared-memory

Run the fast test suite with:

uv run --extra shared-memory pytest -m "not integration and not compat and not manual"

Run the representative real-Pixi integration suite with:

UV_PROJECT_ENVIRONMENT=.venv-py314 uv run --python 3.14 --extra shared-memory pytest tests/test_v2_pixi_integration.py

Run linting with:

uv run ruff check
uv run ruff format --check
uv run mypy src/wetlands

Build the package with:

uv build

Documentation

The complete documentation is available at arthursw.github.io/wetlands. Contributor-facing architecture and codec boundaries are described in the developer guide.

License

Wetlands is licensed under the MIT License.

See the security policy before executing third-party worker code or post-install commands.