Skip to main content

sontris-raytest: Ray-backed remote execution and pytest support

sontris-raytest is a Python library for executing scripts and commands on Ray workers. It also ships reusable pytest fixtures, test helpers, and a sample remote project. The user and API documentation is in docs/; build the HTML site with Sphinx using the instructions in docs/contributing.rst.

  1. The local project installs sontris-raytest and its Ray Client dependencies.
  2. Pytest connects to the configured Ray head and creates a worker-side workspace from the selected project directory.
  3. By default, the worker runs uv sync for that project, creating a separate remote virtual environment.
  4. Tests execute commands and scripts using the selected remote environment and download generated artifacts.
  5. A Rust integration test provisions a toolchain if needed, compiles a standard-library-only HTTP calculator on the worker, and runs an API testharness there.

The local pytest environment and the remote sample project's environment are intentionally separate. The tests verify that remote setup succeeds and that the sample project's python-dotenv dependency is importable from its virtual environment. The Rust example is separate from that Python environment: its source is uploaded with the fixture, then built and tested on the worker.

Requirements and setup

  • Python 3.10+
  • uv on workers when using the default remote environment setup
  • A Ray cluster reachable over Ray Client

The local Ray Client Python version must match the Python version used to start the cluster. This checkout pins uv to Python 3.13.5 in .python-version, matching the example remote head. If connecting to a head started with another Python version, update the pin with uv python pin <version> and recreate the local environment with uv sync --locked. requires-python = ">=3.10" remains the package compatibility range; the pin selects the interpreter for this development environment.

Install the development environment:

uv sync --locked

External test suites can install the test support with:

pip install "sontris-raytest[testing]"

In their conftest.py, load the packaged pytest fixtures:

pytest_plugins = ["sontris.testing.raytest.pytest_plugin"]

The plugin uploads the nearest project root from the current directory by default. Select a different project directory with pytest --sontris-raytest-working-dir=/path/to/project, or override its sontris_raytest_working_dir fixture. The sample project exposed as sontris.testing.raytest.REMOTE_PROJECT is useful for examples and sontris-raytest's own tests; it is not forced on consuming projects.

Then import the common helpers in tests with from sontris.testing.raytest import execute_script, run_command.

Versioning and wheel traceability

The package version is dynamic: Hatchling obtains it from Git through the build dependency uv-dynamic-versioning==0.14.1, as configured in pyproject.toml. The tool uses the latest matching PEP 440 Git tag; unprefixed tags such as 1.2.3 and v-prefixed tags such as v1.2.3 are accepted. A build at a release tag gets that tag's version. Commits after a tag use a bumped PEP 440 development version, including Git revision information.

The configuration also defines fallback-version = "0.0.0". This is a baseline for builds where no matching tag is available; with the configured version bumping and available commit metadata, the resulting wheel can be a development version rather than literally 0.0.0. Therefore, do not infer the version from the latest release tag alone: inspect the wheel produced by the build. Build from a Git checkout with the relevant tags and history available. CI checks out full history (fetch-depth: 0). After checks pass, a push of a non-development release tag builds, verifies, and publishes the wheel and source distribution to PyPI. The workflow rejects development and local versions and verifies that the wheel version matches the tag. Branch pushes do not publish to PyPI. After a release tag passes checks and is published to PyPI, the workflow triggers the Read the Docs latest build and creates a Forgejo release with non-merge commits since the previous tag, a version-specific PyPI link, and a link to the English latest documentation. A shallow clone may not have enough history to calculate the development version correctly; fetch the full history as well as tags before building.

After committing a clean working tree, uv run run.py tag builds a temporary wheel using this versioning configuration and creates a lightweight local tag from its release version (the development and local suffixes are removed). Push the tag to trigger the release workflow; the command does not push it.

For an auditable build, retain the following alongside the wheel:

  • repository URL, full git rev-parse HEAD commit, tag (if any), and whether git status --porcelain was empty;
  • wheel filename and the Version field in its .dist-info/METADATA;
  • SHA-256 checksum of the wheel (and source distribution, if retained);
  • hashes of pyproject.toml and uv.lock, plus the uv and Python versions used to build it.

Build the artifacts with the repository task:

uv run run.py build
git rev-parse HEAD
git describe --tags --long --always --dirty
git status --porcelain
git tag --points-at HEAD
uv --version
python --version
sha256sum pyproject.toml uv.lock dist/sontris_raytest-*

The wheel filename and its embedded Version field are the authoritative package version. A development version may include a short Git revision, but record the full commit separately; the version string alone is not a complete source or build record. Record the build output and checksums with the artifacts, and verify that the working tree is clean for release builds.

The local cluster defaults to 127.0.0.1 and Ray Client port 10001. To override them for a single command, set process environment variables:

RAY_HOST=127.0.0.1 RAY_CLIENT_PORT=10001 ./scripts/cluster/start-head.sh

For client processes, RAY_HOST is the reachable Ray Client host and RAY_CLIENT_PORT is the port the client connects to. In scripts/cluster/start-head.sh, RAY_HOST instead selects the node IP Ray advertises. For exactly 127.0.0.1 or localhost, the script lets Ray choose the local interface; for other values it passes the configured IP as --node-ip-address. The script accepts an explicit IP argument or --ssh-address to infer the server-side IP from SSH_CONNECTION. Its separate RAY_GCS_PORT (default 6379) configures Ray's GCS port, not the Ray Client port. The dashboard is bound to 127.0.0.1:8265, independently of the advertised node address.

The scripts and executor do not load .env files. The executor reads RAY_HOST and RAY_CLIENT_PORT from the process environment; direct RemoteExecutor construction can instead use its host and port arguments. .env files remain excluded from uploaded archives. Never put credentials in the sample project or files uploaded to the cluster. The executor does not take ownership of an already initialized Ray session by default. For the pytest fixtures, use --sontris-raytest-use-existing-ray-session or override the sontris_raytest_manage_ray_lifecycle fixture to False. For direct use, pass manage_ray_lifecycle=False. The caller must ensure the existing session is connected to the intended cluster and is responsible for shutting it down.

Run tests

All project tests run through the nested task group. Run Ray-free unit tests with:

uv run run.py test unit

The unit task also reports line and branch coverage for the library, task runner, and docstring checker, including source lines not exercised by tests.

Start a local Ray head and run the localhost integration suite with:

uv run run.py deploy local
uv run run.py test local
uv run run.py deploy stop-local

For configured remote Ray clusters, pass the reachable host for each named cluster. Use the same --ray-host NAME=HOST option for remote tests and remote deployment; deployment hosts must also be reachable over SSH. The runner adds configured hosts to both NO_PROXY and no_proxy for subprocesses while preserving existing entries:

uv run run.py deploy remote \
  --ray-host plt01=192.0.2.10 \
  --ray-host plt02=192.0.2.11
uv run run.py test remote \
  --ray-host plt01=192.0.2.10 \
  --ray-host plt02=192.0.2.11
uv run run.py test remote-docker \
  --ray-host plt01=192.0.2.10 \
  --ray-host plt02=192.0.2.11

test remote-docker requires local Docker access, Ray Client connectivity to both targets, and Docker installed and usable by the selected Ray workers. It does not install Docker remotely. The example addresses above are documentation placeholders; provide the addresses reachable in your network. Use uv run run.py test collect to collect tests without connecting to Ray.

Run every suite with managed local and remote Ray nodes using:

uv run run.py test all \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example \
  --ssh-password-prompt

This command reports endpoint health, restarts the local head and both remote heads, runs unit, local, remote, and Docker tests, then attempts to stop every node it started even if a test fails. It requires SSH access to both hosts and Docker usable locally and by remote workers. Starting a head stops existing Ray processes, so run this only when restarting and stopping these nodes is intended.

Deploy Ray heads

Pass remote SSH targets as repeated --ray-host NAME=HOST options. Each host must be reachable both as a Ray Client endpoint and over SSH; SSH aliases from ~/.ssh/config are supported. The remote start script discovers the host-side IP used by the SSH connection and passes it to Ray as the advertised node address. Configure SSH access, then deploy the cluster scripts and restart Ray on each host using:

uv run run.py deploy remote \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example

The deployment installs uv for the SSH user when needed and uploads scripts/cluster/, including its Ray-only pyproject.toml and lock file, as well as the repository's .python-version pin. The lifecycle scripts use that pin for Ray head and client compatibility. uv run run.py deploy local starts the local head at 127.0.0.1; deploy stop-local, deploy start-remote, and deploy stop-remote provide the corresponding lifecycle operations.

The start and stop commands use the same named-host options:

uv run run.py deploy start-remote \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example
uv run run.py deploy stop-remote \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example
uv run run.py deploy stop-local

SSH uses the current username and configured SSH keys by default. Set RAYTEST_SSH_USER or RAYTEST_SSH_KEY to override those settings, or add --ssh-password-prompt to a remote deploy/start/stop command to enter an SSH password interactively. Credentials are not stored in the inventory.

Each host starts its own Ray head; this does not join the hosts into one Ray cluster. --dashboard-host controls the dashboard bind address only; it is separate from the advertised node address used for cluster and Ray Client connectivity.

Ray may warn that token authentication is disabled. This is separate from the uv environment warning. A Ray cluster is trusted-code infrastructure: anyone who can reach its ports may be able to run code on it. Local-only development is different from exposing a head on a LAN or the internet; for remote use, configure Ray authentication and encrypted network access before opening the Ray ports. See the Ray token authentication guidance.

For a remote Ray head, the Ray Client port (default 10001) must be allowed through the network firewall. The pytest options accept hostnames or IP addresses; they do not pin a workspace actor to a particular worker.

The configured host selects the Ray cluster, not a specific worker. Ray schedules the workspace actor on a worker. Worker placement requires Ray resource requirements or placement strategies, which are outside this example. The pytest remote_executor_group_factory maps friendly names to hostnames or IP addresses; all executors in one group use the same RAY_CLIENT_PORT.

Project layout

src/sontris/testing/raytest/                        Public executor, pytest API, and plugin
src/sontris/testing/raytest/support/                Reusable execution and Ray host helpers
src/sontris/testing/raytest/fixtures/remote_project/ Packaged remote workload
tests/unittest/                                     Ray-free pytest unit tests
tests/local/                                        Local Ray integration tests
tests/remote/                                       Configured remote Ray integration tests
tests/docker/                                       Docker distribution tests
tests/conftest.py                                   Shared pytest fixtures and host options
scripts/cluster/                                    Local Ray head start/stop helpers
inventory.py                                         Root pyinfra inventory
scripts/pyinfra/                                     Remote pyinfra deployment tasks
pyproject.toml                                      Local package and development dependencies
uv.lock                                             Locked local development environment

The packaged fixture project has its own pyproject.toml and uv.lock; these are uploaded and used by remote uv sync in sontris-raytest's own tests. The fixture also includes a standard-library-only Rust calculator crate. Its pytest test uses an existing worker toolchain or installs minimal stable Rust with rustup, builds the crate remotely, and runs the HTTP API harness. If the worker lacks a C linker, the setup script uses apt-get to install build-essential and ca-certificates. This requires a Debian/Ubuntu worker, root or passwordless sudo, and access to package repositories. Installing Rust for the first time also requires outbound network access from the worker to the official rustup and Rust distribution servers.

Use the executor

The reusable API is available as sontris.testing.raytest.RemoteExecutor:

from pathlib import Path

from sontris.testing.raytest import RemoteExecutor

with RemoteExecutor(
    host="127.0.0.1",
    port=10001,
    working_dir=Path.cwd(),
) as executor:
    environment = executor.prepare_environment()
    result = executor.execute_command(
        [environment["python"], "scripts/example.py", "--name", "Ray"]
    )

    if result["timed_out"]:
        raise TimeoutError("Remote command timed out")
    if result["returncode"] != 0:
        raise RuntimeError(result["stderr"])
    print(result["stdout"])

When omitted, host and port default to the RAY_HOST and RAY_CLIENT_PORT process environment variables, or to 127.0.0.1 and 10001. If working_dir is omitted, the executor searches upward from the current working directory for the nearest pyproject.toml.

The default prepare_environment() uses uv sync. To use another dependency workflow, pass an argument sequence as setup_command and optionally select the interpreter it creates:

environment = executor.prepare_environment(
    setup_command=["python3", "-m", "venv", ".venv"],
    python_executable=".venv/bin/python",
)

To use an already-provisioned interpreter without running setup, pass only python_executable. Paths relative to the workspace are supported. Setup commands are argument lists and are never run through a shell.

The executor can prepare the remote environment, run a Python file from the uploaded workspace, execute a system command, download a file or directory, and inspect workspace/environment information. Its actor-backed workspace persists for the lifetime of the executor and is removed on close. Use it as a context manager or call close() explicitly. Each executor owns one workspace actor; Ray chooses the worker. sontris-raytest provides remote execution and pytest integration helpers, not pytest test collection or scheduling across the cluster.

License

sontris-raytest is licensed under the GNU Lesser General Public License v3.0 or later (LGPL-3.0-or-later); the accompanying COPYING file contains the GNU GPLv3 text referenced by the LGPL. The LGPL permits use by proprietary applications, subject to its conditions for distributing the library or combined works. It does not impose a field-of-use restriction or require publication of private modifications. It does not grant an exemption from export-control, sanctions, or other applicable laws.

Third-party dependencies are distributed under their own licenses and are not relicensed by sontris-raytest. Review each dependency's terms for your use case.

Development

See CONTRIBUTING.md for the architecture, fixture conventions, and guidance for adapting the executor to another test suite.

Metadata

Release files for sontris-raytest 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sontris-raytest 0.0.4
File Size Uploaded Provenance Actions
sontris_raytest-0.0.4.tar.gz 43.0 kB Without provenance Details for sontris_raytest-0.0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sontris-raytest 0.0.4
File Interpreter ABI Platform Actions
sontris_raytest-0.0.4-py3-none-any.whl Python 3 none any Details for sontris_raytest-0.0.4-py3-none-any.whl

Total release size: 94.8 kB

Release files / sontris_raytest-0.0.4.tar.gz

Download URL sontris_raytest-0.0.4.tar.gz
Size 43.0 kB
Tags Source
SHA-256 checksum
How to use checksums
1ef8c8052696857debe2b2e570c5fa4089c75113939cd85d100d54cd336983ac
BLAKE2b-256 checksum
How to use checksums
7d96c8401cc6a889dcec0ec198e132d0f1800321fa0d677764143214391b20a6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / sontris_raytest-0.0.4-py3-none-any.whl

Download URL sontris_raytest-0.0.4-py3-none-any.whl
Size 51.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1b3aa74940662eb70be37092c2fe3272064cc7941632e714fc54c959d7310882
BLAKE2b-256 checksum
How to use checksums
dac84e255d114cb0bc14104264c9495332c6edf93a6b81e181984659e30310ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for sontris-raytest 0.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page