Skip to main content

Polars Nanook over Ray

Runs the Polars Nanook scheduler and workers as Ray actors.

Quickstart

Prerequisites for a completely local test deployment:

  • A Polars Nanook license.json file, or a valid service account
  • A Python virtual environment:
uv venv
source .venv/bin/activate
uv pip install polars-nanook-ray

It will also install ray, and polars-nanook-compute which runs the scheduler and workers, as dependencies. The latter pins the Polars version of the environment, and is only installed on Linux, the only platform it is released for; elsewhere, run the scheduler and workers in containers. Note that a few things need to be set up for the Polars Nanook cluster to function properly:

# need to match the config if manually changed
mkdir --parents /tmp/polars/{anonymous-results,license,observatory,shuffle-data,temporary-data}

This latter can be started using the following command:

ray start \
  --dashboard-host=0.0.0.0 \
  --disable-usage-stats \
  --head \
  --port="6379" \
  --ray-client-server-port="10001" \
  --resources='{"head":1}' # pinning

If you do not already have one, create a Polars Nanook service account through the Polars Nanook portal.

Spawn a local multinode cluster:

import polars as pl
import polars_nanook as pn
import ray

from polars_nanook_ray.cluster import PolarsRayCluster
from polars_nanook_ray.config import (
    PolarsObservatoryConfig,
    PolarsRayClusterConfig,
    PolarsSchedulerConfig,
    PolarsServiceAccountLicenseConfig,
    PolarsSubprocessConfig,
    PolarsWorkerConfig,
)

config = PolarsRayClusterConfig(
    runtime=PolarsSubprocessConfig(single_host_cluster=True),
    num_workers=4,
    license=PolarsServiceAccountLicenseConfig(
        workspace_id="<WORKSPACE_ID>",
        client_id="<SERVICE_ACCOUNT_ID>",
        client_secret="<SERVICE_ACCOUNT_SECRET>",
    ),
    scheduler=PolarsSchedulerConfig(...),
    worker=PolarsWorkerConfig(...),
)

ray.init(address="auto", namespace=config.cluster_id)
cluster = PolarsRayCluster(config)
cluster.start()

print(
    pl.LazyFrame({"a": [1, 2, 3], "b": [4, 4, 5]})
    .with_columns(pl.col("a").max().over("b").alias("c"))
    .remote(pn.ClusterContext(uri=f"http://{cluster.get_client_addr()}"))
    .execute()
    .head
)

cluster.stop()
ray.shutdown()

or:

import polars as pl
import ray

from polars_nanook_ray.config import (
    PolarsObservatoryConfig,
    PolarsRayClusterConfig,
    PolarsSchedulerConfig,
    PolarsServiceAccountLicenseConfig,
    PolarsSubprocessConfig,
    PolarsWorkerConfig,
)
from polars_nanook_ray.context import RayClusterContext

config = PolarsRayClusterConfig(
    runtime=PolarsSubprocessConfig(single_host_cluster=True),
    num_workers=4,
    license=PolarsServiceAccountLicenseConfig(
        workspace_id="<WORKSPACE_ID>",
        client_id="<SERVICE_ACCOUNT_ID>",
        client_secret="<SERVICE_ACCOUNT_SECRET>",
    ),
    scheduler=PolarsSchedulerConfig(...),
    worker=PolarsWorkerConfig(...),
)

ray.init(address="auto", namespace=config.cluster_id)

with RayClusterContext(config) as ctx:
    print(
        pl.LazyFrame({"a": [1, 2, 3], "b": [4, 4, 5]})
        .with_columns(pl.col("a").max().over("b").alias("c"))
        .remote(ctx)
        .execute()
        .head
    )

ray.shutdown()

In case Ray is running on a single host, set the single_host_cluster attribute of PolarsSubprocessConfig to True to offset worker ports and avoid socket collisions.

The actors are running in detached mode and survive past the script: one can reconnect with the same ray.init() and cluster gymnastics from another process. To clean all actors and underlying processes, Ray itself needs to be shutdown using the following command:

ray stop --force

Autoscaling

A dedicated scaler actor, pinned to the scheduler node, runs an HTTP server to handle scaling requests sent by the underlying binary. These requests are relayed to the scheduler actor, which in turn adds or removes Ray worker actors in response.

Enable it via PolarsScalingConfig on the scheduler, and optionally set min_workers and/or max_workers on the cluster config to bound how far it may scale. Note the cluster always starts num_workers workers: the bounds are reported back to the binary on GET /scale_config, seeded into max_workers_per_query, and only enforced on rescaling, where a POST /scale_to count outside them is clamped. Requesting more workers is done via the client: .distributed(min_workers=X).

The HTTP server is unauthenticated and binds 127.0.0.1, or the Docker network gateway when the binaries run in containers. It accepts only application/json bodies, and ignores worker names that are not worker actors.

License server

PolarsRayLicenseServer manages the Polars Nanook offline license server binary as a Ray actor. Ray clusters point their configuration at it via PolarsLicenseServerConfig(uri=...) and it validates them locally, tracking usage into signed reports it periodically emits (and optionally uploads to our control plane).

It is standalone: unlike the scheduler/worker/scaler, it is not wired into PolarsRayCluster. It is meant to be a single, long-lived service that any number of separate clusters register against, so its lifecycle (and Ray namespace) is managed independently, and it should be started before any cluster that points at it.

import ray

from polars_nanook_ray.config import (
    PolarsRayLicenseServerConfig,
    PolarsSubprocessConfig,
)
from polars_nanook_ray.license_server import PolarsRayLicenseServer

config = PolarsRayLicenseServerConfig(
    runtime=PolarsSubprocessConfig(binary_path="/path/to/binary"),
    report_dir="/var/log/polars/license-server",
    license_path="/etc/polars/license.json",
    tls_bundle_path="/etc/polars/tls-bundle.pem",
)

ray.init(address="auto", namespace="license-server")
license_server = PolarsRayLicenseServer(config)
license_server.start()

A cluster then validates against it with:

from polars_nanook_ray.config import PolarsLicenseServerConfig

license = PolarsLicenseServerConfig(uri=license_server.get_bind_addr())

To reconnect to (or stop) an already-running license server from another process, call ray.init() with the same namespace it was started under, then PolarsRayLicenseServer(config).start() (reconnects) or .stop().

Resource requests and limits

Four parameters control resource usage:

  • cpus_hint / memory_hint: forwarded verbatim as Ray's own num_cpus / memory actor options, and used by Ray for bin-packing. Not enforced at the OS level for subprocesses, but enforced as --cpus / --memory for containers (see Running in containers).
  • cpu_reserved: a scheduling/accounting hint consumed internally by the binary for task placement and reported to the observatory; never enforced.
  • memory_limit: enforced by the binary itself via cgroups; but only if a delegated cgroup subtree is made available. The containers started by this package mount cgroups read-only, so there memory_hint is the limit that applies.

A plain session/SSH shell does not provide one (everything lives flatly in one cgroup), so memory_limit fails outright with the following message:

Ensure cgroup is mounted and subgroups are delegated, or disable the memory limit in the configuration file.

Running in containers

The binaries can run in Docker containers rather than as subprocesses of the actors, for two reasons:

  • Resource enforcement: cpus_hint / memory_hint are passed to Docker as --cpus / --memory, which it enforces, whereas a plain session or SSH shell gives the binary no delegated cgroup subtree for its own memory_limit.
  • Running the binary where it is not native: the binary is released for Linux only.

Container images including the binaries still need to be built. Minimal examples are provided in the docker/ folder, and a recipe in the Makefile:

make image COMPUTE_VERSION=<x.x.x> PYTHON_VERSION=<x.y>

Then select the container runtime, naming that image:

from polars_nanook_ray.config import PolarsDockerConfig, PolarsRayClusterConfig

config = PolarsRayClusterConfig(
    runtime=PolarsDockerConfig(image="polars-nanook-ray:latest"),  # Makefile tag
    ...
)

Everything else, PolarsRayCluster and RayClusterContext included, remains identical. single_host_cluster does not apply: each container has its own network namespace, so worker ports never collide.

The license server runs the same way, from its own image built with make image-license-server. Clusters then reach it by container name, so they have to run in containers on the same Docker network.

Metadata

Release files for polars-nanook-ray 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for polars-nanook-ray 0.2.0
File Size Uploaded Provenance Actions
polars_nanook_ray-0.2.0.tar.gz 38.5 kB With provenance Details for polars_nanook_ray-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for polars-nanook-ray 0.2.0
File Interpreter ABI Platform Actions
polars_nanook_ray-0.2.0-py3-none-any.whl Python 3 none any Details for polars_nanook_ray-0.2.0-py3-none-any.whl

Total release size: 74.8 kB

Release files / polars_nanook_ray-0.2.0.tar.gz

Download URL polars_nanook_ray-0.2.0.tar.gz
Size 38.5 kB
Tags Source
SHA-256 checksum
How to use checksums
fa0a9210c0368e7cc4ddf9d8984a3fcf1be10f5ec382320cedb6caadbd5cf84f
BLAKE2b-256 checksum
How to use checksums
af873df65a677a7211b1394a2068be54eb7db312bba5228c6e17ff69919c2a0d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Release files / polars_nanook_ray-0.2.0-py3-none-any.whl

Download URL polars_nanook_ray-0.2.0-py3-none-any.whl
Size 36.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
efc7eb0cbcae4b9c60c3a3b0c712f57302f66a39678047f2eaf1432815827ae3
BLAKE2b-256 checksum
How to use checksums
1869f6eed6c1c9dcd0130570f7c695ee3c6cbb96e146ecba665bdd44753b91c8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Security for polars-nanook-ray 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for polars_nanook_ray-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page