Polars Nanook over Ray
Runs the Polars Nanook scheduler and workers as Ray actors.
Quickstart
Prerequisites for a completely local test deployment:
- A Polars Nanook
license.jsonfile, or a valid service account - A Python virtual environment:
uv venv
source .venv/bin/activate
uv pip install polars-nanook-ray
It will also install ray, and polars-nanook-compute which runs the scheduler
and workers, as dependencies. The latter pins the Polars version of the
environment, and is only installed on Linux, the only platform it is released
for; elsewhere, run the scheduler and workers in containers.
Note that a few things need to be set up for the Polars Nanook cluster to
function properly:
# need to match the config if manually changed
mkdir --parents /tmp/polars/{anonymous-results,license,observatory,shuffle-data,temporary-data}
This latter can be started using the following command:
ray start \
--dashboard-host=0.0.0.0 \
--disable-usage-stats \
--head \
--port="6379" \
--ray-client-server-port="10001" \
--resources='{"head":1}' # pinning
If you do not already have one, create a Polars Nanook service account through the Polars Nanook portal.
Spawn a local multinode cluster:
import polars as pl
import polars_nanook as pn
import ray
from polars_nanook_ray.cluster import PolarsRayCluster
from polars_nanook_ray.config import (
PolarsObservatoryConfig,
PolarsRayClusterConfig,
PolarsSchedulerConfig,
PolarsServiceAccountLicenseConfig,
PolarsSubprocessConfig,
PolarsWorkerConfig,
)
config = PolarsRayClusterConfig(
runtime=PolarsSubprocessConfig(single_host_cluster=True),
num_workers=4,
license=PolarsServiceAccountLicenseConfig(
workspace_id="<WORKSPACE_ID>",
client_id="<SERVICE_ACCOUNT_ID>",
client_secret="<SERVICE_ACCOUNT_SECRET>",
),
scheduler=PolarsSchedulerConfig(...),
worker=PolarsWorkerConfig(...),
)
ray.init(address="auto", namespace=config.cluster_id)
cluster = PolarsRayCluster(config)
cluster.start()
print(
pl.LazyFrame({"a": [1, 2, 3], "b": [4, 4, 5]})
.with_columns(pl.col("a").max().over("b").alias("c"))
.remote(pn.ClusterContext(uri=f"http://{cluster.get_client_addr()}"))
.execute()
.head
)
cluster.stop()
ray.shutdown()
or:
import polars as pl
import ray
from polars_nanook_ray.config import (
PolarsObservatoryConfig,
PolarsRayClusterConfig,
PolarsSchedulerConfig,
PolarsServiceAccountLicenseConfig,
PolarsSubprocessConfig,
PolarsWorkerConfig,
)
from polars_nanook_ray.context import RayClusterContext
config = PolarsRayClusterConfig(
runtime=PolarsSubprocessConfig(single_host_cluster=True),
num_workers=4,
license=PolarsServiceAccountLicenseConfig(
workspace_id="<WORKSPACE_ID>",
client_id="<SERVICE_ACCOUNT_ID>",
client_secret="<SERVICE_ACCOUNT_SECRET>",
),
scheduler=PolarsSchedulerConfig(...),
worker=PolarsWorkerConfig(...),
)
ray.init(address="auto", namespace=config.cluster_id)
with RayClusterContext(config) as ctx:
print(
pl.LazyFrame({"a": [1, 2, 3], "b": [4, 4, 5]})
.with_columns(pl.col("a").max().over("b").alias("c"))
.remote(ctx)
.execute()
.head
)
ray.shutdown()
In case Ray is running on a single host, set the single_host_cluster attribute
of PolarsSubprocessConfig to True to offset worker ports and avoid socket collisions.
The actors are running in detached mode and survive past the script: one can
reconnect with the same ray.init() and cluster gymnastics from another
process.
To clean all actors and underlying processes, Ray itself needs to be shutdown
using the following command:
ray stop --force
Autoscaling
A dedicated scaler actor, pinned to the scheduler node, runs an HTTP server to handle scaling requests sent by the underlying binary. These requests are relayed to the scheduler actor, which in turn adds or removes Ray worker actors in response.
Enable it via PolarsScalingConfig on the scheduler, and optionally set
min_workers and/or max_workers on the cluster config to bound how far it may
scale.
Note the cluster always starts num_workers workers: the bounds are reported
back to the binary on GET /scale_config, seeded into max_workers_per_query,
and only enforced on rescaling, where a POST /scale_to count outside them is
clamped.
Requesting more workers is done via the client: .distributed(min_workers=X).
The HTTP server is unauthenticated and binds 127.0.0.1, or the Docker network
gateway when the binaries run in containers. It accepts only
application/json bodies, and ignores worker names that are not worker actors.
License server
PolarsRayLicenseServer manages the Polars Nanook offline license server
binary as a Ray actor. Ray clusters point their configuration at it via
PolarsLicenseServerConfig(uri=...) and it validates them locally, tracking
usage into signed reports it periodically emits (and optionally uploads to our
control plane).
It is standalone: unlike the scheduler/worker/scaler, it is not wired into
PolarsRayCluster. It is meant to be a single, long-lived service that any number
of separate clusters register against, so its lifecycle (and Ray namespace) is
managed independently, and it should be started before any
cluster that points at it.
import ray
from polars_nanook_ray.config import (
PolarsRayLicenseServerConfig,
PolarsSubprocessConfig,
)
from polars_nanook_ray.license_server import PolarsRayLicenseServer
config = PolarsRayLicenseServerConfig(
runtime=PolarsSubprocessConfig(binary_path="/path/to/binary"),
report_dir="/var/log/polars/license-server",
license_path="/etc/polars/license.json",
tls_bundle_path="/etc/polars/tls-bundle.pem",
)
ray.init(address="auto", namespace="license-server")
license_server = PolarsRayLicenseServer(config)
license_server.start()
A cluster then validates against it with:
from polars_nanook_ray.config import PolarsLicenseServerConfig
license = PolarsLicenseServerConfig(uri=license_server.get_bind_addr())
To reconnect to (or stop) an already-running license server from another
process, call ray.init() with the same namespace it was started under, then
PolarsRayLicenseServer(config).start() (reconnects) or .stop().
Resource requests and limits
Four parameters control resource usage:
cpus_hint/memory_hint: forwarded verbatim as Ray's ownnum_cpus/memoryactor options, and used by Ray for bin-packing. Not enforced at the OS level for subprocesses, but enforced as--cpus/--memoryfor containers (see Running in containers).cpu_reserved: a scheduling/accounting hint consumed internally by the binary for task placement and reported to the observatory; never enforced.memory_limit: enforced by the binary itself via cgroups; but only if a delegated cgroup subtree is made available. The containers started by this package mount cgroups read-only, so therememory_hintis the limit that applies.
A plain session/SSH shell does not provide one (everything lives flatly in one
cgroup), so memory_limit fails outright with the following message:
Ensure cgroup is mounted and subgroups are delegated, or disable the memory limit in the configuration file.
Running in containers
The binaries can run in Docker containers rather than as subprocesses of the actors, for two reasons:
- Resource enforcement:
cpus_hint/memory_hintare passed to Docker as--cpus/--memory, which it enforces, whereas a plain session or SSH shell gives the binary no delegated cgroup subtree for its ownmemory_limit. - Running the binary where it is not native: the binary is released for Linux only.
Container images including the binaries still need to be built. Minimal
examples are provided in the docker/ folder, and a recipe in the Makefile:
make image COMPUTE_VERSION=<x.x.x> PYTHON_VERSION=<x.y>
Then select the container runtime, naming that image:
from polars_nanook_ray.config import PolarsDockerConfig, PolarsRayClusterConfig
config = PolarsRayClusterConfig(
runtime=PolarsDockerConfig(image="polars-nanook-ray:latest"), # Makefile tag
...
)
Everything else, PolarsRayCluster and RayClusterContext included, remains
identical. single_host_cluster does not apply: each container has its own
network namespace, so worker ports never collide.
The license server runs the same way, from its own image built with
make image-license-server. Clusters then reach it by container name, so they
have to run in containers on the same Docker network.
Metadata
Release files for polars-nanook-ray 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| polars_nanook_ray-0.2.0.tar.gz | 38.5 kB | With provenance | Details for polars_nanook_ray-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| polars_nanook_ray-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for polars_nanook_ray-0.2.0-py3-none-any.whl |
Total release size: 74.8 kB
Release files / polars_nanook_ray-0.2.0.tar.gz
| Download URL | polars_nanook_ray-0.2.0.tar.gz |
|---|---|
| Size | 38.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fa0a9210c0368e7cc4ddf9d8984a3fcf1be10f5ec382320cedb6caadbd5cf84f
|
|
BLAKE2b-256 checksum How to use checksums |
af873df65a677a7211b1394a2068be54eb7db312bba5228c6e17ff69919c2a0d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logRelease files / polars_nanook_ray-0.2.0-py3-none-any.whl
| Download URL | polars_nanook_ray-0.2.0-py3-none-any.whl |
|---|---|
| Size | 36.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
efc7eb0cbcae4b9c60c3a3b0c712f57302f66a39678047f2eaf1432815827ae3
|
|
BLAKE2b-256 checksum How to use checksums |
1869f6eed6c1c9dcd0130570f7c695ee3c6cbb96e146ecba665bdd44753b91c8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logSecurity for polars-nanook-ray 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for polars_nanook_ray-0.2.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.