natscale 0.1.2

No project description provided

natscale

A high-performance Python framework for HPC task distribution, load balancing, and monitoring powered by NATS.

Install NATS

deploy a NATS service

go install github.com/nats-io/nats-server/v2@latest

export PATH=$PATH:$(go env GOPATH)/bin

# -js 表示开启 JetStream 持久化功能
# -m 8222 (可选) 开启 HTTP 监控面板，方便在浏览器看状态
nats-server -js

create a NATS stream

nats stream add NATSCALE --subjects "hpc.tasks.>" --storage=file --ack --retention=work --discard new   --max-msgs 100000 --replicas 1

useful commands

check the stream info

watch -n 10 nats stream info NATSCALE

view the stream graph

nats stream graph

Usage

Manually send an ACK signal to confirm that the task has been successfully received and processed.

import time
import natscale as ns

cfg = ns.Config(
    nats_server="nats://127.0.0.1:4222",
    subject="netscale.tasks.*",
    timeout=3,
    retry = 4,
    auto_ack=False,
)

# manually do ack
with ns.Iterator(cfg) as tasks:
    for data, done in tasks:
        print(f"{data.id} --> {data}")
        time.sleep(1)
        done()

Auto send an ACK signal to confirm that the task has been successfully received and processed.

import time
import natscale as ns
from natscale.task.iter import NatsIterator as Iterator

cfg = ns.Config(
    nats_server="nats://127.0.0.1:4222",
    subject="netscale.tasks.*",
    timeout=3,
    retry = 4,
    auto_ack=True,
)

with Iterator(cfg) as tasks:
    for data in tasks:
        print(f"{data.id} --> {data}")
        time.sleep(1)

template-pdm-base

This project is generated by pdm template of https://github.com/POFK/template-pdm-base by command:

pdm init https://github.com/POFK/template-pdm-base <project_name>

also a non-interactive command is:

pdm init --name <project_name> --dist -n  https://github.com/POFK/template-pdm-base

first run after initialization

pdm install
detect-secrets scan > .secrets.baseline

git add .
pre-commit run -a