quapp-hpc 0.0.1.dev5

Quapp HPC library — Slurm integration for Quapp Platform

quapp-hpc

Python library cho Quapp HPC functions — cầu nối giữa Quapp FaaS platform và Slurm HPC cluster.

Architecture

ksvc (Docker)
├── index.py                  FastAPI server
├── quapp_hpc/
│   ├── factory/
│   │   └── hpc_handler_factory.py   Entry point cho user
│   ├── component/backend/
│   │   └── hpc_invocation.py        Orchestrates job lifecycle
│   └── model/
│       ├── provider/slurm_provider.py   Auth headers, base URL
│       └── device/slurm_device.py       Submit → Poll → S3 download
└── function/
    └── handler.py            User-defined processing() + post_processing()

Luồng thực thi

index.py nhận HTTP POST
    → HpcHandlerFactory.create_handler(event, processing_fn, post_processing_fn)
    → InvocationHandler.handle()
    → HpcInvocation.submit_job()
        1. processing_fn(invocation_input)  → bash script string
        2. SlurmDevice._create_job(script)  → POST Slurm REST API → slurm_job_id
        3. SlurmDevice._get_job_result()    → poll mỗi 30s → COMPLETED/FAILED
        4. SlurmDevice._download_s3_result()→ boto3 get s3://$S3_BUCKET/$JOB_UUID/output.json
        5. post_processing_fn(s3_result)    → final response

Environment variables (từ K8s Secret slurm-credentials)

Var	Ví dụ	Mô tả
SLURM_API_URL	http://10.1.0.15:6820	Slurm REST API base URL
SLURM_JWT	eyJ...	JWT token cho Slurm auth
SLURM_USERNAME	quapp-svc	Slurm username
SLURM_ACCOUNT	quapp	Slurm account/allocation
S3_BUCKET	quapp-slurm-output-dev	S3 bucket cho job output
AWS_REGION	ap-southeast-1	AWS region
SLURM_POLL_SEC	30	Polling interval (giây)
SLURM_TIMEOUT_SEC	21600	Max wait time (giây, default 6h)

invocation_input schema

Xem chi tiết tại ../qapp-sdk-templates/slurm-hpc/README.md.

Tóm tắt:

{
  "resources": { "partition", "nodes", "cpus_per_task", "gpus", "memory_gb", "time_limit" },
  "container": { "type": "sif"|"docker"|"none", "image": "..." },
  "job":       { "type": "script"|"command", "script"|"command": "...", "environment": {}, "input_s3_paths": [] }
}

Slurm REST API

• Version: v0.0.40
• Submit: POST {SLURM_API_URL}/slurm/v0.0.40/job/submit
• Status: GET {SLURM_API_URL}/slurm/v0.0.40/job/{job_id}
• Auth headers: X-SLURM-USER-NAME, X-SLURM-USER-TOKEN

Job state mapping

Slurm state	Quapp state
PENDING, CONFIGURING, RUNNING, COMPLETING	RUNNING
COMPLETED	DONE
FAILED, CANCELLED, TIMEOUT, NODE_FAIL, PREEMPTED	ERROR

S3 output pattern

Job script phải upload kết quả:

aws s3 cp /tmp/output.json s3://$S3_BUCKET/$JOB_UUID/output.json

$JOB_UUID và $S3_BUCKET được inject tự động bởi SlurmDevice._create_job() qua Slurm environment array.

K8s Secret

# infrastructure/quapp-job-scheduler/k8s/cts/slurm-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: slurm-credentials
  namespace: quapp-functions-dev
stringData:
  SLURM_JWT: "<generate: sudo scontrol token username=quapp-svc lifespan=2592000>"
  SLURM_API_URL: "http://10.1.0.15:6820"
  SLURM_USERNAME: "quapp-svc"
  SLURM_ACCOUNT: "quapp"
  S3_BUCKET: "quapp-slurm-output-dev"
  AWS_REGION: "ap-southeast-1"

DB seed required

Chạy script trước khi deploy:

infrastructure/quapp-functions-backend/docs/db/seed_slurm_hpc.sql