SLURM Emulator - Time Travel Edition

A comprehensive SLURM command emulator with time manipulation capabilities for testing periodic limits and decay calculations.

Features

• 🎮 Interactive CLI - Full command-line interface with time travel
• ⏰ Time Manipulation - Advance time by days, months, or quarters
• 💾 Usage Injection - Add specific node-hour usage at any time point
• 🔄 Decay Calculations - 15-day half-life fairshare decay simulation
• 🎯 QoS Management - Threshold-based QoS switching (normal → slowdown → blocked)
• 📊 Periodic Limits - Quarterly allocation with carryover logic
• 🎬 Scenario Runner - Complete SLURM_PERIODIC_LIMITS_SEQUENCE.md validation
• 🖥️ Web Dashboard - Browser console to view status and drive the emulator (docs)
• 🔌 API Integration - REST API for waldur-site-agent integration
• 💾 State Management - Checkpoint/restore functionality for testing
• ☸️ Helm Chart - Single-command Kubernetes deploy of both API planes (docs)

Quick Start

Installation

# Clone the repository
git clone https://github.com/waldur/slurm-emulator.git
cd slurm-emulator

# Install dependencies using uv
uv sync

Interactive CLI (CMD-based)

# Run with default configuration
uv run slurm-emulator

# Run with SLURM configuration file
uv run slurm-emulator --config examples/slurm.conf

# Advanced features work the same way
uv run slurm-emulator --config examples/custom_slurm.conf

# Validate configuration only
uv run slurm-emulator --validate-only --config /etc/slurm/slurm.conf

🎮 SLURM Emulator - Time Travel Edition (CMD Interface)
Type 'help' or '?' for commands. TAB for auto-completion.
Type 'help <command>' for detailed help on specific commands.

# Note: native emulator commands use underscores (time_advance, account_create,
# usage_inject, ...). The SLURM passthrough commands (sacctmgr, sacct, sinfo,
# sshare) take their arguments with spaces, as on a real cluster.

[default] slurm-emulator> help
# Shows all available commands

[default] slurm-emulator> help time_advance
# Shows detailed help for specific command

[default] slurm-emulator> time_advance 2 months
⏭️  Advanced 2 months

[default] slurm-emulator> account_create test-account "Test Account" 1000
✅ Created account test-account with 1000Nh allocation

[default] slurm-emulator> usage_inject user1 200 test-account
💾 Injected 200.0Nh usage for user1 in test-account at 2024-01-01 00:00:00

[default] slurm-emulator> time_advance 2 months
⏭️  Advanced 2 months
⏰ New time: 2024-03-01 00:00:00

[default] slurm-emulator> usage_inject user1 400 test-account
💾 Injected 400.0Nh usage for user1 in test-account at 2024-03-01 00:00:00

[default] slurm-emulator> limits_calculate test-account
📊 Periodic Limits for test-account:
   Period: 2024-Q1
   Base allocation: 1000Nh
   Total allocation: 1000.0Nh
   Fairshare: 333
   QoS threshold: 1200.0Nh
   Grace limit: 1200.0Nh
   Billing minutes: 60000

Complete Sequence Scenario

Run the full scenario from SLURM_PERIODIC_LIMITS_SEQUENCE.md:

[default] slurm-emulator> scenario_run sequence --interactive

🎬 Starting SLURM Periodic Limits Sequence Scenario
============================================================

⏸️  Press Enter to execute Step 1: Initial Q1 setup...

📍 Step 1: Initial Q1 2024 Setup
   Setting up 1000Nh quarterly allocation with 20% grace period
   ⚖️  Set fairshare to 333
   🚫 Set GrpTRESMins to 72000 billing-minutes
   🎯 QoS threshold set to 1200.0Nh
   💾 Checkpoint 'initial_setup' created

# ... continues through all 9 steps of the sequence

Direct SLURM Commands

The emulator intercepts and emulates real SLURM commands:

[default] slurm-emulator> sacctmgr add account test-account description="Test"
 Adding Account(s)
  test-account
 Settings
  Parent     = root
  Description = Test

[default] slurm-emulator> sacctmgr modify account test-account set fairshare=333
 Modified account...
  test-account
 Settings
  fairshare=333

[default] slurm-emulator> sacctmgr modify account test-account set GrpTRESMins=billing=72000
 Modified account...
  test-account
 Settings
  GrpTRESMins=billing=72000

[default] slurm-emulator> sacct --accounts=test-account --starttime=2024-01-01 --endtime=2024-12-31
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
1                 job_1    compute test-acco+         64  COMPLETED      0:0

[default] slurm-emulator> sacct --accounts=test-account -S 2024-01-01 --format=Account,ReqTRES,Elapsed,User --noheader --parsable2
test-account|cpu=64,mem=512G,node=1,billing=64,gres/gpu=4|08:00:00|user1

Energy accounting and sreport

Federated allocation portals ask sites for the energy consumed per SLURM account (optionally per user) each month, taken from SLURM's own account aggregation — sreport cluster AccountUtilizationByUser -T energy — rather than from job rows. The emulator reproduces that pipeline:

• Every usage record carries an energy TRES in joules (the unit real slurmdbd uses for TRES_ENERGY). It comes from a per-node power model unless a scenario seeds an exact figure:

  joules = node_hours × 3600 × node_watts(partition) + gpu_hours × 3600 × gpu_watts
  

  Variable	Default	Meaning
  SLURM_EMULATOR_NODE_POWER_W	500	watts per node for partitions not listed below
  SLURM_EMULATOR_PARTITION_POWER_W	–	per-partition overrides, e.g. compute=400,gpu=900
  SLURM_EMULATOR_GPU_POWER_W	300	extra watts per allocated GPU (GRES/gpu hours)

  Jobs completed through the scheduler (sbatch / POST /job/submit) use the same model with their partition; usage_inject records run in compute.

• POST /api/submit-report accepts energy (joules) in usage or in each users entry, plus an optional partition; an explicit value bypasses the model so a month's total can be asserted exactly. The regular_access_energy scenario seeds one month of CPU + GPU usage with known energy for this purpose.

• sreport cluster AccountUtilizationByUser is emulated with the real client's grammar — start=/end= (same time specs as sacct, window on the simulated clock, end exclusive, defaults yesterday→today), accounts=, users=, cluster=/-M/-a, format=, -T/--tres= (comma list; cpu, mem, node, billing, gres/gpu, energy, ALL), -t Seconds|Minutes|Hours|Percent|SecPer|MinPer|HourPer, -p/-P/-n. One row per association and TRES, the account total first, then its users, then sub-accounts:

  $ sreport cluster AccountUtilizationByUser start=2024-03-01 end=2024-04-01 \
      -T energy -t Seconds --parsable2 -n accounts=regular_access
  default|regular_access|||energy|424800000
  default|regular_access|alice||energy|180000000
  default|regular_access|bob||energy|244800000
  

  Two things to know from the real source (src/sreport/sreport.c): there is no -t Joules — the energy TRES is stored in the same slot the time units divide, so joules are what -t Seconds prints (-t Minutes gives joules/60; an unknown format prints unknown time format X and the report continues in Minutes); and unknown TRES names are silently dropped — only an empty list fails (sreport: fatal: No valid TRES given, exit 1). Only this one report is emulated; the header block is printed unless -n, also in parsable mode.

• sacct gains the real ConsumedEnergy / ConsumedEnergyRaw columns (joules per job, scaled K/M/G unless --noconvert). The ReqTRES/AllocTRES string does not include energy, so the site agent's --format=Account,ReqTRES,Elapsed,User output is unchanged. sshare GrpTRESRaw sums it like any other TRES.

Web Dashboard

A lightweight, browser-based control console is mounted on the API server at http://localhost:8080/ui/. It shows live status (time/period, accounts, usage, QoS, jobs, cluster config) and provides full control — advance/set time, create/edit accounts, inject usage, apply periodic settings, inline QoS editing, add/remove account users, and a scenario editor to build/adjust and run scenarios. It shares the same in-memory state as the CLI and JSON API.

# Start the server (Basic-auth credentials via env vars)
SLURM_EMULATOR_UI_USER=admin SLURM_EMULATOR_UI_PASSWORD=secret \
  uv run uvicorn emulator.api.emulator_server:app --host 0.0.0.0 --port 8080
# then open http://localhost:8080/ui/

All /ui routes are protected by HTTP Basic auth (SLURM_EMULATOR_UI_USER / SLURM_EMULATOR_UI_PASSWORD, default admin/admin with a startup warning). See docs/web-ui.md for a full walkthrough with screenshots.

API Integration

Start the API server for waldur-site-agent integration:

# From the slurm-emulator directory
uv run uvicorn emulator.api.emulator_server:app --host 0.0.0.0 --port 8080

API Endpoints

• POST /api/apply-periodic-settings - Apply periodic limits settings
• POST /api/downscale-resource - Set QoS to slowdown
• POST /api/restore-resource - Restore QoS to normal
• POST /api/submit-report - Submit usage reports
• GET /api/status - Get emulator status
• POST /api/time/advance - Advance emulator time

Example API Usage

# Apply periodic settings (from Waldur Mastermind)
curl -X POST http://localhost:8080/api/apply-periodic-settings \\
  -H "Content-Type: application/json" \\
  -d '{
    "resource_id": "slurm_account_123",
    "fairshare": 333,
    "grp_tres_mins": {"billing": 72000},
    "qos_threshold": {"billing": 1000}
  }'

# Submit usage report (from site agent)
curl -X POST http://localhost:8080/api/submit-report \\
  -H "Content-Type: application/json" \\
  -d '{
    "resource_id": "slurm_account_123",
    "usage": {"billing": 167},
    "billing_period": "2024-01-01",
    "date": "2024-01-31T23:59:59Z",
    "users": {
      "user1": {"billing": 100},
      "user2": {"billing": 67}
    }
  }'

# Advance time for testing
curl -X POST "http://localhost:8080/api/time/advance?months=3"

SLURM REST API Emulation (slurmrestd)

The emulator also serves the Slurm 26.05 REST API (slurmrestd, data parser v0.0.45) on port 6820, backed by the same state as the CLI commands and the control API:

uv run slurmrestd-emulator
# or: uv run uvicorn emulator.api.slurmrestd.app:app --host 0.0.0.0 --port 6820

Endpoint families

• /slurmdb/v0.0.45/... — accounting: accounts, users, associations, qos, tres, clusters, jobs (one job per usage record, matching sacct output), ping, diag, config. Write support (POST/DELETE) covers everything Waldur drives via sacctmgr.
• /slurm/v0.0.45/... — controller read paths: jobs (+ DELETE /job/{job_id} as the scancel equivalent), nodes, partitions (static topology matching sinfo), shares, ping, diag, conf; reservations/licenses are empty stubs.
• /openapi.json, /openapi, /openapi/v3 — generated self-description.

Responses use the real envelope (meta/errors/warnings, payload keys and field names from the v0.0.45 data parser). Unsupported URL versions (e.g. v0.0.45), unknown paths, and auth failures reject with slurmrestd's plain-text errors and exit statuses.

Authentication

Requests need an X-SLURM-USER-TOKEN header (or Authorization: Bearer); X-SLURM-USER-NAME optionally names the user. By default any non-empty token is accepted. Set SLURM_EMULATOR_JWT_KEY to enforce real HS256 verification. Mint tokens via the control API (scontrol token stand-in):

curl -X POST http://localhost:8080/api/token \
  -H "Content-Type: application/json" -d '{"username": "alice"}'

curl http://localhost:6820/slurmdb/v0.0.45/accounts/ \
  -H "X-SLURM-USER-TOKEN: <token>"

State sharing and concurrency

Both API servers and the CLI commands persist to the same JSON state files (/tmp/slurm_emulator_db.json, /tmp/slurm_emulator_time.json; override with SLURM_EMULATOR_STATE_FILE / SLURM_EMULATOR_TIME_FILE). The slurmrestd app reloads state on every request and saves after writes — file locking prevents torn writes, but concurrent writers are last-writer-wins. Note the control API on 8080 loads state once at startup, so it can serve stale reads after REST/CLI writes.

The Docker image runs both servers (ports 8080 and 6820) via scripts/docker-entrypoint.sh.

Run on Kubernetes (via Helm)

A published Helm chart deploys the emulator as a single-replica Deployment + ClusterIP Service exposing the control API (8080), the slurmrestd plane (6820), and optionally the SSH plane (2222). Consumers in the same cluster reach it at http://<release>-slurm-emulator.<ns>.svc.cluster.local:8080.

helm repo add slurm-emulator https://waldur.github.io/slurm-emulator/
helm install se slurm-emulator/slurm-emulator \
  --namespace se --create-namespace --wait
helm test se -n se        # probes both API planes

Set auth.uiPassword (dashboard) and auth.jwtKey (slurmrestd) before exposing it anywhere shared — both default to accepting anything. persistence.enabled=true keeps the clock, accounts, and usage records across restarts.

See docs/kubernetes.md for the full operator guide (credentials, persistence, SSH plane, Ingress/Gateway API, troubleshooting). The chart source lives at charts/slurm-emulator/ — also installable from disk via helm install se ./charts/slurm-emulator.

SSH Filesystem Plane

For clients that expect a login node in addition to slurmrestd (notably FireCREST v2), the emulator ships an asyncssh server that provides filesystem operations and dispatches the Slurm CLI commands (sacctmgr, sacct, sinfo, sshare, scancel, sbatch) against the same shared state:

uv run --extra ssh slurm-ssh-emulator   # asyncssh server on port 2222

It shares the same JSON state files as the CLI, control API, and slurmrestd.

NSS identity (SLURM_EMULATOR_NSS=1)

By default the emulator has no uid/gid model: every user is 1000 and its own group. Setting SLURM_EMULATOR_NSS=1 makes it resolve user names through the OS name service switch (getpwnam/getgrgid, the calls real Slurm makes) — inside the Docker image that is libc → sssd → LDAP, with a reference examples/nss/sssd.conf and a one-user examples/nss/seed.ldif. With the mode on:

• id over SSH prints the coreutils shape FireCREST's /status/userinfo parses — uid=9001(hpc_9001) gid=9001(hpc_9001) groups=9001(hpc_9001) — and honours -u/-g/-G/-n; an unknown user exits 1;
• POST /slurm/v0.0.45/job/submit and sbatch record the submitter's uid/gid/group on the job (user_id/group_id/group_name, scontrol show job UserId=name(uid) GroupId=group(gid), sacct UID/GID/Group) and refuse a name the OS cannot resolve (ESLURM_USER_ID_UNKNOWN, HTTP 422, like the real USER_ID parser);
• sacctmgr add user for an unresolvable name stops with There is no uid for user 'x' (exit 1) unless -i/--immediate is given, mirroring real sacctmgr's prompt; the REST /slurmdb user upsert is not gated, so a site agent can create accounts before the directory entry exists;
• sreport fills Proper Name from the gecos field;
• independently of NSS, SLURM_EMULATOR_ACCOUNTING_ENFORCE=associations (the default everywhere; slurm.conf itself leaves it unset, but every managed production cluster sets it) makes POST /job/submit and sbatch refuse a user with no association for the requested — or, when none is requested, default — account: ESLURM_INVALID_ACCOUNT "Invalid account or account/partition combination specified" (HTTP 422), so a person removed from a project can no longer submit against it. Behaviour change in 0.10: earlier releases let any user submit and fell back to root; set SLURM_EMULATOR_ACCOUNTING_ENFORCE=none to get that back. The check is cluster-scoped, so a deployment whose site agent uses cluster_name: linux must set SLURM_EMULATOR_CLUSTER_NAME=linux (chart clusterName) or the agent's rows land on a cluster the submit never looks at. As in slurmdbd, a user's default association cannot be removed while other associations remain (You can not remove the default account of a user), and removing the last one removes the user;
• when the emulator runs as root (the Docker image does), shell commands over SSH run as the login user — uid, gid, supplementary groups, $HOME — so id, stat, chown and file ownership behave like a real login node; a leading coreutils timeout N wrapper (which FireCREST puts in front of every SSH command) is stripped before matching the emulated Slurm binaries.

Try it with the FireCREST UI stack:

cd examples/firecrest/ui
docker compose -f docker-compose.yml -f docker-compose.nss.yml up -d
ldapadd -x -H ldap://localhost:3890 -D cn=admin,dc=example,dc=org -w admin -f ../../nss/seed.ldif
docker compose exec slurm id hpc_9001

In Kubernetes the chart exposes it as nss.enabled + nss.sssdConfSecret; sssd runs as root, so the option excludes a non-root securityContext.

Running FireCREST v2 against the emulator

The scheduler plane (slurmrestd) plus the SSH filesystem plane let the emulator stand in for a real cluster for eth-cscs/firecrest-v2. See examples/firecrest/ for the conformance matrix, a docker-compose overlay, and contract/integration test harnesses.

Waldur Site Agent Integration

Configure waldur-site-agent to use the emulator:

# waldur-site-agent-config.yaml
offerings:
  - name: "SLURM HPC Cluster - Emulator"
    backend_type: "slurm"
    backend_settings:
      # Enable emulator mode
      emulator_mode: true
      emulator_base_url: "http://localhost:8080"

      # Override SLURM commands to use emulator
      command_prefix: ["python", "/path/to/slurm-emulator/emulator/commands/dispatcher.py"]

      # Periodic limits configuration
      periodic_limits:
        enabled: true
        limit_type: "GrpTRESMins"
        tres_billing_enabled: true
        tres_billing_weights:
          CPU: 0.015625
          Mem: 0.001953125G
          "GRES/gpu": 0.25
        fairshare_decay_half_life: 15
        api_endpoints:
          apply_periodic_settings: "http://localhost:8080/api/apply-periodic-settings"
          downscale_resource: "http://localhost:8080/api/downscale-resource"
          restore_resource: "http://localhost:8080/api/restore-resource"

SLURM Configuration Support

The emulator now supports real SLURM configuration files to match actual deployment behavior:

Loading Configuration

# Use system SLURM configuration
uv run slurm-emulator --config /etc/slurm/slurm.conf

# Use custom configuration
uv run slurm-emulator --config examples/slurm.conf

# Validate configuration
uv run slurm-emulator --validate-only --config slurm.conf

Supported Configuration Parameters

The emulator parses and applies these SLURM configuration parameters:

Priority and Decay Settings:

• PriorityDecayHalfLife - Fairshare decay half-life (e.g., "15-00:00:00")
• PriorityUsageResetPeriod - Usage reset period ("None" for manual reset)
• PriorityWeightFairShare - Fairshare weight for priority calculations
• PriorityWeightQOS - QoS weight for priority calculations
• FairShareDampeningFactor - Dampening factor for fairshare

TRES Billing:

• TRESBillingWeights - Billing weights (e.g., "CPU=0.015625,Mem=0.001953125G,GRES/gpu=0.25")

Priority Flags:

• PriorityFlags - Priority calculation flags (e.g., "MAX_TRES,NO_NORMAL_ASSOC")

Example Configuration

# SLURM Configuration
PriorityDecayHalfLife   = 15-00:00:00
PriorityUsageResetPeriod = None # manual reset via sacctmgr RawUsage=0
PriorityWeightFairShare = 259200
PriorityWeightQOS       = 500000
FairShareDampeningFactor = 3
TRESBillingWeights="CPU=0.015625,Mem=0.001953125G,GRES/gpu=0.25"
PriorityFlags=MAX_TRES,NO_NORMAL_ASSOC

Understanding Decay Calculations

The emulator implements SLURM's fairshare decay using the configured half-life:

# Decay formula matches SLURM's implementation
decay_factor = 2 ** (-days_elapsed / half_life_days)

# With default 15-day half-life, after 90 days (1 quarter):
decay_factor = 2 ** (-90 / 15) = 0.0156 (1.56%)

# With 7-day half-life, after 90 days:
decay_factor = 2 ** (-90 / 7) = 0.000135 (0.01%)

Example with 15-day half-life: User consumes 2000 hours in Q1. After Q1 ends (90 days later):

• Original impact: 2000 hours
• Decayed impact: 2000 × 0.0156 = 31 hours equivalent
• Q2 allocation: 1000 + (1000 - 31) = 1969 hours available

Key Commands Reference

Time Manipulation

time_show                             # Show current time and period
time_advance <amount> <unit>          # Advance time (units: days, months, quarters)
time_set YYYY-MM-DD [HH:MM:SS]       # Set specific date/time

# Examples:
time_advance 2 months
time_advance 30 days
time_set 2024-05-20

Usage Simulation

usage_inject <user> <amount> [account]  # Inject node-hour usage
usage_show [account] [period]           # Show usage summary with user breakdown

# Examples:
usage_inject user1 200 test-account
usage_show test-account
usage_show test-account 2024-Q1

Account Management

account_create <name> [description] [allocation]  # Create account
account_list                                      # List all accounts with status
account_show <name>                               # Show detailed account info
account_delete <name>                             # Delete account

# Examples:
account_create test "Test Account" 1000
account_show test
account_list

QoS Management

qos_show [account]                    # Show QoS status and details
qos_set <account> <qos>              # Set QoS level (normal/slowdown/blocked)
qos_check [account]                  # Check thresholds and auto-update QoS

# Examples:
qos_check test-account
qos_set test-account slowdown
qos_show test-account

Limits Calculation

limits_calculate [account]           # Calculate and display periodic limits

# Example:
limits_calculate test-account

Scenario Management

scenario_list [type]                 # List scenarios (optionally filter by type)
scenario_describe <name>             # Show detailed description and learning objectives
scenario_steps <name>                # Show step-by-step command breakdown
scenario_run <name>                  # Run scenario automatically
scenario_run <name> --interactive    # Run with confirmation prompts
scenario_run <name> --step-by-step   # Run with detailed step output
scenario_search <query>              # Search scenarios by keyword

# Examples:
scenario_list qos_management
scenario_describe qos_thresholds
scenario_run qos_thresholds --step-by-step
scenario_search decay

Configuration Management

config_show                          # Show current SLURM configuration
config_reload <path>                 # Hot-reload configuration file

# Examples:
config_show
config_reload examples/slurm.conf

State Management

cleanup_all                          # Clean all accounts and reset to fresh state
cleanup_scenario <name>              # Clean specific scenario accounts
cleanup_account <name>               # Clean specific account completely

# Examples:
cleanup_all
cleanup_scenario qos_thresholds
cleanup_account test-account

SLURM Commands

sacctmgr <args>                      # Run sacctmgr command
sacct <args>                         # Run sacct command
sinfo <args>                         # Run sinfo command
sshare <args>                        # Run sshare command
sreport <args>                       # Run sreport command (cluster AccountUtilizationByUser)

# Examples:
sacctmgr list accounts
sacctmgr modify account test set fairshare=333
sacct --accounts=test --format=Account,User,Elapsed --noheader --parsable2
sshare -A test --parsable2
sreport cluster AccountUtilizationByUser start=2024-01-01 end=2024-02-01 -T energy -t Seconds -P -n

Testing Scenarios

Basic Usage Pattern

# Setup with specific configuration
uv run slurm-emulator --config examples/slurm.conf

# In emulator CLI:
time_set 2024-01-01
account_create test-account "Test" 1000

# Month 1: Light usage
usage_inject user1 100 test-account
time_advance 1 months

# Month 2: Heavy usage
usage_inject user1 600 test-account
limits_calculate test-account
qos_check test-account

# Quarter transition
time_advance 1 months
limits_calculate test-account

Configuration Testing

# Test different decay rates
uv run slurm-emulator --config examples/custom_slurm.conf

# Compare configurations
uv run slurm-emulator --validate-only --config examples/slurm.conf
uv run slurm-emulator --validate-only --config examples/custom_slurm.conf

Decay Validation

# Q1: Heavy usage
time_set 2024-01-01
account_create test-account "Test" 1000
usage_inject user1 1500 test-account

# Q2: Check decay impact
time_set 2024-04-01
limits_calculate test-account
# Should show ~23Nh effective previous usage (1500 * 0.0156)

QoS Threshold Testing

# Setup with 1000Nh allocation (1200Nh threshold with 20% grace)
account_create test-account "Test" 1000
qos_show test-account  # Should show "normal"

usage_inject user1 1100 test-account
qos_check test-account  # Should show approaching threshold

usage_inject user1 200 test-account  # Total: 1300Nh
qos_check test-account  # Should trigger slowdown QoS

Architecture

slurm-emulator/
├── emulator/
│   ├── core/
│   │   ├── time_engine.py          # Time manipulation
│   │   ├── database.py             # In-memory state
│   │   ├── slurm_config.py         # SLURM config parsing
│   │   ├── scheduler.py            # Submitted-job lifecycle
│   │   └── usage_simulator.py      # Usage injection
│   ├── commands/
│   │   ├── sacctmgr.py             # sacctmgr emulator
│   │   ├── sacct.py                # sacct emulator
│   │   ├── sshare.py               # sshare emulator
│   │   ├── print_fields.py         # SLURM-style column formatting
│   │   └── dispatcher.py           # Command routing / script entry points
│   ├── periodic_limits/
│   │   ├── calculator.py           # Decay & carryover
│   │   └── qos_manager.py          # QoS management
│   ├── scenarios/
│   │   ├── sequence_scenario.py    # Complete scenario
│   │   ├── scenario_registry.py    # Scenario discovery & running
│   │   └── limits_configuration_scenarios.py
│   ├── cli/
│   │   ├── main.py                 # Interactive CLI entry point
│   │   └── cmd_cli.py              # CMD-based CLI
│   └── api/
│       ├── emulator_server.py      # Control REST API (port 8080)
│       ├── ui/                     # HTMX + Jinja2 web dashboard (/ui)
│       ├── slurmrestd/             # slurmrestd emulation (port 6820)
│       └── ssh/                    # SSH filesystem + CLI plane (port 2222)
├── scripts/
│   ├── release.py                  # Release management
│   ├── changelog.sh                # Changelog generation
│   ├── generate_changelog_data.py  # Commit data collection
│   └── prompts/
│       └── changelog-prompt.md     # Changelog prompt template
├── examples/
│   └── firecrest/                  # FireCREST v2 integration example
└── tests/                          # Test suites

Development

Running Tests

uv run pytest

Launching as a specific Slurm release

SLURM_EMULATOR_SLURM_VERSION=25.11 uv run slurmrestd-emulator   # serves /slurm/v0.0.44/, release 25.11.7
SLURM_EMULATOR_SLURM_VERSION=24.11 uv run pytest                 # run the suite as 24.11

Tracked releases: 24.11, 25.05, 25.11, 26.05 (default), master. The choice drives the slurmrestd URL prefix, meta.slurm.release, and the few response shapes that changed between releases (emulator/slurm_version.py).

Parity with real Slurm

Every emulated behaviour cites SchedMD's source as slurm://<path>#<symbol>[@versions] and is checked against several Slurm releases (26.05 primary; 25.11, 25.05, 24.11 and master). uv run scripts/slurm_src.py update builds the local source cache, uv run scripts/check_slurm_refs.py --summary verifies the references. See docs/slurm-parity.md.

Releasing

# Full release: update version, generate changelog, tag, push
uv run scripts/release.py release X.Y.Z

# Skip changelog generation
uv run scripts/release.py release X.Y.Z --skip-changelog

Pushing the tag triggers GitLab CI/CD for testing and PyPI publishing.

Adding New Scenarios

# Create new scenario class
class CustomScenario:
    def __init__(self, time_engine, database):
        self.time_engine = time_engine
        self.database = database

    def run_scenario(self):
        # Implement scenario steps
        pass

Extending Commands

# Add new SLURM command support
class NewCommandEmulator:
    def handle_command(self, args):
        # Implement command logic
        return "command output"

Troubleshooting

State Persistence

Emulator state is saved to:

• /tmp/slurm_emulator_time.json - Current time
• /tmp/slurm_emulator_db.json - Database state

Common Issues

"Account not found": Create account first with account_create "No usage records": Inject usage with usage_inject "Time not advancing": Check time with time command "API connection failed": Ensure server is running on port 8080

Reset Emulator

rm /tmp/slurm_emulator_*.json
slurm-emulator
# Start fresh

License

MIT License - See LICENSE file for details.