Jarvis-HEP V2

Jarvis-HEP V2 is a distributed runtime for high-energy-physics parameter scans. You describe a scan as a validated YAML task card; Jarvis generates parameter points, evaluates external calculators and/or Python operators in parallel, and archives observables, sample artifacts, logs, and run summaries.

The PyPI distribution is named Jarvis-HEP (current version 2.0.17). Its Python import package remains jarvishep2, and it exposes a single user-facing command: Jarvis.

What V2 provides

The execution model is deliberately simple:

Task YAML → sampler → Redis → Workers → Archiver
                                      ├─ DATABASE/samples.hdf5
                                      ├─ SAMPLE/<bucket>/<uuid>/
                                      └─ logs/<scan>/ + run_summary.*

Installation

Requirements:

• Python 3.10 or newer
• A Redis-compatible server program (redis-server, redis6-server, or valkey-server) installed on the machine; Redis must be 6.2 or newer. You do not need to start a service: see below. On Windows, install and run both Jarvis-HEP and Redis inside WSL.

From PyPI:

python3 -m pip install Jarvis-HEP

From a source checkout:

python3 -m pip install -e .

For development and tests, install the test tools separately:

python3 -m pip install -e .
python3 -m pip install "pytest>=7.0" "fakeredis>=2.0" "colorlog>=6.0"

Install Redis, but do not start it as a service. Jarvis starts its own Redis server for each scan and stops it when the scan ends. For example, on macOS:

brew install redis   # install only; no `brew services start redis`

If another Redis is already running on port 6379 (for example one started with brew services start redis), Jarvis cannot use that port. For task cards that load the project's default environment (the cards made by Jarvis project create do), it picks the next free port and writes it into deps/environment_default.yaml. Other task cards stop with a "port is already used by another program" error instead.

After the first Jarvis command following installation, Jarvis checks whether a Redis-compatible server executable (redis-server, redis6-server, or valkey-server) is available. If it is missing, the command's normal output is followed by a one-time, OS-specific installation hint (written to stderr so JSON/stdout output remains usable). The hint is advisory only; Jarvis does not install operating-system packages automatically. This is a PATH check on first CLI use, not a hook run by pip install; it does not verify the server version. The check marker is stored at ~/.jarvis/redis-install-check-v1.

You can inspect the environment and read setup instructions at any time:

Jarvis redis                       # system, executable/version, install/start guidance
Jarvis redis check                 # repeat the PATH and bounded --version checks
Jarvis redis install               # print the installation commands for this OS
Jarvis redis start                 # explain automatic startup and optional manual debugging
Jarvis redis info --json           # machine-readable report; --json works on every topic
Jarvis redis --help                # list the info/check/install/start subcommands
Jarvis redis check --help          # help and options for this subcommand

Redis follows the CLI's group/subcommand help layout: the parent lists the four commands, and each Jarvis redis COMMAND --help describes that command's purpose and options. --json works before or after the subcommand, for example Jarvis redis --json check and Jarvis redis check --json.

Jarvis redis and Jarvis redis --json retain the info default. check exits 1 if the executable is absent, unusable, too old, or the runtime is native Windows. These commands do not contact a server, test ports, start a scan, install packages, or start services. install and start only print guidance; they do not invoke the server even to read its version. A successful environment check confirms the local executable, not a running service or scan readiness.

Installation guidance accounts for repository versions: Ubuntu 20.04/22.04 and Debian 11 receive the official signed Redis APT repository steps; RHEL-family 8 systems receive the redis:6 module instructions with the updated AppStream requirement. Older releases without a verified recipe receive documentation and upgrade guidance. Use Jarvis redis install for the full commands for your detected system.

The default install includes every Jarvis runtime dependency: the Redis Python client, msgpack, aiofiles, Textual Monitor, Jarvis-HEP-Portal, Jarvis-Operas, and JarvisPLOT. Jarvis does not publish optional dependency groups.

For the complete installation guide, Redis options, and project-packaging workflow, see INSTALL.md.

First run

The project scaffold includes a runnable Bridson + Operas example that does not need an external calculator:

Jarvis project create MyScan
cd MyScan

Jarvis validate bin/quickstart_bridson_operas.yaml
Jarvis run bin/quickstart_bridson_operas.yaml

The scaffold also contains calculator, CSV, Dynesty, and MultiNest examples. List the built-in examples with:

Jarvis man example

Writing a task card

A task card is the YAML file that describes one scan. Every Jarvis command (validate, check, run, man) reads the same set of fields:

• Every card needs three top-level sections: Scan (the scan name), Sampling (how parameter points are chosen), and EnvReqs (runtime settings).
• A card also needs at least one of Calculators (external programs) or Operas (Python functions) to compute results for each point.
• Settings for the chosen sampling method go under Sampling.Bounds, written in lowercase with underscores, e.g. point_number.
• Runtime settings go under EnvReqs.V2, for example the number of parallel workers and checkpoint.heartbeat (how often, in seconds, progress is saved so an interrupted scan can be resumed; minimum 30).
• To test your calculators on a few points before a full scan, use Jarvis check TASK.yaml.

A minimal card looks like this:

Scan:
  name: my_scan

Sampling:
  Method: Random
  Bounds:
    point_number: 100
    seed: 7
  Variables:
    - name: x
      distribution:
        type: Flat
        parameters: {min: 0.0, max: 1.0}

EnvReqs:
  V2:
    workers: 2

Operas:
  Modules:
    - name: my_operator
      operator: my_package.my_function

Jarvis has a built-in manual for every field, so you don't need to guess names:

Jarvis man                         # interactive YAML authoring guide
Jarvis man --json                  # structured output for tooling and agents
Jarvis man sampler                 # sampler catalog
Jarvis man sampler.ToyMCMC         # reference independent-chain MCMC Bounds
Jarvis man sampler.mcmc-runtime    # MCMC multi-chain Redis pipeline design
Jarvis man yaml.EnvReqs.V2         # one YAML section
Jarvis man calculator.execution.output --type JSON

Jarvis validate checks a card for mistakes without starting a scan (and without needing Redis). Run it before every Jarvis run.

CLI workflow

Jarvis validate TASK.yaml          # validate only
Jarvis check TASK.yaml             # quick test of calculators on a few points
Jarvis run TASK.yaml               # execute a distributed scan
Jarvis run TASK.yaml --resume      # resume from a checkpoint
Jarvis convert TASK.yaml           # refresh DATABASE/samples.csv from HDF5
Jarvis monitor                     # read one live monitor snapshot
Jarvis ps                          # list Jarvis process groups
Jarvis kill --yes                  # terminate selected runtime processes
Jarvis --refs                      # print framework and sampler references

Additional integrations are available through native subcommands:

Jarvis project create MyScan
Jarvis project browse
Jarvis project fetch Eggbox
Jarvis portal man
Jarvis operas list
Jarvis gen-plot-yaml TASK.yaml
Jarvis plot path/to/plot.yaml      # Jarvis-PLOT is installed by default

Use Jarvis COMMAND -h for command-specific options. Screen logging defaults to WARNING; use --console-level INFO, --debug, or --silence as needed. File logs remain available under logs/<scan>/.

AI agents and YAML assistance

Codex, Claude and other agents should start with the installed CLI contract:

Jarvis agent --brief --json        # short contract retains all rules/workflows
Jarvis man agent-readme --brief --json  # same contract, through the manual index
Jarvis agent --section commands --command run --json
Jarvis agent search "checkpoint heartbeat" --json
Jarvis agent inspect bin/task.yaml --section runtime --json
Jarvis agent schema --list --json  # small schema index
Jarvis agent schema --json         # bundled schemas; all references resolve locally
Jarvis agent template --json       # list task cards and Sampling fragments
Jarvis agent template random-operas --json
Jarvis validate bin/task.yaml --json

The contract includes the installed version, question routes, argument arrays, command effects, authorization boundaries and authoring workflows. Follow the installed man pages for field names and built-in sampler Bounds, and the owning Portal/Operas/PLOT CLI for runtime behavior. The schema bundle describes the bundled structural vocabulary; semantic checks still require validate. Jarvis agent schema --id URI prints one original schema document or local JSONPointer fragment from that bundle without fetching the URI. Read Jarvis man sampler.NAME for the chosen built-in method's Bounds. Omit --brief to read the full original contract; --section reads one section. Search performs keyword discovery and returns real manual/help next_argv; absence of a match is not proof that a feature is unsupported.

inspect explains the selected card's loaded defaults, merged environment, normalized runtime, planned paths and validation, with field sources. Use --section paths|runtime|sources|validation|card|policies to limit output. Configured worker counts and Redis ports are distinct from runtime selection; this query does not import user operators, probe ports or create scan resources.

Named templates print their exact YAML to stdout; --json adds artifact kind and prerequisites. Some examples depend on the files copied by project create; Sampling fragments provide a full base-template query and instructions to replace its Sampling block, then validate the composed card. A full card needs Scan, EnvReqs and at least one Calculators or Operas section. Agents should edit the user's file within the requested scope, use each validation issue's manual_argv to look up the correction, then validate again. Preserve scientific choices and existing files. Repeated explicit YAML keys in tasks and selected default files are rejected with the field and both source locations; legitimate YAML merge overrides remain allowed. Unrecognized manual subtopics fail with parent navigation instead of returning a misleading successful parent lookup.

validate does not execute calculators or start Redis/Workers. A successful report does not prove runtime or physics correctness. check and run execute configured code and create runtime resources; bare Jarvis TASK.yaml also runs the task. Existing user authorization remains valid, but YAML assistance alone does not authorize execution, dependency installation or process termination. CLI onboarding and dependency imports may initialize user caches even during queries. JSON command output uses stdout; advisory diagnostics use stderr.

For a missing capability, produce an email draft with:

Jarvis agent request --title "Required feature" --description "Goal and constraints" --json

The recipient is Jarvis.HEP.AI@mail.grokbot.com, with the exact marker Resquest@Jarvis-HEP (the spelling follows Jarvis-PLOT). Include installed version, OS/Python/installation method, minimal YAML, commands and diagnostics, actual and expected behavior, and scientific/compatibility constraints. Report directly in a maintainer conversation; otherwise provide the draft for the user to submit. The CLI does not send mail or write the draft to a file. Use --yaml, --commands, --diagnostics, --actual, --expected, --constraints and --installation-method to fill the matching draft sections. These inputs are literal text; paths are not read and commands are not executed. missing_fields identifies remaining blanks; draft_ready only means all required text sections are filled, not that their contents were verified or mail was sent.

The implementation audit and validation record are in the CLI agent integration report.

Outputs

For a scan named my_scan, the project folder typically contains:

outputs/my_scan/
├── DATABASE/
│   ├── samples.hdf5              # results for every evaluated point
│   └── samples.csv               # same results as CSV, written when the scan finishes
├── SAMPLE/
│   ├── 000001/<uuid>/             # files produced for each point (if kept)
│   └── 000001.tar.gz              # the same files, compressed in groups (if enabled)
├── run_summary.json              # counts, timing and speed of the run
├── run_summary.csv
└── run_summary.txt

logs/my_scan/
├── core.log                      # main process
├── sampler.log                   # choosing parameter points
├── archiver.log                  # writing results to disk
└── worker-00.log ...             # one log per parallel worker

If a scan was interrupted, or you want to regenerate samples.csv, run Jarvis convert TASK.yaml.

Progress for resuming is saved under checkpoints/<scan>/<sampler>/state.pkl. Press Ctrl+C to stop a scan cleanly; Jarvis stops all of its worker processes and any Redis server it started itself. Continue later with Jarvis run TASK.yaml --resume.

Saving disk space. By default Jarvis keeps a folder of files for every point under SAMPLE/. For large scans where you only need the numbers in DATABASE/, set:

EnvReqs:
  V2:
    store_samples: false

Calculators still run normally, but their files are written to a temporary folder and deleted after each point. Results, error status for failed points, resume data and the main logs are kept; per-point files and per-point logs are not. See SAMPLE storage for details.

Documentation

• Online documentation
• Jarvis man — built-in manual for every task-card field, with examples
• INSTALL.md — installation, Redis setup, full command list, and sharing projects
• SAMPLE storage — keeping or discarding per-point files
• Project template — what Jarvis project create puts in a new project

Coming from Jarvis-HEP 1.x? Version 2 replaces it and 1.x is no longer developed. The command is still called Jarvis; check older task cards with Jarvis validate before running them.

Citation

If you use Jarvis-HEP in your research, please cite:

@article{Guo:2026jarvishep,
  title         = {Jarvis-HEP: A lightweight Python framework for workflow
                   composition and parameter scans in high-energy physics},
  author        = {Guo, Erdong and Jackson, Paul and Yang, Jin-Min and Zhu, Pengxuan},
  year          = {2026},
  eprint        = {2604.25557},
  archivePrefix = {arXiv}
}

Please also cite the sampling algorithms you use. Jarvis --refs prints the references for Jarvis-HEP and for each built-in sampler.

Development

The fixes and release gates for 2.0.15 are tracked in the development ledger.

release_dir="$(mktemp -d)"
./jhrel --output "$release_dir"
python3 -m pip install --force-reinstall "$release_dir/"*.whl
python3 -m pip install "pytest>=7.0" "fakeredis>=2.0" "colorlog>=6.0"
python3 -m pytest -q

The long-running AdaptiveBridson integration tests are skipped by default; run them explicitly with python3 -m pytest -q tests/test_adaptive_bridson.py when changing that sampler.

The long-running feedback-loop coverage in tests/test_ensemble_samplers.py is also skipped by default; run it explicitly for EnsembleMCMC/DEMCMC/PT changes:

python3 -m pytest -q tests/test_ensemble_samplers.py

The slow distributed acceptance gates in tests/test_distributed_acceptance.py are also skipped by default; run them explicitly when validating Worker/ Archiver performance:

python3 -m pytest -q tests/test_distributed_acceptance.py

tests/test_distributed_resume.py, tests/test_mcmc_sampler.py, and tests/test_worker_pool.py are also excluded from the default suite. They exercise interruption/checkpoint resume, multi-process sampler execution, and Worker calculator-pool concurrency; run the relevant file explicitly after changing those paths:

python3 -m pytest -q tests/test_distributed_resume.py
python3 -m pytest -q tests/test_mcmc_sampler.py
python3 -m pytest -q tests/test_worker_pool.py

tests/test_variable_distributions.py and tests/test_worker_failure.py are also excluded from the default suite while their V1-card/schema fixture and calculator-pool SIGKILL fixture are reconciled. Run either explicitly when working on those paths:

python3 -m pytest -q tests/test_variable_distributions.py
python3 -m pytest -q tests/test_worker_failure.py

License

Jarvis-HEP is released under the MIT License. Bundled third-party code keeps its own license (for example Dynesty).