research-repo-tools

Shared development and maintenance tooling for Rust research repositories using Python scripting and Jupyter notebooks. Each capability has one implementation and one set of contracts, with tests grouped by capability. Shared toolchain setup supports changelog handling and the maintenance core; further workflows require a clear shared contract.

This package consolidates shared utilities from la-stack, delaunay, markov-chain-monte-carlo, and causal-triangulations. Its goal is to simplify that stack around a correct, performant, orthogonal, and simple toolchain, keeping consuming repositories focused on Rust.

uv is required. Install it using Astral's instructions and make it available on PATH. This package uses uv to obtain Python 3.14+ and Python tooling. Git and platform build prerequisites are listed in the toolchain guide. See release status and publishing and the supported CLI and Python interfaces.

Install with uv

Users install pinned releases directly from PyPI in their own projects. Consumers never need to clone research-repo-tools. Cloning this repository is for developing or fixing the package.

Once the desired version is published, the consuming project's maintainer adds it to a tooling dependency group, replacing X.Y.Z with that version:

uv add --group tooling --no-sync "research-repo-tools==X.Y.Z"

Include tooling in dev, retaining the project's other development dependencies:

[dependency-groups]
tooling = ["research-repo-tools==X.Y.Z"]
dev = [{include-group = "tooling"}]

Follow the toolchain guide to declare uv, Python, and Rust tool versions, merge the consumer justfile template, and commit the manifest and refreshed lockfile. The uv add --no-sync command records the dependency without installing or synchronizing it. The setup invocation below installs research-repo-tools before running setup, with no installation hooks or generated scripts.

Just recipes require sh on PATH. On Windows, expose Git for Windows' bin directory as described in the toolchain guide; setup checks this prerequisite before installing tools.

In a configured consumer checkout, run this once on Linux, macOS, or Windows:

uv run --locked --managed-python --only-group tooling research-repo-tools setup

Setup installs the declared tools, installs a pinned user-level Just through uv, configures PATH, and synchronizes the locked Python environment with managed Cargo available for native builds. Open a new terminal if PATH changed. From then on, use just help and just <recipe> without environment activation. Recipes select the locked project environment internally. Package contributors use the setup command described in CONTRIBUTING.md.

Common workflows

Just recipes

For routine workflows, merge the consumer justfile template into the consuming repository's justfile. These thin recipes invoke its locked package version. After initialization, use just directly. Run just help, just help-workflows, or bare just to list commands and their arguments in lexicographic order.

To preview a prospective release, run just changelog-preview --tag v1.2.3 --date YYYY-MM-DD. Follow the toolchain guide for declarations, first-time setup, and strictly read-only tool checks.

CodeRabbit review

Install and authenticate the CodeRabbit CLI explicitly and ensure coderabbit is on PATH. The package does not install it, authenticate, enable usage credits, or retry reviews automatically. These recipes are opt-in and remain outside just check and just ci. Agents must have an explicit maintainer request before invoking a live review.

just review
just review main
just review-uncommitted

just review includes committed branch changes plus staged, unstaged, and non-ignored untracked files. Before invoking CodeRabbit, it compares the local origin/main commit with the live origin main branch. Missing or stale refs stop with fetch guidance; failed or malformed remote lookups also stop review. The wrapper never fetches or changes Git state. Verification applies at invocation, not throughout a long review. An explicit local base such as main is checked locally without a remote query. Uncommitted review skips the base check entirely.

The configured consumer root must contain AGENTS.md and exactly one of .coderabbit.yaml or .coderabbit.yml; missing files, directories in place of files, or both configuration names are errors. Both files are passed as additional instructions. Output uses CodeRabbit's structured --agent mode and streams directly to the terminal without the usual five-minute subprocess timeout. Nonzero exit statuses propagate, and interruption returns 130. Service or authentication errors are unavailable reviews, not clean results. Verify findings and suggested changes against current code before acting on them.

Workflow examples

just changelog-archive
just changelog-preview --tag v1.2.3 --date 2026-09-07
just help
just release-check
just release-notes v1.2.3
just tools-check
just update

just update upgrades uv first. Standalone uv installations use uv self update; Homebrew installations use brew upgrade uv. Other installation owners must update uv themselves. The updater records the resulting stable version in pyproject.toml. This changes the shared user installation; other projects with different exact uv pins must be reconciled before their commands will run. Review the generated changes before committing. just setup installs declared versions without upgrading them.

The underlying CLI remains available for integrations and custom recipes; see supported interfaces.

Dependency and tool updates

just update runs update-tools before update-dependencies. Every recipe stops at its first failed command; completed steps remain applied for review and retry. The aggregate is not a transaction across package managers. Dependency-only recipes use the installed, declared toolchain and leave uv and Cargo tool pins unchanged. Run setup first when adopting or changing tool declarations. Update launchers synchronize only tooling and retain installed consumer dependencies, so they can start before a native consumer is ready to rebuild.

just update-python-dependencies advances exact direct dev pins, upgrades the entire lock resolution within the resulting manifest constraints, and explicitly synchronizes dev, even with default-groups = []. The final sync runs with checked managed tools so native Python builds can find Rust. Included groups, including the pinned shared package in tooling, retain their declared constraints. The older update-python-deps spelling is an alias for this complete workflow.

For Rust projects, pin cargo-edit to an exact version in the managed Cargo tool table (for example, cargo-edit = "0.13.13") and keep the compiler pinned in rust-toolchain.toml, then run just setup to install and verify the declared tool. For both cargo upgrade and direct cargo-upgrade invocations, toolchain run rejects missing or non-exact cargo-edit declarations even when an unmanaged copy is on PATH. The default Cargo recipe upgrades requirements with incompatible releases allowed, then updates Cargo.lock. It skips Cargo when the root has no Cargo.toml, keeping Python-only consumers supported. Additional resolution roots and coupled dependency exclusions belong in the consumer's update-cargo-dependencies recipe. When merging the template, preserve those decisions; the aggregate calls that recipe by name. For example:

update-cargo-dependencies:
    uv run --locked --only-group tooling --inexact research-repo-tools toolchain run -- \
        cargo upgrade --incompatible allow --exclude coupled-a --exclude coupled-b
    uv run --locked --only-group tooling --inexact research-repo-tools toolchain run -- \
        cargo upgrade --manifest-path "fixtures/extra root/Cargo.toml" --incompatible allow
    uv run --locked --only-group tooling --inexact research-repo-tools toolchain run -- cargo update
    uv run --locked --only-group tooling --inexact research-repo-tools toolchain run -- cargo update --manifest-path "fixtures/extra root/Cargo.toml"

Exclusions apply to requirement upgrades; lock resolution still follows all declared constraints. Keep the required order of these commands and retain consumer checks for coupled dependencies and additional manifests. Review both manifests and lockfiles after updating. See update adoption for superseded policies and consumer integration requirements.

changelog tag TAG creates a local annotated tag when explicitly invoked. --dry-run previews it. Generation needs git-cliff; tagging needs Git. Optional formatting needs rumdl, fixture validation needs Semgrep, and dependency updates need uv or Cargo. These executables are required only by the commands that invoke them. The CLI does not publish packages or hosted releases; this repository's tagged-release workflow publishes the tooling package to PyPI.

just changelog generates and normalizes history, then moves completed minor series to docs/archives/changelog/. The root file keeps Unreleased, the newest minor series, and archive links. The same changelog recipes are available here and in the consumer justfile template.

External-tool installation is explicit through setup, toolchain sync, or toolchain upgrade; managed execution selects the checked versions. Scientific benchmarks, broader evidence schemas, plotting, and deployment workflows remain outside this package. Scientific algorithms, case inventories, repository rules, and custom release commands stay in their consuming repositories.

Notebooks

For notebook work, add research-repo-tools[notebooks]==X.Y.Z at the same version as the tooling pin to a notebook dependency group. Keep analysis libraries in that group, and declare Ruff 0.16.8 or newer and ty 0.0.82 or newer in dev for linting. Refresh the lockfile, then use just notebook-sync. It registers a kernel in the project environment. Maintenance-only users need no Jupyter packages.

just notebook-sync
just notebook-lint notebooks/analysis.ipynb
just notebook-execute notebooks/analysis.ipynb

Execution leaves source files untouched and writes executed notebooks and JSON reports under target/notebooks, preserving root-relative paths. Consumers own fast/slow selections, input preparation, scientific assertions, and figure destinations. See the notebook contract for output policy, failure reports, and environment configuration. Linting uses the locked project's Ruff and ty with the consumer's configuration and preserves cell IDs in diagnostics. notebook-check provides structure and output checks alone.

Performance evidence

These APIs and recipes require a published release newer than 0.1.2 containing them. Merge the packaged performance recipe into the consumer justfile, then compare existing Criterion trees without executing benchmarks:

just performance compare target/baseline/criterion target/current/criterion --output target/comparison.json

Both samples default to new, the statistic to median, and the unit to ns. Use --baseline-sample NAME, --current-sample NAME, --statistic mean, or --unit when the measurement harness requires them. The JSON retains both full samples, including added and missing benchmarks. The CLI rejects comparisons with no common benchmarks. It does not infer statistical significance. Place --output outside both Criterion input roots, including path aliases.

Use Python to attach provenance captured by the consumer at measurement time:

from pathlib import Path

from research_repo_tools.criterion import COMPARISON_SCHEMA, parse_comparison, render_comparison
from research_repo_tools.evidence import Evidence, Provenance, compare_provenance, publish_evidence

def retain_comparison(payload: bytes, baseline: Provenance, current: Provenance) -> None:
    comparison = parse_comparison(payload)
    compatibility = compare_provenance(
        baseline, current, fields=("harness_sha256", "context.cpu", "context.rustc"),
    )
    if not compatibility.compatible:
        raise ValueError(f"Measurement context differs or is unknown: {compatibility.differences}")
    retained = Evidence(payload, COMPARISON_SCHEMA, (("baseline", baseline), ("current", current)))
    publish_evidence(
        retained, Path("evidence/comparison.json"), Path("evidence/manifest.json"),
        reports={Path("PERFORMANCE.md"): render_comparison(comparison).encode("utf-8")},
        immutable=(Path("evidence/comparison.json"), Path("evidence/manifest.json")),
    )

Choose compatibility fields and scientific acceptance rules in the consumer. Provenance records the full source commit, optional exact source and harness digests, and explicit context. Unknown values fail a requested compatibility check. Render or verify retained evidence independently of measurement:

just performance verify evidence/comparison.json evidence/manifest.json
just performance render evidence/comparison.json evidence/manifest.json --output PERFORMANCE.md

verify checks envelope structure and the exact payload digest; consumers still validate their own payload schemas. render additionally parses the shared comparison schema. Domain reports can keep their existing renderer and use the shared comparison results and publication transaction.

For published assets, use just performance fetch HTTPS_URL ASSET --sha256 DIGEST with an independently recorded digest, then just performance extract ASSET ABSENT_DIRECTORY --sha256 DIGEST. The extraction parent must exist. See the performance API for limits and retained-evidence migration before replacing consumer helpers or changing artifact formats.

Document publication

Publish a selected timing table and optional SVG into an existing marked section:

just performance publish publication.toml --preview
just performance publish publication.toml
just performance publish publication.toml --check

Preview prints validated candidate diffs without writing. Check exits 1 for stale or invalid outputs and 0 when current. Default mode updates the document and figure in one rollback-capable transaction. All modes verify the retained payload hash, selected measurements, configured provenance, and release/report references. No measurements or network retrieval run. This capability requires a published release newer than 0.1.2 containing it.

Place exactly one ordered <!-- PERFORMANCE:BEGIN --> and <!-- PERFORMANCE:END --> pair in the document. Configure shared comparison evidence and explicit benchmark selection in publication.toml:

schema = 1
document = "README.md"
payload = "evidence/comparison.json"
manifest = "evidence/manifest.json"
begin = "<!-- PERFORMANCE:BEGIN -->"
end = "<!-- PERFORMANCE:END -->"
unit = "ns"
svg = "figures/timings.svg" # Omit for table-only publication.
rows = [{benchmark = "step/2", label = "Two-dimensional step"}]
links = [
    {label = "Report and measurement context", path = "PERFORMANCE.md"},
    {label = "Retained measurements", path = "evidence/comparison.json"},
    {label = "Provenance", path = "evidence/manifest.json"},
]

[provenance.baseline]
revision = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" # Replace with expected commit.
release = "v1.0.0"

[provenance.current]
revision = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" # Replace with expected commit.
release = "v1.1.0"

[[references]]
path = "Cargo.toml"
pattern = '^version = "(?P<value>[^"]+)"\r?$'
source = "version"

[[references]]
path = "PERFORMANCE.md"
pattern = '^Baseline: (?P<value>[^\r\n]+)'
source = "previous-tag"

[[references]]
path = "PERFORMANCE.md"
pattern = '^Current: (?P<value>[^\r\n]+)'
source = "tag"

Use the consumer's actual package/report paths and precise capture patterns. This example expects version = "1.1.0" and report lines Baseline: v1.0.0 and Current: v1.1.0. Record each source's release in its evidence Provenance.context, for example (("release", "v1.1.0"),). Maintain expected revisions independently of the files being checked. Add source-sha256, harness-sha256, or a context table to a provenance table when those identities are required. Unknown or mismatched values fail.

Rows follow configuration order; each must have both baseline and current data. The unit must match the recorded unit. Optional baseline-label and current-label change displayed headings without changing release expectations. The dependency-free SVG shows point ratios; scientific commentary stays in the consumer. Bytes outside the marker interior, including historical links and line endings, are preserved.

Links default to document-relative paths. To publish tagged GitHub links, set top-level repository = "owner/repository". The current release tag must exist locally and contain the exact retained, referenced, linked, and generated figure bytes. To assert that a measured source itself is a tagged commit, also set verify-tag = true in that source's provenance table. These are read-only local Git checks; fetching release assets remains a separate explicit operation.

Historical CSV/JSON formats and custom renderers use the Python planner with their consumer schema adapter. See the publication API and migration guide, including MCMC's adapter path.

Clippy SARIF helpers

Declare the exact converter versions in pyproject.toml. Keep the consumer's Rust compiler pinned in rust-toolchain.toml, with the clippy component:

[tool.research-repo-tools.toolchain.cargo]
clippy-sarif = "0.8.0"
sarif-fmt = "0.8.0"

Use just setup to install them and just tools-check to verify them. Keep the Cargo invocation and SARIF upload workflow in the consumer. A Bash recipe can preserve failure from every pipeline stage with pipefail:

clippy-sarif:
    #!/usr/bin/env bash
    set -euo pipefail
    tool() { uv run --locked --no-sync --no-python-downloads research-repo-tools toolchain run -- "$@"; }
    tool cargo clippy --message-format=json | tool clippy-sarif | tee results.sarif | tool sarif-fmt

just update-cargo-tools upgrades only declared tools. Both helpers use the shared exact, locked Cargo installer and checked execution. See the platform contract.

Calling from Python

A consumer that needs a Python wrapper can invoke the same CLI contract without a subprocess. For a script in the consumer's scripts/ directory:

from pathlib import Path

from research_repo_tools import __version__
from research_repo_tools.cli import main

root = Path(__file__).resolve().parents[1]
print(f"Using research-repo-tools {__version__}")
raise SystemExit(main(["--root", str(root), "release", "check"]))

The process and file APIs below require a release newer than 0.1.2 that includes them. Pin the published version before removing local consumer helpers. They leave command policy and scientific decisions in the consumer:

import subprocess
import sys
from pathlib import Path

from research_repo_tools.files import replace_many
from research_repo_tools.process import format_exception_diagnostics, run_command, run_git_bytes

root = Path(__file__).resolve().parents[1]
try:
    result = run_command(sys.executable, ["--version"], cwd=root, timeout=30)
    payload = (root / "results.json").read_bytes()
    # Git receives the original bytes and applies its own configured filters.
    digest = run_git_bytes(
        ["--no-pager", "hash-object", "--path=results.json", "--stdin"],
        cwd=root, input=payload,
    ).stdout.strip()
    replace_many({
        root / "reports/results.json": payload,
        root / "reports/results.hash": digest + b"\n",
    })
except (OSError, subprocess.SubprocessError, ExceptionGroup) as error:
    raise SystemExit(format_exception_diagnostics(error)) from error

Text execution uses explicit UTF-8 by default; byte execution preserves LF, CRLF, and binary data. Publication stages every file first and rolls back caught replacement failures. See supported interfaces for lookup behavior, typed results/errors, recovery backups, concurrency limits, and API stability.

Preparing a structured release

The release API requires a published version newer than 0.1.2 containing it. It replaces wrapper-owned discovery, staging, CLI-output parsing, and rollback:

from pathlib import Path

from research_repo_tools.releases import apply_release, plan_release

plan = plan_release(
    Path.cwd(), "v1.2.4", previous_tag="v1.2.3", release_date="2026-09-20",
)
for edit in plan.edits:
    print(edit.path)  # Inspect edit.before and edit.after for the exact byte diff.
# Omit apply_release for a fully validated preview.
result = apply_release(plan)

Declare required files, fixed metadata, and selected active references in release configuration. Add a small Python adapter when validation needs consumer-owned evidence rules. See the worked MCMC-style migration and typed API contract. Historical evidence stays excluded from updates.

Templates and optional settings

Standard changelog archiving and release commands need no configuration file. For generated GitHub links, specify the owner and repository in existing project metadata:

[tool.research-repo-tools.changelog]
owner = "example"
repository = "consumer"
# formatter = "rumdl.toml"  # optional external formatting

[tool.research-repo-tools.deps.tools]
just_version = "just"
rumdl_version = "rumdl"
uv_version = "uv"

[tool.research-repo-tools.semgrep]
config = "semgrep.yaml"
fixtures = "tests/semgrep"

--config PATH reads unprefixed tables from a separate TOML file when needed. Paths are relative to the consumer root, which defaults to the configuration directory; --root PATH overrides that root. Explicit executable paths such as deps.uv = "./bin/uv" use the same root; bare names use PATH. The selected uv executable is used for dependency resolution, Python pin updates, and uv version checks. Unknown settings fail. There are no repository profiles.

deps.tools values name packages installed separately with Cargo, as listed by cargo install --list. The special uv entry reads the selected uv executable. The Cargo package is named just; the Python dependency that supplies the bundled executable is named rust-just.

uv run --locked research-repo-tools templates CHANGELOG.md
uv run --locked research-repo-tools templates cliff.toml --owner example --repository consumer
uv run --locked research-repo-tools templates justfile
uv run --locked research-repo-tools templates research-repo-tools.toml

Templates print to stdout. --output PATH creates a file and refuses to replace existing content. Generation uses the packaged git-cliff template by default, so consumers need not maintain a copy.

Cargo pins accept canonical prereleases and build metadata, such as 1.2.3-rc.1+build.5; uv pins require stable X.Y.Z. All pin changes are validated before replacement. Comments, line endings, permissions, and symlink targets are preserved. deps update-python updates exact development requirements while retaining ranges, markers, extras, and other unmanaged constraints. Before uv changes the manifest and lockfile, the updater saves both originals. A failed update restores them; if restoration also fails, the command reports the retained backup paths for recovery.

See changelog behavior, release behavior, and scope and adoption for detailed contracts and migration guidance.

Contributing

To develop or fix this package, see CONTRIBUTING.md for environment setup, coding conventions, maintainer recipes, testing, and release preparation.

License

BSD-3-Clause. See LICENSE.