Highstorm

Highstorm stress-tests Kafka and Redpanda deployments by replaying historical records at a controlled speed. A Rust core does all per-record work; a thin CLI and a thin Python package drive the same engine, so a run configured the same way behaves the same way from a shell or a script.

• Sources: a finite snapshot of a Kafka/Redpanda topic, or local Parquet files in a message-envelope layout or a structured (one column per field) archive layout.
• Destination: one Kafka/Redpanda topic, produced with acks=all and idempotence by default, with partition preservation or key partitioning.
• Timing: records are read ahead as fast as bounded buffers allow and sent on the historical schedule scaled by --speed (default 1.0), or as fast as possible with --unpaced. Every deadline derives from one anchor, so overhead never accumulates into drift, and lateness is measured rather than hidden.
• Range: [start, end) in RFC 3339 with an explicit timezone.
• Configuration: a kubectl-style TOML file with named clusters, registries, sources, destinations, and contexts, each with its own credentials; environment overrides; or a fully environment-based setup. See docs/configuration.md.
• Reports: every run is a stress-test result with delivery outcomes, scheduling lateness, acknowledgement latency, buffer and queue pressure, starvation, and the redacted effective configuration. See docs/reports.md.

Quick start

Install the Python package and its highstorm command with Python 3.9 or newer:

python -m pip install highstorm

Wheels contain the Rust extension. If a wheel is unavailable for your platform, installation builds from source and requires Rust 1.85+, a C/C++ compiler, CMake, pkg-config, and the development libraries for OpenSSL, SASL, curl, and zlib. For a checkout, Pixi provides these dependencies:

pixi run develop

highstorm config init                                  # writes highstorm.toml
highstorm config set source kholinar
highstorm config set destination urithiru
highstorm config view --effective

highstorm inspect --source kholinar                    # metadata, timestamp samples, mapping issues
highstorm replay --dry-run                             # connectivity and compatibility, no records
highstorm replay --dry-run --scan                      # also read the whole source through the engine

highstorm replay \
  --start 2026-04-09T09:00:00Z \
  --end 2026-04-09T10:00:00Z \
  --speed 10

highstorm replay --unpaced --report run.json

Ctrl-C stops reading, drains outstanding deliveries within the delivery timeout, and still prints the report with status = "cancelled". Progress lines go to standard error every --progress-seconds (--quiet disables them); the JSON report goes to standard output and to --report <file>.

The same workflow from Python (see docs/python.md):

import highstorm

configuration = highstorm.Config.load("highstorm.toml")
report = highstorm.replay(
    configuration,
    source="kholinar",
    destination="urithiru",
    start="2026-04-09T09:00:00Z",
    end="2026-04-09T10:00:00Z",
    speed=10.0,
)
print(report.status, report.records_sent, report.scheduling_lateness.p99_seconds)

highstorm.start_replay returns a handle with cancel(), statistics(), and wait(timeout); highstorm.inspect and highstorm.dry_run mirror the CLI.

Replay contract

Exactly-once replay across clusters or restarts is not promised: a restarted run can produce duplicates even with idempotence enabled.

Development

Pixi manages the toolchain and tasks:

Python releases

Hatchling is the Python build backend. Its Maturin hook compiles the Rust extension and preserves its native wheel tags and bundled libraries. The Python distribution version is read dynamically from highstorm/__version__.py and exposed as highstorm.__version__. highstorm.core_version() and the CLI's --version report the Rust workspace version in Cargo.toml; update that separately when releasing core changes.

With just and Pixi installed:

pixi run -e build hatch version patch  # or an explicit version, e.g. 0.2.0
just build                            # builds dist/*.tar.gz and dist/*.whl, then checks metadata

The source archive includes the Rust workspace and lockfile, so it can rebuild the extension. Each wheel targets the build machine's OS and architecture; build on each platform you intend to provide wheels for. Linux builds must produce a PyPI-compatible manylinux or musllinux wheel; the build rejects plain linux wheel tags.

To publish, configure Twine with your PyPI API token (for example, set TWINE_USERNAME=__token__ and TWINE_PASSWORD in your shell), then run:

just release

This rebuilds dist/, runs twine check --strict, and uploads both artifacts to PyPI. For TestPyPI, set TWINE_REPOSITORY=testpypi and use a TestPyPI token.

Repository layout

docker build -t highstorm . produces a standalone image whose entrypoint is the executable. Benchmark results and how to reproduce them are in docs/benchmarks.md.

Layout:

crates/highstorm-core/     configuration, sources, Avro, engine, delivery, runner, inspection
crates/highstorm-cli/      argument parsing and terminal output
crates/highstorm-python/   PyO3 bindings
highstorm/                 Python package: typed API, exceptions, reports, console entry point
tests/fixtures/            golden DBN, Avro, and Parquet fixtures
examples/                  configuration files and a Python script
docs/                      configuration, Python, reports, benchmarks

Out of scope for this release: distributed workers, live tailing, resumable checkpoints, direct S3 reads, and arbitrary Python record transforms.