codetool-shell

Python interface library for compressing plain shell-output text before it is returned to an LLM/tool consumer.

from codetool_shell import compress_text

compressed = compress_text("one  \n\n\n two\t\n")
print(compressed)

The core pipeline is intentionally conservative:

• normalize CRLF/CR line endings to LF
• trim trailing whitespace on each line
• run dedicated output filters only when they make the text smaller
• collapse repeated blank lines to max_blank_lines (default: 1)

Initial dedicated filters cover:

• git output: short and long status, tabbed name-status, diff stats, and patch hunks/binary-patch payloads with verbose unchanged or encoded context summarized
• test-runner output: conservative ANSI-aware pytest (including quiet progress/summary output), unittest-like, vitest-like, Playwright, and cargo-test summaries that preserve failure identifiers, assertion/error text, source locations, artifacts/traces, and final totals
• CI/job logs: conservative GitHub Actions / gh run-style summaries that collapse repeated timestamped step noise while preserving step names, annotations, warnings/errors, file locations, artifact/URL/path references, and job/workflow conclusions
• standalone Python tracebacks: conservative traceback summaries that preserve exception type/message, file stack frames, chained-exception separators, source/caret context, and small command noise while collapsing repeated or internal stack frames
• structured tables: conservative psql, SQLite/column, MySQL boxed, and long Markdown pipe table summaries that preserve headers, footers, first/last rows, and warning/error/URL/path/ID rows while counting omitted middle rows
• file/tree/listing output: conservative path-only, tree, and safe ls -R reducers that render repeated path prefixes as compact trees while preserving visible summary totals and leaving metadata-rich listings unchanged
• ripgrep output: direct path:line[:column]:text, heading-mode, context-mode path-line-context with -- separators, and ANSI-colored matches grouped into a compact path tree when repeated paths or prefixes would otherwise be duplicated, while diagnostic/table/prose-looking output is left alone
• generic application logs: conservative long timestamp/level-prefixed log summaries that collapse repeated INFO/DEBUG/TRACE patterns while preserving first/last context, warnings/errors/fatal/panic/exception lines, file paths/URLs, failure summaries, and unusual levels
• package-manager logs: conservative uv/pip, npm, bun, and cargo package install/update summaries that collapse progress/download/resolve/install noise while preserving warnings/errors, package summaries, versions, lock/frozen conflicts, audit/vulnerability summaries, hints, paths, and URLs
• build/compiler diagnostics: conservative cargo/rustc, tsc, ESLint, and mypy-like reducers that collapse build progress while preserving warnings/errors, codes, file spans, source/caret snippets, notes/help, rule IDs, and final summaries

Backends

codetool-shell is a Python library with a Rust compression engine. All compression/filter logic runs in the codetool-shell-rust helper binary; Python keeps the public API, binary discovery, and subprocess boundary.

• backend="auto" uses the Rust CLI backend and raises a clear RustBackendUnavailable/RustBackendError if the helper cannot be found or fails.
• backend="rust" is the same strict Rust path.
• backend="python" is no longer supported.

Rust helper discovery order:

1. CODETOOL_SHELL_RUST_BINARY
2. packaged wheel binaries under codetool_shell/bin/<platform>/
3. development binaries under rust/target/{release,debug}/
4. PATH

Build and stage the Rust CLI before creating a target wheel with:

python scripts/package_rust_binary.py --target-runtime linux-x86_64

Then build the Python distributions with:

uv build

During local development you can also rely on the development build directly:

cd rust
cargo build

Packaging and release

codetool-shell publishes target-specific wheels that include one matching Rust helper binary under codetool_shell/bin/<runtime>/. Supported runtime keys are:

• linux-x86_64
• linux-aarch64
• macos-x86_64
• macos-arm64
• windows-x86_64
• windows-arm64

To build a Linux x86_64 wheel locally (replace the runtime key for other targets):

uv run python scripts/package_rust_binary.py --target-runtime linux-x86_64
CODETOOL_SHELL_TARGET_RUNTIME=linux-x86_64 uv build --wheel

The release workflow builds an sdist, manylinux wheels, native macOS/Windows wheels, checks the distributions, and publishes to PyPI via trusted publishing for tagged releases (v*) or manual dispatch with publish=true.

Development

uv sync --dev
uv run pytest
uv run python benchmarks/benchmark_compress_text.py
uv run python scripts/benchmark_filters.py --iterations 1000