incontext 0.0.2

incontext is a Hermes Agent plugin that gives each LLM request exactly the output budget still available below Hermes' context-compression boundary. It prevents a fixed, oversized max_tokens value from consuming the input space where Hermes must still be able to compress the conversation.

The plugin reads the effective compression window from the installed Hermes ContextCompressor, asks the selected inference backend to tokenize the exact provider-visible prompt, and applies:

max_tokens = min(
    caller_max_tokens,
    compression_window - prompt_tokens,
)

When the caller does not provide an output cap, incontext uses the whole free remainder of the compression window. A smaller positive caller cap is preserved, which keeps bounded auxiliary operations such as Hermes context summarization from becoming unexpectedly long.

Hermes auxiliary calls (including context-compression summaries, generated titles, and vision helpers) do not pass through the public llm_request middleware. Hermes also omits max_tokens for most custom providers. The plugin therefore wraps Hermes' auxiliary request builder at registration time: those requests receive the same exact budget, and an explicit smaller caller cap is not lost. If exact counting cannot produce a safe result, the wrapper leaves Hermes' original request unchanged.

The expression is applied only while its result is positive. A zero or negative result is not converted to the invalid sentinel max_tokens=1. Instead, incontext installs the same exact counter into Hermes' pre-API compression estimator, so Hermes compacts the conversation before it creates the provider request. If the tokenizer is temporarily unavailable, this bridge falls back to Hermes' native rough estimator and the request middleware itself remains fail-open.

This addresses the same output-budget arithmetic discussed in NousResearch/hermes-agent#38652.

Installation

Once release 0.0.2 or newer is available on PyPI, install and enable the package using the same plugin name, incontext:

python -m pip install 'incontext>=0.0.2'
hermes plugins enable incontext

Until that safety-complete release is published, install the current develop branch:

python -m pip install 'git+https://github.com/pomponchik/incontext.git@develop'
hermes plugins enable incontext

Restart the long-running Hermes gateway after installing or upgrading the Python package. Hermes discovers it through the official hermes_agent.plugins entry-point group; no source file has to be copied into $HERMES_HOME/plugins.

Configuration

The bundled vllm backend is selected by default. Its INCONTEXT_TOKENIZER_URL setting is required and must point to the /tokenize endpoint of the same vLLM model Hermes uses:

export INCONTEXT_BACKEND='vllm'
export INCONTEXT_TOKENIZER_URL='https://inference.example/tokenize'
export INCONTEXT_TOKENIZER_TIMEOUT_SECONDS='30'
export INCONTEXT_TOKENIZER_USER_AGENT='incontext/0.0.2'
export INCONTEXT_FALLBACK_MARGIN_TOKENS='1024'

Environment variables are loaded through typed skelet.Storage fields backed by ordered skelet.EnvSource instances. Primary INCONTEXT_* names take precedence over the supported legacy aliases. Text normalization and blank value rejection are implemented by the fields' native conversion and validation rules.

Hermes' model.default, model.context_length, and compression.threshold remain the source of truth. The plugin constructs Hermes' installed ContextCompressor and uses its resolved threshold_tokens; it does not copy version-sensitive threshold arithmetic.

The optional variables are:

The former HERMES_VLLM_TOKENIZER_* and HERMES_DYNAMIC_BUDGET_FALLBACK_MARGIN_TOKENS names are accepted as migration aliases. New deployments should use the INCONTEXT_* names.

Replacing the inference backend

The budgeting core depends only on the abstract incontext.Backend contract. It has no import or construction dependency on vLLM. A backend supplies its safe diagnostic source, exact count(...), cache invalidation, and an optional output-field normalization hook.

Backend implementations are named pristan plugins in the incontext.backends entry-point group. The generic skelet environment has a typed backend field whose default is vllm. At runtime incontext performs the single named resolution directly:

backend = backends[environment.backend].one()

The incontext distribution itself publishes the vllm entry point. Loading that entry point imports incontext.vllm_provider, whose only responsibility is to construct VllmBackend. All /tokenize payload rules, vLLM response fields, context-length validation, transport settings, and caching live inside that class rather than in the budgeting core.

A third-party distribution can provide another backend without changing incontext. Its implementation subclasses the stable abstract contract and its plugin module registers a provider under a new name:

# acme_backend/plugin.py
from __future__ import annotations

from typing import Any, Dict

from incontext import Backend, backends


class AcmeBackend(Backend):
    @property
    def source(self) -> str:
        return "acme-tokenizer"

    def count(
        self,
        request: Dict[str, Any],
        *,
        context_length: int,
    ) -> int:
        ...

    def clear_cache(self) -> None:
        ...


@backends.plugin("acme")
def provide_acme_backend() -> Backend:
    return AcmeBackend()

The third-party package makes that module discoverable in pyproject.toml:

[project.entry-points."incontext.backends"]
acme = "acme_backend.plugin"

After installing the package, select it through the same typed configuration field and restart the Hermes process:

export INCONTEXT_BACKEND='acme'

Only the selected provider is instantiated. An unknown name fails .one(); the unique slot rejects duplicate providers under the same name while loading entry points. Startup therefore fails instead of choosing a backend implicitly. Each backend owns and validates its backend-specific configuration; the generic settings object contains only the compression-window and fallback-budget policy.

Safety properties

• With the bundled backend, vLLM applies its real chat template to messages, tools, and chat_template_kwargs; local tokenizer approximations are not used.
• The returned max_model_len must equal Hermes' configured context length.
• max_tokens, max_completion_tokens, and max_output_tokens are reduced to the smallest positive caller cap while preserving the corresponding provider-selected field name; a full compression window is handed to preflight compression.
• The incoming request is copied and never mutated.
• Exact counts use a bounded, thread-safe cache.
• The exact counter is also used by Hermes' preflight compressor, eliminating the former gap where compression used a rough count but budgeting used an exact one.
• If /tokenize fails, Hermes' own rough estimator is used with an additional safety margin. If both counters fail, the middleware leaves the request unchanged instead of taking Hermes down.
• Logs contain counts and exception types, never prompts, credentials, or raw provider errors.

The tokenizer endpoint sees the prompt content by design. Run it on a trusted network path and use the same access controls as the inference endpoint.