Skip to main content
ⓘ

Tests Hermes e2e Python versions PyPI version Checked with mypy Ruff

incontext logo

incontext is a Hermes Agent plugin that gives each LLM request exactly the output budget still available below Hermes' context-compression boundary. It prevents a fixed, oversized max_tokens value from consuming the input space where Hermes must still be able to compress the conversation.

The package supports CPython 3.8 through 3.15, including free-threaded Python 3.14. Unit tests, formatting, linting, and static type checks run across the complete version matrix in CI.

The plugin reads the effective compression window from the installed Hermes ContextCompressor, asks the selected inference backend to tokenize the exact provider-visible prompt, and applies:

max_tokens = max(1, compression_window - prompt_tokens)

This addresses the same output-budget arithmetic discussed in NousResearch/hermes-agent#38652.

Installation

Once a release is available on PyPI, install and enable the package using the same plugin name, incontext:

python -m pip install incontext
hermes plugins enable incontext

Until the first PyPI release, install the reviewed develop revision:

python -m pip install 'git+https://github.com/pomponchik/incontext.git@develop'
hermes plugins enable incontext

Restart the long-running Hermes gateway after installing or upgrading the Python package. Hermes discovers it through the official hermes_agent.plugins entry-point group; no source file has to be copied into $HERMES_HOME/plugins.

Configuration

The bundled vllm backend is selected by default. Its INCONTEXT_TOKENIZER_URL setting is required and must point to the /tokenize endpoint of the same vLLM model Hermes uses:

export INCONTEXT_BACKEND='vllm'
export INCONTEXT_TOKENIZER_URL='https://inference.example/tokenize'
export INCONTEXT_TOKENIZER_TIMEOUT_SECONDS='30'
export INCONTEXT_TOKENIZER_USER_AGENT='incontext/0.0.1'
export INCONTEXT_FALLBACK_MARGIN_TOKENS='1024'

Environment variables are loaded through typed skelet.Storage fields backed by ordered skelet.EnvSource instances. Primary INCONTEXT_* names take precedence over the supported legacy aliases. Text normalization and blank value rejection are implemented by the fields' native conversion and validation rules.

Hermes' model.default, model.context_length, and compression.threshold remain the source of truth. The plugin constructs Hermes' installed ContextCompressor and uses its resolved threshold_tokens; it does not copy version-sensitive threshold arithmetic.

The optional variables are:

Variable Default Meaning
INCONTEXT_BACKEND vllm Named pristan backend plugin
INCONTEXT_TOKENIZER_TIMEOUT_SECONDS 30 /tokenize request timeout
INCONTEXT_TOKENIZER_USER_AGENT incontext/0.0.1 HTTP user agent
INCONTEXT_FALLBACK_MARGIN_TOKENS 1024 Extra reserve only when exact tokenization fails
INCONTEXT_COMPRESSION_WINDOW_TOKENS unset Explicit emergency override for the resolved Hermes boundary

The former HERMES_VLLM_TOKENIZER_* and HERMES_DYNAMIC_BUDGET_FALLBACK_MARGIN_TOKENS names are accepted as migration aliases. New deployments should use the INCONTEXT_* names.

Replacing the inference backend

The budgeting core depends only on the abstract incontext.Backend contract. It has no import or construction dependency on vLLM. A backend supplies three operations: its safe diagnostic source, exact count(...), and clear_cache().

Backend implementations are named pristan plugins in the incontext.backends entry-point group. The generic skelet environment has a typed backend field whose default is vllm. At runtime incontext performs the single named resolution directly:

backend = backends[environment.backend].one()

The incontext distribution itself publishes the vllm entry point. Loading that entry point imports incontext.vllm_provider, whose only responsibility is to construct VllmBackend. All /tokenize payload rules, vLLM response fields, context-length validation, transport settings, and caching live inside that class rather than in the budgeting core.

A third-party distribution can provide another backend without changing incontext. Its implementation subclasses the stable abstract contract and its plugin module registers a provider under a new name:

# acme_backend/plugin.py
from __future__ import annotations

from typing import Any

from incontext import Backend, backends


class AcmeBackend(Backend):
    @property
    def source(self) -> str:
        return "acme-tokenizer"

    def count(
        self,
        request: dict[str, Any],
        *,
        context_length: int,
    ) -> int:
        ...

    def clear_cache(self) -> None:
        ...


@backends.plugin("acme")
def provide_acme_backend() -> Backend:
    return AcmeBackend()

The third-party package makes that module discoverable in pyproject.toml:

[project.entry-points."incontext.backends"]
acme = "acme_backend.plugin"

After installing the package, select it through the same typed configuration field and restart the Hermes process:

export INCONTEXT_BACKEND='acme'

Only the selected provider is instantiated. An unknown name fails .one(); the unique slot rejects duplicate providers under the same name while loading entry points. Startup therefore fails instead of choosing a backend implicitly. Each backend owns and validates its backend-specific configuration; the generic settings object contains only the compression-window and fallback-budget policy.

Safety properties

  • With the bundled backend, vLLM applies its real chat template to messages, tools, and chat_template_kwargs; local tokenizer approximations are not used.
  • The returned max_model_len must equal Hermes' configured context length.
  • max_tokens, max_completion_tokens, and max_output_tokens are normalized to one unambiguous max_tokens field.
  • The incoming request is copied and never mutated.
  • Exact counts use a bounded, thread-safe cache.
  • If /tokenize fails, Hermes' own rough estimator is used with an additional safety margin. If both counters fail, the middleware leaves the request unchanged instead of taking Hermes down.
  • Logs contain counts and exception types, never prompts, credentials, or raw provider errors.

The tokenizer endpoint sees the prompt content by design. Run it on a trusted network path and use the same access controls as the inference endpoint.

Metadata

Release files for incontext 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for incontext 0.0.1
File Size Uploaded Provenance Actions
incontext-0.0.1.tar.gz 15.9 kB With provenance Details for incontext-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for incontext 0.0.1
File Interpreter ABI Platform Actions
incontext-0.0.1-py3-none-any.whl Python 3 none any Details for incontext-0.0.1-py3-none-any.whl

Total release size: 30.9 kB

Release files / incontext-0.0.1.tar.gz

Download URL incontext-0.0.1.tar.gz
Size 15.9 kB
Tags Source
SHA-256 checksum
How to use checksums
a90b49da40003fa5222b6466b330890490b81c8f078ace37079f23ed5fe760fa
BLAKE2b-256 checksum
How to use checksums
9fef6b813f385746da836c108b6143ba84fe320e6fc6d508b1c26235f9e706b6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 17, 2026.

Transparency log

Release files / incontext-0.0.1-py3-none-any.whl

Download URL incontext-0.0.1-py3-none-any.whl
Size 15.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
87e7f9b8ea608c49147990ffe98c94a20a38b0a0ebea245fe9cb70fc234c62f4
BLAKE2b-256 checksum
How to use checksums
189f3df4361108c9b111eee12d029de2cb2e0cd8cc5cc79a6faecbc8e47f8f92
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 17, 2026.

Transparency log

Security for incontext 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for incontext-0.0.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page