Skip to main content

LatentPort

LatentPort makes LLM internal states programmable.

We're building a library of learned state transformations that enable models and agents to transfer, compact, optimise and combine their internal states during inference.

This allows developers to improve model performance, reduce inference costs and build longer-running agents without repeatedly processing the same context.

LatentPort: programmable model state

Website | Get started | Models | Cookbook | API

License: Apache-2.0 PyPI

A library of state operations

A model's internal state carries what it has already processed. LatentPort exposes that state as an object developers can transform, persist, reuse and branch. Learned operators work directly on state tensors, allowing inference to continue from the transformed state.

Operation Purpose Available today
Transfer Move working context into another model's state space. Learned Qwen3.5-4B to Qwen3.8-27B bridge via bridge.translate(state).
Compact Reduce attention memory while retaining the model's recurrent and convolutional state. LPCompact for Qwen3.8-27B via state.compact().
Optimise Improve a state for the task or inference objective ahead. Part of the roadmap.
Combine Bring information from multiple states into a shared working state. Part of the roadmap.

The shared runtime also provides capture, append, save, load and fork operations. Snapshots are immutable, so branches can share a starting context while advancing independently. Each learned transformation is trained for its supported model architecture or model pair.

Get started

Install the lightweight Python package with Python 3.11 or later:

python -m pip install latentport
latentport doctor

The base package includes the API, diagnostics and demo interface. Inference dependencies are optional: see installation for latentport[gguf], latentport[bf16] and CUDA PyTorch setup.

Try the browser demo

On a qualified NVIDIA GPU host with Docker GPU access configured:

docker run --rm --gpus all -p 127.0.0.1:8768:8768 -v latentport-cache:/cache ghcr.io/latentport/latentport:0.2.1-gguf-cu128-sm120

Open http://localhost:8768. The demo prepares the models, transfers context, forks specialist sessions, and checks a generated repair with executable tests. The initial transfer profile requires an SM120 GPU with at least 32 GB VRAM and approximately 22.10 GB of model downloads; subsequent runs reuse the volume. Windows Docker GPU execution requires the WSL 2 backend.

Release status: the Python package is available on PyPI. The GHCR images are pending GPU qualification and publication; until then, use the local image build.

Choose the setup for your operation:

Set LATENTPORT_HOME to a directory with enough space for downloaded models and caches. The runtime downloads transformation weights when needed; base models are separate artifacts. Prebuilt GGUF runtimes are available for Windows x64 / CUDA 13 and Linux x64 / CUDA 12.8 on Blackwell SM120.

Compact a running model's state

LPCompact-Qwen3.8-27B provides a learned selector through a single operation:

state = state.compact()

Given an agent's accumulated history, continue from its compacted state:

import latentport as lp

with lp.load_model(backend="bf16") as model:
    state = model.prefill(history)
    state = state.compact()
    answer = model.session(state).ask("Continue the task.")
    print(answer.text)

The default policy retains one eighth of attention KV positions on histories with at least 65,536 retained positions. A 4x setting is available with state.compact(ratio=4). Recurrent state, convolution state and the original logical positions are preserved. Compaction uses no text summary or historical token replay.

One selector checkpoint serves Qwen3.8-27B BF16 and GGUF weight quantizations. For GGUF, use backend="gguf" and supply model_path with your chosen base-model file. See the compaction guide for context sizing, policy details and the state.compaction receipt.

Transfer context between models

The current transfer bridge lets Qwen3.5-4B read a context and Qwen3.8-27B continue from the translated state. It transfers attention K/V, recurrent memory, convolution history and the terminal hidden state.

import latentport as lp

with lp.load(backend="gguf") as port:
    port.warm()
    context = port.source.capture([("user", "Repository context: ...")])
    transferred = port.bridge.translate(context)
    answer = port.target.session(transferred).ask("Explain the repository.")
    print(answer.text)

For a complete conversation, port.composite.session() handles the initial handoff and retains the target's state across turns. The default cached_handoff policy translates once, then processes new input on the target. An explicit strict policy keeps the source model responsible for new user prompts.

The cookbook covers both policies, answer continuation, streaming and MTP. Existing local GGUF files can be supplied through source_path and target_path.

Reuse, branch and resume

The same state abstraction supports persistence and independent continuations:

review_state, planning_state = state.fork(2)
state.save("saved-state")
restored = lp.State.load("saved-state")

Prepare a repository once, let different agents work from the same snapshot, or save a long-running session and resume it later. Restoring a state reuses the captured context. Appending new tool results processes the new input.

Each snapshot records its exact model identity and runtime compatibility. Forks share immutable storage; execution uses the selected backend's scheduling.

Measured results

Transformation Measurement Validation
LPCompact, 128K history at 8x Attention KV tensors reduced from 8 GiB to 1 GiB. BF16, Q4_K_M and Q8_0 tested at 64K and 128K on RTX PRO 6000; continued generation, save/load, forks and repeated compaction passed.
4B to 27B transfer, approximately 16K prompts 2.19x paired time-to-first-token speedup; native median 24.663 s, composite median 11.087 s. E120 resident NF4/BF16 research runtime on RTX 5090; paired 95% interval 2.15-2.23x.

These measurements describe their stated workloads and configurations. Compaction reduces attention-state storage; model weights and recurrent state retain their existing representation. The transfer timing includes prefill, translation, installation and the first emitted token, excluding cold model loading.

See compaction validation, transfer results and benchmark methodology for the full records.

A state operating layer for inference

The long-term direction is a state operating layer for AI inference:

  • Model-specific translators connect the internal state spaces of compatible models.
  • Reusable operators transfer, compact, optimise and combine state through a shared interface.
  • A policy engine selects transformations automatically according to the application's capability, latency and memory requirements.

The current release provides explicit transfer and compaction operations, plus the state lifecycle primitives. Learned optimisation, state combination and automatic policy selection are the next layers of that vision.

One runtime, separate transformation models

The shared Python runtime lives in this GitHub repository. Each trained state transformation has its own Hugging Face repository, with model compatibility, weights, versions and a model card.

Transformation repositories contain their own learned artifacts. Base-model weights are obtained separately.

Demos and documentation

Report reproducible issues at GitHub Issues. Include the output of python -m latentport doctor, model identities and a minimal example, with private prompts and credentials removed.

License

The LatentPort runtime and transformation weights are Apache-2.0. Qwen base models retain their upstream licenses, and llama.cpp retains its MIT license. See LICENSE and the notices supplied with the model and native downloads.

Metadata

Release files for latentport 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for latentport 0.2.1
File Size Uploaded Provenance Actions
latentport-0.2.1.tar.gz 6.7 MB Without provenance Details for latentport-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for latentport 0.2.1
File Interpreter ABI Platform Actions
latentport-0.2.1-py3-none-any.whl Python 3 none any Details for latentport-0.2.1-py3-none-any.whl

Total release size: 7.7 MB

Release files / latentport-0.2.1.tar.gz

Download URL latentport-0.2.1.tar.gz
Size 6.7 MB
Tags Source
SHA-256 checksum
How to use checksums
1c27884ee5fe513b326d2b3425889319151e75280ddfa5f416c62dba32ebcaec
BLAKE2b-256 checksum
How to use checksums
50f4ddad31c7039b7e41fc77905122b24d9e412cde50becb703fff4169852032
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Release files / latentport-0.2.1-py3-none-any.whl

Download URL latentport-0.2.1-py3-none-any.whl
Size 942.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1a9e24a6631150f64e04f6e07e59f369bf1e90c0a2cbd09a705b4c77b81b2499
BLAKE2b-256 checksum
How to use checksums
99f5d92496eb7958954a29c7d96898c9ff15ab22bdebcdfe278b8e2354a927cb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Security for latentport 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page