LatentPort
LatentPort makes LLM internal states programmable.
We're building a library of learned state transformations that enable models and agents to transfer, compact, optimise and combine their internal states during inference.
This allows developers to improve model performance, reduce inference costs and build longer-running agents without repeatedly processing the same context.
Website | Get started | Models | Cookbook | API
A library of state operations
A model's internal state carries what it has already processed. LatentPort exposes that state as an object developers can transform, persist, reuse and branch. Learned operators work directly on state tensors, allowing inference to continue from the transformed state.
| Operation | Purpose | Available today |
|---|---|---|
| Transfer | Move working context into another model's state space. | Learned Qwen3.5-4B to Qwen3.8-27B bridge via bridge.translate(state). |
| Compact | Reduce attention memory while retaining the model's recurrent and convolutional state. | LPCompact 0.1 for Qwen3.8-27B via state.compact(). |
| Optimise | Improve a state for the task or inference objective ahead. | Part of the roadmap. |
| Combine | Bring information from multiple states into a shared working state. | Part of the roadmap. |
The shared runtime also provides capture, append, save, load and fork operations. Snapshots are immutable, so branches can share a starting context while advancing independently. Each learned transformation is trained for its supported model architecture or model pair.
Get started
Install the lightweight Python package with Python 3.11 or later:
python -m pip install latentport
latentport doctor
The base package includes the API, diagnostics and demo interface. Inference
dependencies are optional: see installation for
latentport[gguf], latentport[bf16] and CUDA PyTorch setup.
Try the browser demo
On a qualified NVIDIA GPU host with Docker GPU access configured:
docker run --rm --gpus all -p 127.0.0.1:8768:8768 -v latentport-cache:/cache ghcr.io/latentport/latentport:0.2.0-gguf-cu128-sm120
Open http://localhost:8768. The demo prepares the models, transfers context, forks specialist sessions, and checks a generated repair with executable tests. The initial transfer profile requires an SM120 GPU with at least 32 GB VRAM and approximately 22.10 GB of model downloads; subsequent runs reuse the volume. Windows Docker GPU execution requires the WSL 2 backend.
Release status: the Python package is available on PyPI. The GHCR images are pending GPU qualification and publication; until then, use the local image build.
Choose the setup for your operation:
- Compaction: BF16 and GGUF setup, including the BF16 kernel dependencies and native GGUF runtime.
- Model transfer: Windows GGUF setup and conversation recipes.
Set LATENTPORT_HOME to a directory with enough space for downloaded models and
caches. The runtime downloads transformation weights when needed; base models are
separate artifacts. Prebuilt GGUF runtimes are available for Windows x64 / CUDA 13
and Linux x64 / CUDA 12.8 on Blackwell SM120.
Compact a running model's state
LPCompact-Qwen3.8-27B provides a learned selector through a single operation:
state = state.compact()
Given an agent's accumulated history, continue from its compacted state:
import latentport as lp
with lp.load_model(backend="bf16") as model:
state = model.prefill(history)
state = state.compact()
answer = model.session(state).ask("Continue the task.")
print(answer.text)
The default policy retains one eighth of attention KV positions on histories with
at least 65,536 retained positions. A 4x setting is available with
state.compact(ratio=4). Recurrent state, convolution state and the original
logical positions are preserved. Compaction uses no text summary or historical
token replay.
One selector checkpoint serves Qwen3.8-27B BF16 and GGUF weight quantizations.
For GGUF, use backend="gguf" and supply model_path with your chosen base-model
file. See the compaction guide for context sizing, policy
details and the state.compaction receipt.
Transfer context between models
The current transfer bridge lets Qwen3.5-4B read a context and Qwen3.8-27B continue from the translated state. It transfers attention K/V, recurrent memory, convolution history and the terminal hidden state.
import latentport as lp
with lp.load(backend="gguf") as port:
port.warm()
context = port.source.capture([("user", "Repository context: ...")])
transferred = port.bridge.translate(context)
answer = port.target.session(transferred).ask("Explain the repository.")
print(answer.text)
For a complete conversation, port.composite.session() handles the initial
handoff and retains the target's state across turns. The default cached_handoff
policy translates once, then processes new input on the target. An explicit
strict policy keeps the source model responsible for new user prompts.
The cookbook covers both policies, answer continuation,
streaming and MTP. Existing local GGUF files can be supplied through source_path
and target_path.
Reuse, branch and resume
The same state abstraction supports persistence and independent continuations:
review_state, planning_state = state.fork(2)
state.save("saved-state")
restored = lp.State.load("saved-state")
Prepare a repository once, let different agents work from the same snapshot, or save a long-running session and resume it later. Restoring a state reuses the captured context. Appending new tool results processes the new input.
Each snapshot records its exact model identity and runtime compatibility. Forks share immutable storage; execution uses the selected backend's scheduling.
Measured results
| Transformation | Measurement | Validation |
|---|---|---|
| LPCompact, 128K history at 8x | Attention KV tensors reduced from 8 GiB to 1 GiB. | BF16, Q4_K_M and Q8_0 tested at 64K and 128K on RTX PRO 6000; continued generation, save/load, forks and repeated compaction passed. |
| 4B to 27B transfer, approximately 16K prompts | 2.19x paired time-to-first-token speedup; native median 24.663 s, composite median 11.087 s. | E120 resident NF4/BF16 research runtime on RTX 5090; paired 95% interval 2.15-2.23x. |
These measurements describe their stated workloads and configurations. Compaction reduces attention-state storage; model weights and recurrent state retain their existing representation. The transfer timing includes prefill, translation, installation and the first emitted token, excluding cold model loading.
See compaction validation, transfer results and benchmark methodology for the full records.
A state operating layer for inference
The long-term direction is a state operating layer for AI inference:
- Model-specific translators connect the internal state spaces of compatible models.
- Reusable operators transfer, compact, optimise and combine state through a shared interface.
- A policy engine selects transformations automatically according to the application's capability, latency and memory requirements.
The current release provides explicit transfer and compaction operations, plus the state lifecycle primitives. Learned optimisation, state combination and automatic policy selection are the next layers of that vision.
One runtime, separate transformation models
The shared Python runtime lives in this GitHub repository. Each trained state transformation has its own Hugging Face repository, with model compatibility, weights, versions and a model card.
- LPCompact-Qwen3.8-27B: the compaction selector shared across BF16 and GGUF.
- 4B to 27B transfer bridge: the current directional translator; GGUF artifacts are also available.
Transformation repositories contain their own learned artifacts. Base-model weights are obtained separately.
Demos and documentation
- Memory transplant demo: live state transfer, specialist branches and a repair checked by executable tests.
- State demos: cached conversations, branching, save/resume and fresh context updates.
- Time-to-first-token demo: compare native 27B and composite inference on a fresh 16K prompt.
- Compaction guide, cookbook and API contracts.
- Native integration and contributing.
Report reproducible issues at GitHub Issues.
Include the output of python -m latentport doctor, model identities and a minimal
example, with private prompts and credentials removed.
License
The LatentPort runtime and transformation weights are Apache-2.0. Qwen base models retain their upstream licenses, and llama.cpp retains its MIT license. See LICENSE and the notices supplied with the model and native downloads.
Metadata
Release files for latentport 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| latentport-0.2.0.tar.gz | 6.7 MB | Without provenance | Details for latentport-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| latentport-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for latentport-0.2.0-py3-none-any.whl |
Total release size: 7.7 MB
Release files / latentport-0.2.0.tar.gz
| Download URL | latentport-0.2.0.tar.gz |
|---|---|
| Size | 6.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8fc1d1ea70174266267c37459a12f087b212d824ab51cb9eb648295d874783d9
|
|
BLAKE2b-256 checksum How to use checksums |
6c608cae36c8b1f66c6954e1937eabc7d71baa8b973b5cd9f42f5a6e4b270533
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Release files / latentport-0.2.0-py3-none-any.whl
| Download URL | latentport-0.2.0-py3-none-any.whl |
|---|---|
| Size | 942.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
baebdad70b208365f16cccf6186b930fda93c9e93a11e2130e2d808c14164073
|
|
BLAKE2b-256 checksum How to use checksums |
57b29bb4aee4c2536ef1a7b55e47d3476db8c7be571305dba0d9e8f38c11bdf0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.9
|
Security for latentport 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.