hs_connectors
hs_connectors is a lightweight package that enables the transfer of hidden states between vLLM and Speculators training processes during both online and offline training. It provides a pluggable backend system where each backend registers itself via @HiddenStatesBackend.register(name) and exposes hooks for CLI argument injection and configuration, so that training and serving scripts can discover and configure backends without hardcoding.
The package should be installed alongside Speculators as well as within the same environment as vLLM.
Backends
Filesystem
The filesystem backend enables the transfer of hidden states between vLLM and Speculators on a single node with a shared filesystem. Hidden states are serialized as safetensors files with file-lock synchronization to prevent partial reads. Cached samples are stored by index and can be reused across training runs.
Mooncake
The Mooncake backend enables multi-node training by transferring hidden states between vLLM and Speculators across nodes over TCP or RDMA, without requiring a shared filesystem. It uses a Mooncake distributed key-value store where each sample is written under a sanitized request ID, with a metadata marker written last to signal that the sample is complete and ready for consumption. This follows a similar pattern to disaggregated prefill and decode.
When using this backend, install the Mooncake transfer engine separately (pip install mooncake-transfer-engine).
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file hs_connectors-0.8.0.tar.gz.
File metadata
- Download URL: hs_connectors-0.8.0.tar.gz
- Upload date:
- Size: 18.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
07fcb1b6d0ea2014f9c7f12cffcd67338954d0552a44a2dfb54a833c37722c5e
|
|
| MD5 |
855c80da0261d6e7d75d90deb8dc03f2
|
|
| BLAKE2b-256 |
3ed86038076ae53a9637e2f8644f65a80758cf9811458bf2e1d5da51f91f6555
|
Provenance
The following attestation bundles were made for hs_connectors-0.8.0.tar.gz:
Publisher:
hsc-upload.yml on neuralmagic/llm-compressor-testing
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
hs_connectors-0.8.0.tar.gz -
Subject digest:
07fcb1b6d0ea2014f9c7f12cffcd67338954d0552a44a2dfb54a833c37722c5e - Sigstore transparency entry: 2703219851
- Sigstore integration time:
-
Permalink:
neuralmagic/llm-compressor-testing@9fa0c0ca74ed7c714d66f3cd362dfcd5d18a9eb0 -
Branch / Tag:
refs/heads/fixupload - Owner: https://github.com/neuralmagic
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
hsc-upload.yml@9fa0c0ca74ed7c714d66f3cd362dfcd5d18a9eb0 -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file hs_connectors-0.8.0-py3-none-any.whl.
File metadata
- Download URL: hs_connectors-0.8.0-py3-none-any.whl
- Upload date:
- Size: 18.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c2c7826b9f02e5701ee7d3215a9583fc5a19fd38ce7b42383d224f32868052ca
|
|
| MD5 |
a0800526e66d643eb51511e8debe903b
|
|
| BLAKE2b-256 |
2e244ab4664408e11544681fb114ef5b7248b11dc6458d998aa4c0526aceffc6
|
Provenance
The following attestation bundles were made for hs_connectors-0.8.0-py3-none-any.whl:
Publisher:
hsc-upload.yml on neuralmagic/llm-compressor-testing
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
hs_connectors-0.8.0-py3-none-any.whl -
Subject digest:
c2c7826b9f02e5701ee7d3215a9583fc5a19fd38ce7b42383d224f32868052ca - Sigstore transparency entry: 2703220016
- Sigstore integration time:
-
Permalink:
neuralmagic/llm-compressor-testing@9fa0c0ca74ed7c714d66f3cd362dfcd5d18a9eb0 -
Branch / Tag:
refs/heads/fixupload - Owner: https://github.com/neuralmagic
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
hsc-upload.yml@9fa0c0ca74ed7c714d66f3cd362dfcd5d18a9eb0 -
Trigger Event:
workflow_dispatch
-
Statement type:
