Skip to main content

Inferlab Integration for Specialized Engines

This package connects Inferlab to a hardware-by-model Specialized Engine that implements the canonical inferlab-token-engine smg-worker command. The Engine accepts prompt token IDs and returns generated token IDs over SMG's worker protocol. TokenSpeed SMG remains responsible for the public HTTP API, tokenization, chat templates, detokenization, and response formatting.

The integration contains no model-, architecture-, hardware-, or Engine-implementation-specific lowering. A downstream workspace supplies the Rust Engine binary, SMG, model intent, source revision, locked environment, and private placement bindings. Consequently, a new conforming Engine does not need another Inferlab integration package.

The supported shape is deliberately closed: one single Engine replica in one process behind one SMG Gateway. The process owns an arbitrary nonzero pure-TP device set; attention, expert, and dense-expert tensor parallelism all equal the outer TP width, while pipeline, data, context, and expert parallelism stay at one. The contract remains serial and has no P/D Router, KV-transfer, batching, or Engine-local profiling surface. InferLab can profile the Engine process tree while TokenSpeed SMG exposes the capture-window POST /start_profile and POST /stop_profile actions on the Gateway.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

File details

Details for the file inferlab_integration_specialized_engine-0.2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for inferlab_integration_specialized_engine-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 236b9878058ad9e5c5f672c7d377b8f050ea147ae80ce65f51e1d1dcabecd3f2
MD5 7f83a390ff08cc0523d1421528141395
BLAKE2b-256 ebef4907511e6a81ed1ac3017092122f08d4c8db18d6e76bfcc08ed69c6f5576

See more details on using hashes here.

Release history Release notifications | RSS feed

0.10.0

1 file

0.9.0

1 file

0.8.0

1 file

0.5.0

1 file

0.4.0

1 file

0.3.1

1 file

0.3.0

1 file

0.2.2

1 file

0.2.1

1 file

This release

0.2.0 This release

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page