Skip to main content

Inferlab Integration for Specialized Engines

This package connects Inferlab to a hardware-by-model Specialized Engine that implements the canonical inferlab-token-engine smg-worker command. The Engine accepts prompt token IDs and returns generated token IDs over SMG's worker protocol. TokenSpeed SMG remains responsible for the public HTTP API, tokenization, chat templates, detokenization, and response formatting.

The integration contains no model-, architecture-, hardware-, or Engine-implementation-specific lowering. A downstream workspace supplies the Rust Engine binary, SMG, model intent, source revision, locked environment, and private placement bindings. Consequently, a new conforming Engine does not need another Inferlab integration package.

The supported shape is deliberately closed: one single Engine replica in one process behind one SMG Gateway. The process owns an arbitrary nonzero pure-TP device set; attention, expert, and dense-expert tensor parallelism all equal the outer TP width, while pipeline, data, context, and expert parallelism stay at one. The contract remains serial and has no P/D Router, KV-transfer, batching, or Engine-local profiling surface. InferLab can profile the Engine process tree while TokenSpeed SMG exposes the capture-window POST /start_profile and POST /stop_profile actions on the Gateway.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

File details

Details for the file inferlab_integration_specialized_engine-0.2.1-py3-none-any.whl.

File metadata

File hashes

Hashes for inferlab_integration_specialized_engine-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 2b666fd7587825a7044f8b32e1695f28fcb865aa533813e992457f02336f9f6f
MD5 29c2d6565fb5d48e76ccf19b66fdf08d
BLAKE2b-256 124be6d02555268119a1032987160eded2ad45dc39e4b686075335e6241e4151

See more details on using hashes here.

Release history Release notifications | RSS feed

0.10.0

1 file

0.9.0

1 file

0.8.0

1 file

0.5.0

1 file

0.4.0

1 file

0.3.1

1 file

0.3.0

1 file

0.2.2

1 file

This release

0.2.1 This release

1 file

0.2.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page