pyxislm 0.1.0

High-performance, vendor-agnostic LLM inference library

Pyxis

High-performance, vendor-agnostic LLM inference library.

Status snapshot (2026-02-13)

• Sprints 1-5 are completed (docs/SPRINT_CHECKLIST.md).

Quick start (local, 3 processes)

1. Install deps (in a venv):

   • pip install -e .

   Model downloads follow your local HuggingFace/Transformers cache settings.

2. Start the core worker:

   • python scripts/run_core.py --model TinyLlama/TinyLlama-1.1B-Chat-v1.0

3. Start the tokenizer + detokenizer worker:

   • python scripts/run_tokenizer.py --model TinyLlama/TinyLlama-1.1B-Chat-v1.0

4. Start the HTTP API:

   • python scripts/run_api.py --host 127.0.0.1 --port 8000

5. Verify streaming:

   • python scripts/verify_api_real.py

Useful environment variables

• PYXIS_MODEL_PATH: model name/path for scripts/run_core.py (defaults to TinyLlama/TinyLlama-1.1B-Chat-v1.0)
• PYXIS_TOKENIZER_PATH: tokenizer name/path for scripts/run_tokenizer.py (defaults to TinyLlama/TinyLlama-1.1B-Chat-v1.0)
• PYXIS_TOKENIZER_INGRESS: API → tokenizer IPC address override
• PYXIS_DETOK_TO_API: detok → API IPC address override
• PYXIS_CORE_REQUEST_QUEUE_SIZE: max queued generation requests inside core worker (default 1024)
• PYXIS_MAX_INFLIGHT_REQUESTS: max concurrent API streaming requests before 429 overloaded (default 128)
• PYXIS_PER_REQUEST_QUEUE_MAXSIZE: per-request detok queue bound in API (default 128)
• PYXIS_STREAM_IDLE_TIMEOUT_S: stream idle timeout before API emits an error chunk (default 30)
• PYXIS_TOKENIZER_READY_WAIT_S: wait for tokenizer ingress readiness per enqueue (default 1.0)

Smoke / integration scripts

• python scripts/verify_ingestion.py: API-less ingestion test (tokenizer → core request)
• python scripts/verify_api_ingress.py: API streaming test with a mocked core
• python scripts/verify_api_real.py: API streaming test with real core+tokenizer

Realtime usage

• Interactive chat REPL:
  • python scripts/chat_repl.py
• End-to-end realtime harness (starts services, checks streaming/cancel/backpressure):
  • powershell -ExecutionPolicy Bypass -File scripts/test_realtime.ps1 -SkipInstall

Architecture (high level)

HTTP API → TokenizerWorker → CoreWorker → TokenizerWorker (detok) → HTTP streaming response

POST /v1/chat/completions streams SSE (text/event-stream) with OpenAI-style chat.completion.chunk payloads and a final [DONE].

GET /health includes readiness and API stage latency snapshots (stage_latency_ms).

See docs/ARCHITECTURE.md for details.

Session notes and recent implementation memory are tracked in docs/MEMORY.md.