myel 0.5.0

Myelin — bio-mimetic dynamic layer-masking local LLM engine (Python bindings)

myelin — Python bindings

First-class PyO3 bindings for the Myelin engine: a bio-mimetic, dynamic layer-masking local LLM runtime. import myelin loads a native extension module — no ctypes, no manual signatures, no server process.

• Local: the model runs on your GPU; nothing leaves your machine.
• Lean: weights live in VRAM int8-quantized; host RAM stays near the embedding table (~1.6 GB for Llama 3.2 1B).
• Bio-mimetic masking: per-layer importance is calibrated from the model itself at load; under hardware pressure (or an explicit quality budget) unimportant layers are skipped for speed.

Install

pip install myel

The PyPI distribution name is myel; the importable module is myelin (import myelin) — the two differ because myelin was already taken.

Terminal (Myelin CLI)

Installing the package also installs a myelin command:

myelin list                    # catalog + downloaded/active state
myelin pull gemma-2-2b-it      # download a model (live progress bar)
myelin run gemma-2-2b-it       # interactive chat (REPL; /exit, /clear, /model <id>)
myelin run gemma-2-2b-it "What is 2+2?"   # one-shot prompt
myelin run llama-3.1-8b-instruct --stop "###" --min-p 0.05 --temperature 0.8
myelin show gemma-2-2b-it      # model details
myelin rm qwen2.5-0.5b-instruct  # delete a downloaded model from disk
myelin ps                      # active model + hardware state
myelin hardware                # detected GPU / telemetry
myelin version                 # package + engine version

Every reply prints a dim ⏱ N token, X tok/s stats line on stderr.

Quick start

import myelin

eng = myelin.Engine()                      # returns immediately; the default
                                           # model loads in the background
eng.wait_ready()                           # optional — generate() also waits

print(eng.generate("The capital of France is", max_tokens=32))
print(eng.chat("Explain transformers in one sentence."))

for piece in eng.stream_iter("Write a haiku about GPUs."):
    print(piece, end="", flush=True)

First run with no model on disk:

eng = myelin.Engine("llama-3.2-1b-instruct", download=True)   # ~2.5 GB, once

API

Engine

Engine(model=None, *, download=False, models_dir=None, profile=None)

• model — catalog id to activate; blocks until loaded. Without it the default model auto-loads in the background.
• download — with model, fetch it first if missing.
• models_dir — model library root (default MYELIN_MODELS_DIR or ./models).
• profile — "quality" (uniform int8, default) or "speed" (calibrated mixed int8/int4, ~25% faster at a small perplexity cost).

Generation

eng.generate(prompt, max_tokens=128, *, temperature, top_k, top_p,
             repeat_penalty, seed, quality=1.0) -> str

Raw completion (no chat template). Sampling defaults are the Llama-family chat settings (temperature 0.7, top-k 40, top-p 0.95, repeat penalty 1.1); pass temperature=0.0 for greedy decoding, seed= for reproducibility.

quality is the decision engine's quality floor: 1.0 runs every layer; lower values let the bio-mimetic mask skip calibrated-unimportant layers.

eng.chat(messages, max_tokens=256, *, system=None, ...) -> str

Chat completion through the Llama 3 instruct template. messages is a plain string (single user turn) or a full conversation:

history = [
    {"role": "user", "content": "My name is Ada."},
    {"role": "assistant", "content": "Nice to meet you, Ada!"},
    {"role": "user", "content": "What's my name?"},
]
eng.chat(history, system="Answer briefly.")

Streaming

eng.stream(messages, callback, max_tokens=256, ...) -> str   # callback per fragment
for piece in eng.stream_iter(messages, ...):                 # or iterate
    ...

stream(..., inspect=True, on_activity=fn) additionally calls fn(layer_norms, depth) per token with the real per-layer activation — the "neuron liveness" signal (slower; deliberate).

Introspection

text, info = eng.generate_detailed(prompt, ...)
info["active_layers"]   # per-layer bool mask this generation ran with
info["depth"]           # how many layers were active
info["policy"]          # layer-selection policy (e.g. BIOMIMETIC)

eng.hardware()          # {'vram_used_mb', 'temperature_c', 'utilization_pct', ...}

Model library

eng.list_models()       # curated catalog + download/active/loading status
eng.download_model("llama-3.2-3b-instruct", lambda p: print(p["downloaded_bytes"]))
eng.load_model("llama-3.2-3b-instruct")     # hot-swap; frees old model's VRAM
eng.active_model()      # -> str | None
eng.is_ready()          # -> bool
eng.wait_ready(timeout=None)

Errors

Engine-level failures raise myelin.MyelinError (a RuntimeError subclass); argument mistakes raise TypeError/ValueError.

Notes

• The GIL is released during Rust-side compute, so generations never block other Python threads; callbacks re-acquire it per fragment — keep them light.
• One Engine per process is the intended pattern (it owns the GPU state). It is thread-safe; generations serialize on the single GPU.
• The catalog is curated (Llama 3.2 1B/3B Instruct) — the runtime implements the Llama 3.x architecture only.