EigenCore

A CPU-first Python runtime for local language models, hardware-aware inference, and agent building blocks.

EigenCore profiles a machine, estimates whether a local model fits, and runs inference through llama-cpp-python (llama.cpp). It also ships a benchmark harness, a terminal chat, and agent building blocks with a tool policy layer. The transformer math is llama.cpp's; EigenCore owns the judgement around it.

Reality check: a dense 30–40B model at 4-bit is roughly 18–24 GB of weights, more than 8–16 GB of RAM before any KV cache. It can only run by paging from SSD, and decode is then expected to be well under 1 token/s (estimate). Even when a model fits, CPU decode is memory-bandwidth-bound: tok/s ≤ bandwidth ÷ weight bytes. The realistic route to 30B-class models on a 16 GB laptop is a Mixture-of-Experts model, which is planned and not yet measured. See the review for the arithmetic. No speed figure in this repository is a measurement until a named-hardware run is published.

Quick start

Install from PyPI:

# core CLI (no compiler needed):
python -m pip install eigencore
# + the native inference backend (llama.cpp has no Windows wheel on PyPI,
#   so use the prebuilt index; on GPU machines use the CUDA index instead):
python -m pip install "eigencore[inference]" --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
# pipx alternative: pipx install "eigencore[inference]"
# from source (development): git clone ... && python -m pip install -e ".[dev,inference]"

eigencore profile
eigencore models
eigencore download tinyllama-1.1b
eigencore run --prompt "Explain binary search in one paragraph"
eigencore chat            # add --plain for scripts or minimal terminals

Use eigencore --help for every command. llama-cpp-python may need a backend-specific install; see its documentation before assuming GPU acceleration (EigenCore currently loads models CPU-only).

Status

Labels: SHIPPED runs on main and is tested · REPORTED-ONLY computes plans or numbers that never reach llama.cpp · PLANNED design docs only.

Architecture

flowchart LR
  U[User] --> CLI[Click CLI]
  U --> API[Forge Python API]
  CLI --> HAL[Hardware profile and model registry]
  API --> HAL
  HAL --> ROUTE[Task and model routing]
  ROUTE --> CTX[Context manager]
  CTX --> ENGINE[InferenceEngine]
  ENGINE --> BACKEND[llama-cpp-python backend]
  API --> AGENT[ReAct agent]
  AGENT --> REG[Tool registry]
  REG --> EXEC[ToolExecutor]
  EXEC -. handlers run in process today .-> HOST[Python process]
  TRAIN[Optional training components] --> ADAPTER[LoRA adapter output]
  HAL -. proposed: verified fit estimates .-> UX[Future adaptive TUI]

This is a module map, not a process-isolation diagram. In particular, the current executor’s validation and timeout are not an OS sandbox or an approval workflow.

Development

python -m pip install -e ".[dev,inference]"
pytest tests/ -v
ruff check src/ tests/
ruff format --check src/ tests/

See CONTRIBUTING.md. Report security issues privately as described in SECURITY.md. This project follows a code of conduct.

Documentation

• Review and plan (2026-11): truth audit, 30–40B arithmetic, prioritized plan, benchmarks to run.
• Design handbook: v2 identity, palette, terminal UI, approval flow.
• Terminal UI status and v2 screens.
• Brand assets.
• System architecture and local inference roadmap.
• Terminal chat design.
• Documentation index.
• Changelog.

Project status

Pre-alpha. APIs and behaviour may change. Feature presence is not a performance claim.

License

MIT © 2026 Khubaib Nazeer.