kalavai-client 0.8.17

Client app for kalavai platform

⭐⭐⭐ Kalavai platform is open source, and free to use in both commercial and non-commercial purposes. If you find it useful, consider supporting us by giving a star to our GitHub project, joining our discord channel and follow our Substack.

Kalavai aggregates and coordinates spare GPU capacity

Kalavai is an open source platform that unlocks computing from spare capacity. It aggregates resources from multiple sources to increase your computing budget and run large AI workloads.

Core features

Kalavai helps teams use GPU resources more efficiently. It acts as a control plane for all your GPUs, wherever they are: local, on prem and multi-cloud.

• Increase GPU utilisation from your devices (fractional GPU).
• Multi-node, multi-GPU and multi-architecture support (AMD and NVIDIA).
• Aggregate computing resources from multiple sources: home desktops, on premise servers<0>, multi cloud VMs, raspberry pi's, etc. Including our own GPU fleets.
• Ready-made templates to deploy common AI building blocks: model inference (vLLM, llama.cpp, SGLang), GPU clusters (Ray, GPUStack), automation workflows (n8n and Flowise), evaluation and monitoring tools (Langfuse), production dev tools (LiteLLM, OpenWebUI) and more.
• Easy to expand to custom workloads

Powered by Kalavai

• CoGen AI: A community hosted alternative to OpenAI API for unlimited inference.
• Create your own Free Cursor/Windsurf Clone

Latest updates

• November: Kalavai is now opening a managed service to create and manage AI workloads on a fleet of GPUs. We are inviting Beta Testers for early access. If you are interested Apply here
• September: Kalavai now supports Ray clusters for massively distributed ML.
• August 2025: Added support for AMD GPUs (experimental)
• July 2025: Added support for GPUStack clusters for managed LLM deployments (experimental).
• June 2025: Native support for Mac and Raspberry pi devices (ARM).
• May 2025: Added support for diffusion pipelines (experimental)
• April 2025: Added support for workflow automation engines n8n and Flowise (experimental)
• March 2025: Added support for AI Gateway LiteLLM

More news

• 20 February 2025: New shiny GUI interface to control LLM pools and deploy models- 31 January 2025: kalavai-client is now a PyPI package, easier to install than ever!
• 27 January 2025: Support for accessing pools from remote computers
• 9 January 2025: Added support for SGLang models
• 9 January 2025: Added support for vLLM models
• 9 January 2025: Added support for llama.cpp models
• 24 December 2024: Release of public BOINC pool to donate computing to scientific projects
• 23 December 2024: Release of public petals swarm
• 24 November 2024: Common pools with private user spaces

Support for AI engines

We currently support out of the box the following AI engines:

• vLLM: most popular GPU-based model inference.
• Ray Clusters inference.
• GPUstack (experimental)

Coming soon:

• llama.cpp: CPU-based GGUF model inference.
• SGLang: Super fast GPU-based model inference.
• n8n (experimental): no-code workload automation framework.
• Flowise (experimental): no-code agentic AI workload framework.
• Speaches: audio (speech-to-text and text-to-speech) model inference.
• Langfuse (experimental): open source evaluation and monitoring GenAI framework.
• OpenWebUI: ChatGPT-like UI playground to interface with any models.
• diffusers (experimental)
• RayServe inference.
• GPUstack (experimental)

Not what you were looking for? Tell us what engines you'd like to see.

Kalavai is at an early stage of its development. We encourage people to use it and give us feedback! Although we are trying to minimise breaking changes, these may occur until we have a stable version (v1.0).

Want to know more?

• Get a free Kalavai account and access unlimited AI.
• Full documentation for the project.
• Join our Substack for updates and be part of our community
• Join our discord community

Getting started

The kalavai-client is the main tool to interact with the Kalavai platform, to create and manage GPU pools and also to interact with them (e.g. deploy models). A pool consists of:

• A seed node(s): one (or more for high availability deployments) machine that acts as central control plane
• One or many worker nodes: any machine connected to the seed node that can carry out workloads (generally with access to a GPU)

Requirements

For seed nodes:

• A 64 bits x86 based Linux machine (laptop, desktop or VM)
• Docker engine installed with privilege access.

For workers sharing resources with the pool:

• A laptop, desktop or Virtual Machine. Full support: Linux and Windows; x86 architecture. Limited support: Mac and ARM architecture.
• If self-hosting, workers should be on the same network as the seed node. Looking for over-the-internet connectivity? Check out our managed seeds
• Docker engine installed (for linux, Windows and MacOS) with privilege access.

Compatibility matrix

If your system is not currently supported, open an issue and request it. We are expanding this list constantly.

Install the client

The client is a python package and can be installed with one command:

pip install kalavai-client

Create a a local private pool

For a quick start, get a pool going with:

kalavai pool start

And then start the GUI:

kalavai gui start

This will expose the GUI and the backend services in localhost. By default, the GUI is accessible via http://localhost:49153.

Check out our getting started guide for next steps on how to add more workers to your pool, or use our managed platform for over-the-internet AI pools.

Enough already, let's run stuff!

Check out our use cases documentation for inspiration on what you can do with Kalavai:

• Multi-GPU LLM
• Fine tune
• BYO Model Gateway
• Easy LLMs with GPUstack
• Production GPU fleets

Contribute

Anything missing here? Give us a shout in the discussion board. We welcome discussions, feature requests, issues and PRs!

• Join the community and share ideas!
• Report bugs, issues and new features.
• Help improve our compatibility matrix by testing on different operative systems.
• Follow our Substack channel for news, guides and more.
• Community integrations are template jobs built by Kalavai and the community that makes deploying distributed workflows easy for users. Anyone can extend them and contribute to the repo.

Star History

Build from source

Add Secrets to GitHub

You must store your Docker Hub username and the token you just created as secrets in your GitHub repository:

1. Go to your GitHub repository.

2. Navigate to Settings > Security > Secrets and variables > Actions.

3. Click New repository secret.

4. Create the following two secrets:

Name: DOCKER_HUB_USERNAME
Value: Your Docker Hub username or organization name.

Name: DOCKER_HUB_TOKEN
Value: The Personal Access Token you copied from Docker Hub.

Expand

Python version >= 3.10.

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.10 python3.10-dev python3-virtualenv python3-venv
virtualenv -p python3.10 env
source env/bin/activate
sudo apt install  python3.10-venv python3.10-dev -y
pip install -U setuptools
pip install -e .[dev]

Build python wheels:

bash publish.sh build

Unit tests

To run the unit tests, use:

python -m unittest