Skip to main content

kllm: your coding tools talk to a local relay, which runs the model on two Tesla T4 GPUs in a Kaggle notebook

tests PyPI Python 3.10 to 3.13 Linux, macOS, Windows MIT license

kllm runs open-weight LLMs on Kaggle's free GPUs and gives your coding tools a local API for them.

Your laptop has no GPU. You still want Claude Code to refactor a 2,000-line module on a 35B model, with a 256K context window, without paying per token. With kllm you install one package, paste a Kaggle token once, and run claude-kllm. Ten minutes later the model is answering from two Tesla T4s in a Kaggle notebook, and Claude Code talks to it as if it were running on 127.0.0.1.

That raises a few questions, answered below:

How do I get started?

Terminal: pipx install kllm-gateway, then kllm finds the Kaggle token, writes the config and prints the panel URL, then claude-kllm

1. Install with one command. The one-line installers bring their own Python through uv.

You have Run
Nothing (Linux, macOS) curl -fsSL https://raw.githubusercontent.com/SANTHAN-KUMAR/kllm/main/install.sh | sh
Nothing (Windows) powershell -ExecutionPolicy ByPass -c "irm https://raw.githubusercontent.com/SANTHAN-KUMAR/kllm/main/install.ps1 | iex"
uv uv tool install kllm-gateway
pipx pipx install kllm-gateway
pip pip install kllm-gateway

The package is kllm-gateway on PyPI. It installs two commands: kllm and claude-kllm.

2. Run kllm. It starts the local relay and opens the control panel in your browser. On a first run the panel shows a four-step setup: add your Kaggle account, pick a model, connect a tool, say hello.

The kllm panel's first-run page: open your Kaggle settings, verify your phone, create a token, then paste it or drop kaggle.json

First run: the panel links to your Kaggle settings and takes the token as a paste or a dropped kaggle.json. No file to edit.

You need a Kaggle account with a verified phone number, because Kaggle only gives GPUs to verified accounts. If ~/.kaggle/kaggle.json already exists, kllm finds it and skips the paste.

3. Use it.

claude-kllm                                  # Claude Code on the default model
opencode -m kllm/qwen3.6-35b-abliterated     # opencode
kllm down                                    # stop the GPU when you are done, to save quota

The first request starts a GPU if none is running. Expect about 8 to 10 minutes for a cold start, while Kaggle boots the notebook and downloads the model. The request waits and then continues on its own.

What happens when I run it?

Your tools never talk to Kaggle. They talk to a small relay on your PC (one Python process, about 55 MB of RAM), and the relay does the Kaggle work.

sequenceDiagram
    autonumber
    participant T as Claude Code
    participant R as kllm relay (your PC)
    participant K as Kaggle notebook (2x T4)
    T->>R: first request for qwen3.6-35b-abliterated
    R->>K: push a private notebook with your Kaggle token
    Note over K: build llama.cpp, download the model, open a tunnel
    K-->>R: tunnel URL, sealed and signed
    R->>K: forward the request through the tunnel
    K-->>T: streamed answer (the relay sends keep-alives while it waits)

The relay holds the first request while the GPU starts, so the tool needs no retry logic.

After that, every request goes straight from the relay through the tunnel. The relay keeps each conversation on the same GPU so the prompt cache stays warm: in measured sessions, 99% of each turn's prompt came from the cache at 55K tokens of context. Details are in How kllm works.

Which models can I run, and how fast are they?

Two models are measured end to end on two T4s. The panel lists 46 models in all, and adds any GGUF or diffusers repo from Hugging Face.

Model Best for Context Measured speed
qwen3.6-35b-abliterated (default) Long agent sessions; thinks before answering 256K 70 tok/s short, 43 tok/s at 209K
qwen3-coder-30b-abliterated Fastest turnaround, no thinking 96K 84 tok/s short, 35 tok/s at 84K

The kllm panel's model library: 46 models, filterable by chat, image and video, with context size, quantisation and VRAM for each

The model library shows what fits two T4s before you launch anything.

Image and video models (SDXL Turbo, LTX-Video, Wan 2.1) run in the same notebook behind the OpenAI Images API. Anything above 30 GB of VRAM does not fit. See Models for adding models and calling image and video endpoints.

How do I connect Claude Code, opencode or my own code?

The relay speaks both the OpenAI and the Anthropic API at http://127.0.0.1:8080. Requests from your own machine need no key.

Tool How
Claude Code claude-kllm, which sets the model and the real context window for you
opencode, Codex CLI, Gemini CLI, Antigravity, Cursor the panel's Connect page writes their config and keeps a backup for Undo
Any OpenAI client base URL http://127.0.0.1:8080/v1, model kllm (whatever is running)
Any Anthropic client ANTHROPIC_BASE_URL=http://127.0.0.1:8080

From your own code it is a normal chat call:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
reply = client.chat.completions.create(
    model="kllm",
    messages=[{"role": "user", "content": "Write a SQL query for the 5 newest orders."}],
)
print(reply.choices[0].message.content)

Other devices on your network, or a public URL, can use the same API with the gateway key shown in the panel's Settings. See Configuration.

What happens when Kaggle's 12-hour limit hits?

A Kaggle notebook stops after 12 hours. kllm plans for it: 25 minutes before the limit it starts a successor, replays each conversation's latest prompt there so the cache is warm, and only then moves the conversation.

gantt
    title One long Claude Code session across two Kaggle notebooks
    dateFormat HH:mm
    axisFormat %H:%M
    section Notebook A
    serving requests          :a1, 00:00, 11h
    section Notebook B
    starting (8 to 10 min)    :b0, 10:35, 10m
    warm handoff              :crit, b1, 10:45, 5m
    serving requests          :b2, 10:50, 6h

The successor is ready before the old notebook stops, so the session never sees the 12-hour cap.

The same handoff runs when a notebook dies early. In a 72-minute test with three agent sessions in parallel, contexts grew to 165K tokens and generation held at 50 to 63 tok/s from empty to 175K (full results).

Each Kaggle account gets about 30 GPU hours a week. kllm down stops the GPU, and an idle GPU stops by itself after 25 minutes without requests.

Is my code and my Kaggle token safe?

  • Your Kaggle token stays on your PC, in your user folder. kllm uses it only to call Kaggle's API from your machine; it is never put into the notebook or sent anywhere else.
  • Your prompts go only to your notebook, through a tunnel protected by a per-notebook API key.
  • The notebook's address is sealed. The notebook reports its tunnel URL through ntfy.sh, encrypted and signed with a secret generated on your PC, so nobody can read it or swap in a fake URL.
  • The panel is local only. Requests from other machines need the gateway key and can reach only the model API.

kllm uses your Kaggle account and its free quota. Follow Kaggle's Terms of Service, which allow one account per person. kllm can manage several accounts (for example a team where each person adds their own), but which accounts you add is your responsibility.

What does not work yet?

  • The first launch of a model on an account downloads about 20 GB, so it takes 8 to 10 minutes. Kaggle's GPU queue occasionally adds more.
  • Models above about 28 GB of weights plus cache do not fit two T4s. A 230B model such as MiniMax-M2 cannot run.
  • The bundled image and video entries are marked unverified: they were sized and checked, but not run on a GPU.
  • Qwen3.6 sometimes thinks at length on trivial requests. That costs seconds, not quality.

Back to that refactor

Claude Code is working through the 2,000-line module. It started on Notebook A at 09:00, the relay moved it to Notebook B at 19:45 with its cache warm, and it is still answering at 43 tok/s with 200K tokens of context. Nothing ran on your laptop but a 55 MB relay.

More

Read For
How kllm works speed tuning, loop guard, tunnel keep-alive, GPU scheduling, the long-session test
Models choosing a model, adding one from Hugging Face, image and video endpoints
Configuration every tool's setup, remote access, gateway key, file layout
Contributing running the tests

Contributing

Issues and pull requests are welcome. Run the tests with pip install -e ".[dev]" and python -m pytest. They never contact Kaggle, GitHub or ntfy, and never touch your home directory.

© 2026 SANTHAN-KUMAR. Released under the MIT license.

Metadata

Release files for kllm-gateway 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kllm-gateway 0.3.0
File Size Uploaded Provenance Actions
kllm_gateway-0.3.0.tar.gz 4.3 MB With provenance Details for kllm_gateway-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kllm-gateway 0.3.0
File Interpreter ABI Platform Actions
kllm_gateway-0.3.0-py3-none-any.whl Python 3 none any Details for kllm_gateway-0.3.0-py3-none-any.whl

Total release size: 8.0 MB

Release files / kllm_gateway-0.3.0.tar.gz

Download URL kllm_gateway-0.3.0.tar.gz
Size 4.3 MB
Tags Source
SHA-256 checksum
How to use checksums
7e37bd5d6a35eb6a32554503c25de8aba3cd8d0a583fb6b1f2078cc5af665096
BLAKE2b-256 checksum
How to use checksums
88e216979f74931bc70f4d68c2702e1f6bb122ef558688898cd9fbd103174f57
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / kllm_gateway-0.3.0-py3-none-any.whl

Download URL kllm_gateway-0.3.0-py3-none-any.whl
Size 3.7 MB
Tags Python 3
SHA-256 checksum
How to use checksums
9fd134fa7fb3f18a1ac9326807eb9458c479f3f10f71f1ab3c6d45cf74223e42
BLAKE2b-256 checksum
How to use checksums
29d49af69eb85170e7650555c766e6694d2d94c7da97331be871d14d51a31f22
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for kllm-gateway 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for kllm_gateway-0.3.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page