karotte
karotte runs LLM agents on tasks and scores the results.
You write an environment: a Python project with one or more tasks. A task is a
list of steps. Each step gives the agent instructions and a judge that decides
whether the agent succeeded. karotte builds the environment into a container
image, lets the model work inside it through tools like bash, and records every
message, tool call and score in a transcript.
The agent runs as an unprivileged user with its own resource limits, a firewall, and a disk quota, so a task can hand it a real shell without trusting it. The model under test is called the student.
Documentation: karotte.dev
Install
karotte needs Python 3.12+ and uv. Runs go into a
VM by default: Apple container on macOS, Firecracker on Linux. docker or
podman work too. See Installation
and Runtimes for what each needs.
uv tool install karotte
Quick start
Create an environment from the default template and run its example task:
karotte create-env my_env
cd my_env
uv sync --extra dev
uv run setup_data.py
uv run karotte create-run-config --model anthropic/claude-fable-5
export ANTHROPIC_API_KEY=...
uv run karotte run --config run_config.json
karotte run builds the image, runs the task, and writes the transcript to
out/transcript.json. karotte dashboard out/ shows it. The
quick start explains each
step, and Writing tasks shows how
to add your own.
Development
just lint
just test
just test-template default
Every merge to main is released. See
Developing karotte.
License
karotte is under the MIT license. The templates in
src/karotte/templates/ are under MIT No Attribution
(MIT-0), so environments created from them need no license notice.
Third-party software in built images
Images built from the templates contain third-party software under its own
licenses. They are based on Amazon Linux 2023. The language-toolchains
template installs GPL and LGPL software (gcc, GnuCOBOL, GNU Prolog, Free Pascal,
the libraries bundled with Julia, and others), Amazon Corretto (GPL-2.0 with the
Classpath Exception) and Clojure (EPL-1.0) for the languages you enable. Mojo,
off by default, has its own license terms; check them for the version you
enable. If you distribute a built image, you're responsible for complying with
the licenses of the software in it.
1 maintainer
Metadata
1 maintainer
Release files for karotte 3.0.36
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| karotte-3.0.36.tar.gz | 385.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| karotte-3.0.36-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 863.4 kB
Release files / karotte-3.0.36.tar.gz
| Download URL | karotte-3.0.36.tar.gz |
|---|---|
| Size | 385.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f28c37231638bda761a4e09ef7f9b49ccf71abf0bf337e3ed9b4781b27f3f3f9
|
|
BLAKE2b-256 checksum How to use checksums |
55d4b72b41f4f24d63ca2830ae97f9b6043d8ba335e71b1a24c9814c22086e72
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / karotte-3.0.36-py3-none-any.whl
| Download URL | karotte-3.0.36-py3-none-any.whl |
|---|---|
| Size | 478.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cf28cce86f8bcab97a0fd5fa7878740fdb2dc8de0552d9f078979bc532944c58
|
|
BLAKE2b-256 checksum How to use checksums |
587f116f8caf87003ec048541a3dd33982f3194103992d594a15cd11a18e468f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
