Potato: The Portable Annotation Tool

Potato is a free, self-hosted annotation platform for NLP, Agentic, GenAI, and qualitative research. Annotate text, audio, video, images, documents, agent traces, and more — or run a full qualitative data analysis (QDA) workflow with a living codebook, memos, and cases. Configured entirely through YAML. No coding required.

Try the live demo on HuggingFace Spaces — no installation needed. More at www.potatoannotator.com.

Quick Start

pip install potato-annotation
# The examples/ folder ships with the source repo (see "run from source" below).
# After a PyPI install, clone the repo for the examples, or point `potato start`
# at your own config (see docs/quick-start.md).
potato start examples/classification/single-choice/config.yaml -p 8000

Or run from source (recommended to get the examples/):

git clone https://github.com/davidjurgens/potato.git
cd potato && pip install -r requirements.txt
python potato/flask_server.py start examples/classification/single-choice/config.yaml -p 8000

Open http://localhost:8000 and start annotating. Browse the examples/ directory for ready-to-use templates.

What Can You Annotate?

Potato covers traditional NLP labeling, AI agent evaluation, and interpretive qualitative analysis.

The tables below are a representative sample, not a complete list. Schemes and data types compose freely, custom layouts and raw HTML let you build interfaces beyond these, and new schema types can be added. If you don't see your task here, it's likely still possible.

Data Types

Annotation Schemes

Agent & LLM Evaluation

Potato reads traces from the major agent frameworks and evaluates them at the trajectory, step, span, or comparison level.

Trace Formats

Import traces from any major agent framework with the built-in converter:

python -m potato.trace_converter --input traces.json --input-format openai --output data.jsonl

Supported formats: OpenAI, Anthropic/Claude, ReAct, LangChain, LangFuse, WebArena, SWE-bench, OpenTelemetry, CrewAI/AutoGen/LangGraph, MCP, Aider, Claude Code, ATIF, SWE-Agent, and Web Agent. Auto-detection is available with --auto-detect.

Evaluation Levels

Web Agent Viewer

A viewer for GUI agent traces. You step through the screenshots one action at a time, with SVG overlays for clicks, bounding boxes, mouse paths, and scrolls. Annotators rate each step with inline controls, and a filmstrip bar jumps between steps.

Ready-to-Use Agent Examples

Qualitative Data Analysis (QDA)

Potato also supports interpretive qualitative research, the kind of work done in NVivo, ATLAS.ti, or MAXQDA, self-hosted and free.

Enable it with qda_mode, which turns these features on together; see the QDA Mode guide and the runnable qda-mode-example.

AI-Powered Annotation

LLM Label Suggestions

Connect any LLM provider to pre-annotate instances and suggest labels. Annotators then review and correct rather than labeling from scratch, which is the point: correcting a draft is faster than producing one.

Supported backends: OpenAI, Anthropic, Ollama, vLLM, Gemini, HuggingFace, OpenRouter

Active Learning

Potato reorders your annotation queue based on model uncertainty so annotators label the most informative instances first. Supports uncertainty sampling, BADGE, BALD, diversity, and hybrid strategies (docs).

Solo Mode

A human-LLM collaborative workflow where the system learns from annotator feedback and progressively transitions to autonomous LLM labeling as agreement improves (docs).

Chat Assistant

An LLM-powered sidebar where annotators can ask questions about difficult instances. It answers from your task description and annotation guidelines, and never writes a label itself (docs).

Quality Control & Workflows

Quality Assurance

Annotation Workflows

Continuous Evaluation Loop

Close the loop from production traces to graded, regression-gated evaluation:

Authentication & Deployment

Potato supports six authentication methods:

Passwords are hashed with per-user PBKDF2-SHA256 salts. Admins can reset passwords via CLI (potato reset-password) or REST API. Self-service token-based reset is also available.

Example Projects

Ready-to-use templates organized by type in examples/:

Live Demos on HuggingFace

Try Potato in your browser — no installation. A growing catalog of one-click demo Spaces covers classification, span/NER, agent-trace evaluation, multimodal, QDA, and more:

• Flagship demo — agent trace evaluation
• Full demo catalog & collection — every annotation type as a live Space
• Deploy your own — build_space.py + deploy_space.py from a single manifest

Research Showcase

The Potato Showcase contains annotation projects from published research — sentiment analysis, dialogue evaluation, summarization, and more.

Documentation

Potato has two complementary doc sites: potatoannotator.com/docs for guides, tutorials, and higher-level walkthroughs, and Read the Docs for the complete, version-matched technical reference (every config option, the full HTTP API, and internals). The links below point to the guide pages.

For coding agents

If you point Claude Code, Codex, or Cursor at Potato, give it these generated, machine-checkable specs rather than prose — they are built from the running code, so they cannot drift from it.

Every config in examples/ carries a # yaml-language-server: $schema=… modeline, so editors validate it live. See Machine-Readable Specs for editor setup, CI validation, and jq recipes.

Development

# Run tests
pytest tests/ -v

# By category
pytest tests/unit/ -v        # Unit tests (fast)
pytest tests/server/ -v      # Integration tests
pytest tests/selenium/ -v    # Browser tests

# With coverage
pytest --cov=potato --cov-report=html

See the Testing guide for which tier to write in, the test-file security rules, the annotation-persistence testing pattern, and the drift tests that keep the generated specs honest.

Support

• Issues: GitHub Issues
• Questions: jurgens@umich.edu
• Docs: potatoannotator.readthedocs.io

License

Potato is free software, licensed under the GNU General Public License v3.0 or later (GPLv3+). You are free to use, study, modify, and redistribute it — including for commercial purposes — provided that any distributed derivative works are also licensed under the GPLv3+ and made available with their source code. See the LICENSE file for the full terms.

Citation

If you use Potato in your research, please cite the Potato 2.0 paper (ACL 2026 System Demonstrations):

@inproceedings{jurgens-etal-2026-potato,
    title = "Potato 2.0: A Comprehensive Annotation Platform with {AI}-in-the-Loop Support",
    author = "Jurgens, David  and
      Chen, Michael  and
      Iyer, Lina",
    editor = "Durrett, Greg  and
      Jian, Ping",
    booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-demo.37/",
    pages = "374--386",
    ISBN = "979-8-89176-392-0",
}

To reference the original Potato release, cite the Potato 1.0 paper (EMNLP 2022 System Demonstrations):

@inproceedings{pei-etal-2022-potato,
    title = "{POTATO}: The Portable Text Annotation Tool",
    author = "Pei, Jiaxin  and
      Ananthasubramaniam, Aparna  and
      Wang, Xingyao  and
      Zhou, Naitian  and
      Dedeloudis, Apostolos  and
      Sargent, Jackson  and
      Jurgens, David",
    editor = "Che, Wanxiang  and
      Shutova, Ekaterina",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-demos.33/",
    doi = "10.18653/v1/2022.emnlp-demos.33",
    pages = "327--337",
}