snippet-checker

Check code snippets in anki or files via docker.

• snippet-checker
  • Install
  • How to check anki
    • Write config
    • Add tags
    • Run
  • How to check files
    • Structure your directory
    • Write config
    • Run
  • Bring your own images
    • Runner images
    • Formatter images
  • Examples
    • Hello world
    • Timing
    • Normalising exceptions
    • Normalising memory locations
    • Normalising errnos
    • Normalising hangs
  • Q&A
    • "The" output?
    • What to do when snippet-checker complains?
    • Which languages can it check?
    • Can I check snippets which use third-party packages?
    • How sandboxed?
    • What formatters does it use?
    • What's no_compress?

Install

For example:

uv tool install snippet-checker

How to check anki

Write config

In ~/.snippet-checker/ or $XDG_CONFIG_HOME/snippet-checker/ write snippet-checker.toml, e.g.

profile = "cosmo"  # Name of your anki profile, used to locate your collection.
# You can set `collection_path = "/path/to/collection"` instead of setting the profile if you like.
timeout = 10.0  # Seconds. The tool assumes any snippet that runs for longer than this is hanging. Default is no timeout.

# The [[notes]] blocks describe how to extract the code and output from your notes.
[[notes]]
note_type = "Code output"  # Must match anki exactly.

# Information about the field containing the code.
[notes.code_field]
name = "Code"
# Your field may contain markup, as well as the code.
# The pattern should be a Python regex with a group named "target", which matches just the code.
# The pattern below works for fields like '<pre><code class="lang-python">print(1 + 1)</code></pre>'.
# The markup is added back when the tool writes to anki.
pattern = '(?s)^<pre><code class="lang-\w+?">(?P<target>.*)</code></pre>$'

# Information about the field containing the output.
[notes.output_field]
name = "Output"
# This pattern works for fields like '<pre><samp>2\n</samp></pre>'.
pattern = "(?s)^<pre><samp>(?P<target>.*)</samp></pre>$"

# More [[notes]] blocks if needed, one per target note type.

Add tags

In anki:

• add a tag to the notes you want to check
  • e.g. check_me
• to check the outputs, add a tag snip:runner_image:<image tag> to the notes
  • e.g. snip:runner_image:my-python-runner
• to check formatting, add a tag snip:formatter_image:<image tag> to the notes
  • e.g. snip:formatter_image:my-python-formatter
• the images must satisfy a contract - see Bring your own images
• add other tags to customize how the tool treats them
  • snip:no_check_format to skip when checking formatting
  • snip:no_check_output to skip when checking outputs
  • snip:output_verbosity:0 or 1 or 2 (default is 0)
  • snip:no_compress to keep double blank lines in code
  • more details on these below

(Anki lets you batch edit tags: select the notes, right click, Notes > Add/Remove Tags.)

Run

Ensure Docker is running.

Check outputs:

snippet-checker --anki output check_me

Check formatting:

snippet-checker --anki format check_me

Pass --interactive to fix interactively. Pass --fix to auto-fix (back up your collection first).

How to check files

Structure your directory

Something like

your_dir
├── a_snippet
│   ├── main.py
│   └── output.txt
├── more_snippets
│   ├── extra_files_anywhere_are_ok
│   ├── a_go_snippet
│   │   ├── go.mod
│   │   ├── main.go
│   │   └── output.txt
│   ├── a_javascript_snippet
│   │   ├── main.js
│   │   └── output.txt
│   └── another_python_snippet
│       ├── main.py
│       └── output.txt

Write config

At your_dir's root write snippet_checker.toml, e.g.

# Set how tracebacks, panics etc. are abbreviated.
output_verbosity = 0  # Or 1 or 2. Default is 1.

# Set runner image tags if checking outputs (the snippets are executed using these)
# <file extension> = <tag>
[runner_images]
js = "my-javascript-runner"
py = "my-python-runner"
go = "my-go-runner"

# Set formatter image tags if checking formatting (the snippets are formatted using these)
# <file extension> = <tag>
[formatter_images]
js = "my-javascript-formatter"
py = "my-python-formatter"
go = "my-go-formatter"

Add other keys to customize how the tool treats snippets:

• timeout = 10.0 to assume any snippet that runs for longer than 10s is hanging (default is no timeout)
• check_output = false to skip when checking outputs
• check_format = false to skip when checking formatting
• output_verbosity = 0 or 1 or 2 (default is 1)
• compress = true to replace double blank lines by single

To override a setting for a particular snippet, add another snippet_checker.toml alongside it:

check_format = false

[runner_images]
go = "my-alternative-go-runner"  # e.g. a different Go version

Run

Ensure Docker is running.

Check outputs:

snippet-checker output your_dir

Check formatting:

snippet-checker format your_dir

Pass --interactive to fix interactively. Pass --fix to auto-fix (version control your collection first).

Bring your own images

You must create your own runner and formatter images. A hassle, yes. But it means you can check snippets in any language, at any version, with any dependencies, and can control runtime and formatting configuration.

Runner images

Contract:

• The image must have prepare.sh, run.sh scripts which can be executed via ./prepare.sh, ./run.sh.
• The tool copies the snippet into the image's working directory as main.
• prepare.sh does any setup, e.g. compilation, install dependencies.
• If it exits non-zero, its output is treated as the snippet's output.
• Else, run.sh executes the snippet, and its output is treated as the snippet's output.

For example, to run Go snippets you could create

my-go-runner
├── Dockerfile
└── prepare.sh
└── run.sh

where prepare.sh is

#!/bin/sh
set -e
mv main main.go
exec go build main.go

and run.sh is

#!/bin/sh
exec ./main

and the Dockerfile is

FROM golang:1.21
WORKDIR /tmp
COPY prepare.sh run.sh ./

(More examples in images/runners/ in the source.)

Then

chmod +x prepare.sh run.sh
docker image build -t my-go-runner .

For anki, tag the target notes snip:runner_image:my-go-runner, or, for files, add

[formatter_images]
py = "my-go-runner"

to the snippet_checker.toml.

Formatter images

Contract:

• The image must have a format.sh script
• which can be executed via ./format.sh
• and which reads ./input and writes the formatted version to ./output
• and which exits 0 just if there was no error when formatting (whether or not changes were made).

For example, to format Python snippets you could create

my-python-formatter
├── Dockerfile
└── format.sh

where format.sh is

#!/bin/sh
set -e
ruff format ./input
mv ./input ./output

and the Dockerfile is

FROM ghcr.io/astral-sh/ruff:0.16-alpine
WORKDIR /tmp
ENTRYPOINT [ "" ]
COPY format.sh .

(More examples in images/runners/ in the source.)

Then

chmod +x format.sh
docker image build -t my-python-formatter .

For anki, tag the target notes snip:formatter_image:my-python-formatter, or, for files, add

[formatter_images]
py = "my-python-formatter"

to the snippet_checker.toml.

Examples

snippet-checker checks your snippet's timed, normalised output (or, really, run.sh's).

Hello world

print("hello world")

hello world

Trailing newline included.

Timing

from threading import Thread
from time import sleep


def io_bound():
    sleep(3)
    print("done")


thread1 = Thread(target=io_bound)
thread2 = Thread(target=io_bound)
thread1.start()
thread2.start()
print("here")

here
<~3s>
done
done

Timing matters, so it's included in the output. Gaps are rounded to the nearest second, are only included if at least 1s after rounding, and are included in the form "<~Xs>".

Normalising exceptions

1 / 0

Output verbosity 0:

ZeroDivisionError: division by zero

Output verbosity 1

Traceback (most recent call last):
  ...
ZeroDivisionError: division by zero

Output verbosity 2:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    1 / 0
    ~~^~~
ZeroDivisionError: division by zero

Similar for exceptions in other languages.

Normalising memory locations

class C:
    pass


class D:
    pass


c = C()
d = D()
print(c)
print(d)
print(c)

<__main__.C object at 0x100>
<__main__.D object at 0x200>
<__main__.C object at 0x100>

Memory addresses vary from run to run. The tool replaces them by consistent, simpler addresses.

Normalising errnos

with open("does_not_exist") as f:
  print(f.read())

FileNotFoundError: [Errno NN] No such file or directory: 'does_not_exist'

Errnos vary across platforms. The tool replaces them all with a placeholder.

Normalising hangs

import socket

srv = socket.create_server(("127.0.0.1", 65432))
print("socket created")
srv.accept()

socket created
...

For a blocking socket (the default), accept() blocks until a connection is available. So this snippet hangs. The tool assumes a snippet is hanging if it runs for more than timeout seconds. It kills it, appends ... on its own line to any output so far, and returns that as the output.

Q&A

"The" output?

A snippet's output is not determined by its code.

We saw examples above: memory addresses vary across runs; errnos vary across platforms.

There are plenty more (examples are Python-centric but similar is true of all languages):

• print(os.environ["PWD"])
• print(random.random())
• socket.bind(('127.0.0.1', 65432)) errors if address is already in use
• open('foo') errors if no such file
• print("foo\rbar") shows as bar if printing to a terminal but not if printing to a file
• timing depends on machine, contention, ...
• how you invoke the runtime (e.g. python -c 'contents of main.py' versus python main.py)
• which compile or runtime options you set (e.g. python -u, python -Wignore)
• and so on

Some variation you can pin down via your runner image. But maybe not all, in which case the tool can only tell you an output, not the output.

What to do when snippet-checker complains?

If you agree, then it's done its job and you can update the snippet or output.

If you disagree, then you have options:

1. adapt your image so the output snippet-checker generates matches what you expect
2. open an issue to adapt snippet-checker to handle your snippet
3. adapt your snippet to something snippet-checker can handle
4. tag your snippet so snippet-checker ignores it

Some examples.

snippet-checker can't handle

# Assume my_file.txt is "first\nsecond\nthird\n".
with open("my_file.txt") as f:
    for x in f:
        print(x)

because it doesn't understand the comment.

But we can adapt the snippet to something it can handle.

with open("my_file.txt", "w") as f:
    f.write("first\nsecond\nthird\n"

with open("my_file.txt") as f:
    for x in f:
        print(x)

Similarly, it can't handle

print(({char for char in "a0b2b3" if char}))

because set order is non-deterministic.

We could adapt it

print(sorted({char for char in "a0b2b3" if char}))

or maybe the underlying points would be better captured differently.

It can't handle this either

try:
    x = input()
    # user enters Ctrl-C
except Exception:
    print("exception")
finally:
    print("finally")

and I don't see how to adapt the snippet or the tool. Better just tag it so snippet-checker ignores it.

Or a formatting example:

print("foo" "bar")

ruff formats this as print("foobar"). But if the point of the question is to show implicit string concatenation, again better to tag it so snippet-checker ignores it.

Which languages can it check?

Any, because Bring your own images.

However, the tool does output normalisation itself, so may normalise a lot (Python), or a little (Go, Ruby, Rust, Node), or not at all (everything else).

If you want more/different normalisations, open a PR :)

Can I check snippets which use third-party packages?

Yes, because Bring your own images. Just write a runner image meeting the contract.

For example, to test numpy snippets create an image like

FROM python:3.13
WORKDIR /tmp
ENV NO_COLOR=true PYTHONWARNINGS=ignore
COPY prepare.sh run.sh ./
RUN <<EOF
python -m venv numpy_env
. numpy_env/bin/activate
python -m pip install --no-cache-dir numpy==2.5
EOF

where prepare.sh is

#!/bin/sh
mv main main.py

and run.sh is

#!/bin/sh
. numpy_env/bin/activate
exec python main.py

How sandboxed?

The snippets run in Docker containers. No mounts or volumes.

Don't point the tool at arbitrary code. The sandboxing protects against accidents, not attacks.

What formatters does it use?

Any you like, because Bring your own images.

What's up with anki unicode normalization?

Anki normalizes strings to NFC before writing them to the database, by default. So if your snippet is print("a\N{COMBINING TILDE}"), the tool will keep saying that the output in the anki note is wrong. You can either:

• stop anki NFCing your strings by running mw.col.conf["normalize_note_text"] = False in the debug console
• rewrite your note to avoid the issue
• tag it snip:no_check_output

What's no_compress?

Some formatters like double blank lines, e.g. between class definitions. But space is at a premium in anki notes. So by default when formatting anki double blanks are replaced by single. To keep doubles add a snip:no_compress tag.