Skip to main content

leeuwenhoek

A debugging microscope that lives inside your Flask app. Leeuwenhoek records what happens during each request, CLI command and background job (the request itself, SQL queries, exceptions, log lines, ad-hoc dumps, outgoing HTTP calls, cache and Redis operations, mail, ORM changes and more), groups it all by batch, and shows it in a panel at /leeuwenhoek.

  • No separate service. It is a Blueprint plus a few hooks. The panel is rendered on the server with Jinja and Bulma, with no frontend build and no CDN.
  • No changes to your code. Everything is captured through Flask hooks, signals, SQLAlchemy events and a logging handler.
  • Grouped by batch. Every entry produced while handling one request shares a batch_id, so you can open a request and see every query, log line and exception it produced.
  • Its own database. Entries go to a separate SQLAlchemy engine (SQLite by default, Postgres in production), so they never mix with your app's data or record themselves.
  • Safe enough for staging. Sensitive values are redacted, payloads are truncated, and writes happen in a background thread with a bounded queue. Sampling, an only-errors mode and access control are built in.

Status: early (v0.1, not on PyPI yet). See the roadmap for what has shipped and what's coming.

Installation

Requires Python 3.10+, Flask 2.3+ and SQLAlchemy 2.x.

uv add leeuwenhoek
# or
pip install leeuwenhoek

Add the login extra (leeuwenhoek[login]) to record the current Flask-Login user on each request. If Flask-Login is already installed, this happens automatically.

Add the cache extra (leeuwenhoek[cache]) to record Flask-Caching operations. As with Flask-Login, the cache watcher turns on by itself when Flask-Caching is installed.

Add the mail extra (leeuwenhoek[mail]) to record emails sent with Flask-Mail. As with every integration watcher, it turns on by itself when Flask-Mail is installed.

Add the redis extra (leeuwenhoek[redis]) to record the commands sent with redis-py. The Redis watcher turns on by itself when redis is installed.

Add the celery extra (leeuwenhoek[celery]) to record Celery tasks, see Celery jobs. The watcher turns on by itself when Celery is installed.

Add the http extra to install requests and HTTPX for the outgoing HTTP watcher. Either one is enough, and the watcher turns on by itself if your app already uses one of them.

Add the sched extra (leeuwenhoek[sched]) to record APScheduler job runs. Only APScheduler 3.x is supported (apscheduler>=3.10,<4); version 4 has a different API. The watcher turns on by itself when APScheduler is installed.

Add the rq extra (leeuwenhoek[rq]) to record RQ jobs (RQ 1.15 or newer). The watcher turns on by itself when RQ is installed.

Quickstart

from flask import Flask
from leeuwenhoek import Leeuwenhoek

app = Flask(__name__)
Leeuwenhoek(app)

Or with the application factory pattern:

lw = Leeuwenhoek()


def create_app():
    app = Flask(__name__)
    lw.init_app(app, engine=db.engine)  # engine: only record queries from this engine
    return app

Run the app in debug mode (flask run --debug), make a few requests, then open http://localhost:5000/leeuwenhoek.

By default Leeuwenhoek is enabled only when app.debug is True, and the panel is only reachable in debug mode. See Access control to use it anywhere else.

Try the demo

git clone https://github.com/Manuel-Espinosa/leeuwenhoek
cd leeuwenhoek
uv sync
uv run flask --app examples/demo run --debug

Visit /users, /boom and /dump, then open /leeuwenhoek.

What it records

Watcher Captures
request Method, URL, URL rule, endpoint, blueprint, view args, IP, headers, query string, body, session, user, status, response headers, response body (JSON/text only), duration, and optionally peak memory
query SQL, parameters, duration, and the file and line in your code that ran the query. Queries slower than LEEUWENHOEK_SLOW_QUERY_MS are flagged as slow, and queries repeated many times in one request as N+1.
exception Class, message, full traceback with a code snippet for each frame. Repeated exceptions are grouped into families (same class, file and line). Also captures exceptions turned into 5xx responses by your own error handlers, and uncaught exceptions in scripts, CLI commands and threads (through sys.excepthook and threading.excepthook, which are chained and restored by lw.close()).
log Level, message, logger name, extra fields and exc_info, from a handler on the root logger
dump Whatever you pass to dump(), with the file and line it was called from
mail Emails sent with Flask-Mail (its email_dispatched signal, which also fires when sending is suppressed with MAIL_SUPPRESS_SEND or TESTING): sender, to/cc/bcc, reply-to, subject, extra headers (redacted with LEEUWENHOEK_HIDDEN_HEADERS), HTML and text bodies (truncated to LEEUWENHOEK_MAX_BODY_KB) and attachment names and types, never their content. The panel previews the HTML body in a fully sandboxed <iframe> (no scripts, no same-origin). Needs the mail extra.
template Each Jinja template rendered with render_template, render_template_string or stream_template: name, file, context keys (never their values, and without Flask's own g, request and session) and render duration. Templates pulled in with {% extends %} or {% include %} are part of the top-level render, not separate entries. Leeuwenhoek's own templates are never recorded.
command flask CLI commands of app.cli: full name (db seed for subcommands), arguments, options, exit code and duration. Each command gets its own batch, see CLI commands.
job Celery task runs: task name and id, args and kwargs, queue, attempt, state (SUCCESS, FAILURE, RETRY), duration, result, exception and retry reason. Each run gets its own batch, linked to the request that dispatched it, see Celery jobs.
http Outgoing HTTP calls made with requests or HTTPX (sync and async clients): method, URL, request headers, payload, status, response headers, response body (JSON/text only), duration, redirects, and the error if the call raised. Enabled when either library is installed.
cache Flask-Caching operations (get, get_many, get_dict, has, set, add, set_many, delete, delete_many, unlink, clear), including the ones made by @cache.cached and @cache.memoize (with Flask-Caching 2.5 or newer; older versions, the last ones for Python 3.10, make the decorators talk to the backend directly and only the direct calls are recorded): keys, hit or miss for reads, TTL for writes, backend and duration. Values are stored only with LEEUWENHOEK_CACHE_VALUES. The Cache card of a request shows its hit and miss counts.
model Inserts, updates and deletes made through the SQLAlchemy ORM, in any Session (Flask-SQLAlchemy's or your own): model, table, action (created, updated, deleted), primary key and the changed columns with their old and new values. Recorded when the transaction commits, see ORM models.
event Blinker signals: the Flask signals in LEEUWENHOEK_SIGNALS (by default message_flashed) and your own signals added with lw.watch_signal(): signal name, sender (class name or truncated repr) and keyword arguments (redacted with LEEUWENHOEK_HIDDEN_PARAMS, truncated to LEEUWENHOEK_MAX_BODY_KB), tagged event:<name>. See Signals.
redis Commands sent with redis-py (redis.Redis and redis.asyncio.Redis): command, arguments (redacted and truncated), server, database, duration and the error if the command raised. A pipeline is one entry with its list of commands. See Redis.
schedule Each run of an APScheduler 3.x job: id, name, function, trigger, executor, scheduled and next run time, arguments (redacted with LEEUWENHOEK_HIDDEN_PARAMS), result (repr, truncated) or exception, and duration; also runs missed past their misfire grace time and runs skipped because the job reached max_instances. Each run gets its own batch, see Scheduled jobs. Needs the sched extra.
rq RQ jobs: function path, queue, job id, status, arguments and keyword arguments (redacted with LEEUWENHOEK_HIDDEN_PARAMS, truncated to LEEUWENHOEK_MAX_BODY_KB), duration and exception. Each execution gets its own batch, linked to the request that enqueued the job, see RQ jobs. Needs the rq extra.

By default every watcher whose library is installed is enabled, so integration watchers (mail, Celery, Redis...) turn on by themselves when you install the package they observe. Set LEEUWENHOEK_WATCHERS to pick them explicitly, e.g. LEEUWENHOEK_WATCHERS=request,exception. Only enabled watchers appear in the panel menu.

Requests automatically get the tags endpoint:<name>, status:<code> and user:<id>. Commands get command:<name> and exit_code:<code>, Celery tasks task:<name> and state:<STATE>, RQ entries rq_job:<id>, rq_queue:<name> and rq_status:<status>, scheduled job runs job:<id> and job_status:<success|error|missed|skipped>. Log entries get level:<LEVEL>, templates template:<name> and ORM changes model:<Name>. Slow queries get slow, show a yellow slow badge in the panel, and the Queries card of a request shows how many of its queries were slow. Repeated queries get n+1 (see N+1 queries).

N+1 queries

When a batch closes, its queries are grouped by their normalized SQL: string and number literals and bound parameters (?, %s, :name, %(name)s, $1) become ?, lists such as IN (1, 2, 3) collapse to IN (?), whitespace is collapsed and keywords are upper-cased. So select * from users where id = 1 and SELECT * FROM users WHERE id = 2 are the same statement.

Every query in a group of at least LEEUWENHOEK_N_PLUS_ONE_THRESHOLD (default 5) gets content["n_plus_one"] = True and the n+1 tag, and shows a red n+1 badge in the panel. The Queries card of the request lists each repeated statement with its count, for example SELECT * FROM users WHERE id = ? × 12. Set the threshold to 0 to turn detection off.

The detection runs as an on_finish callback, before the batch is saved, so filters and taggers already see the flag and the tag (lw.filter(lambda e: not e.content.get("n_plus_one")) works, for example). Queries a filter drops afterwards were still counted.

Outgoing HTTP

Every call made through requests or httpx (including httpx.AsyncClient) is recorded, whichever client or session made it, and appears in the HTTP client section and as a card in the detail of the request that made it. Calls made outside a request are stored as standalone entries. The URL's query parameters and the payload and JSON response keys listed in LEEUWENHOEK_HIDDEN_PARAMS are masked, as are the headers in LEEUWENHOEK_HIDDEN_HEADERS; bodies are cut at LEEUWENHOEK_MAX_BODY_KB. Only JSON and text response bodies are kept, and never for streamed responses (stream=True), which are left unread. A requests redirect chain is one entry with the final URL and the number of redirects. Entries are tagged host:<hostname>, and calls that raised are tagged failed.

ORM models

The model watcher listens to after_flush on SQLAlchemy's Session class, so every session is covered without configuration. For each object a flush inserts, updates or deletes it records the model class, the table, the action, the primary key (also for new rows, whose key the database just assigned) and the changes, as {"column": {"old": ..., "new": ...}}:

  • created: every loaded column, from null.
  • updated: only the columns that really changed (an object in session.dirty whose values are the same is skipped). An old value that was never loaded is null.
  • deleted: every loaded column, to null.

Only columns are recorded; relationships are skipped, since their foreign key columns already show the link. Values are made JSON-safe (datetimes as ISO strings, Decimal as a string, binary data as <N bytes>), columns named in LEEUWENHOEK_HIDDEN_PARAMS are masked, and each value is cut at LEEUWENHOEK_MAX_BODY_KB. Nothing is ever loaded from the database to fill a value.

Changes are captured at flush time, while their history still exists, but only recorded when the transaction commits. Changes flushed and then rolled back (including a rolled-back begin_nested() savepoint) or left uncommitted when the session closes are dropped, so the panel only shows what reached the database. A commit made after the request ended is stored as standalone. Leeuwenhoek's own storage (its engine and its tables) is never recorded.

Signals

Watch your own blinker signals with lw.watch_signal(), before or after init_app. Every send is recorded as an event entry in the Events section and in the batch of the request, command or job that sent it, or as a standalone entry outside one. Watching a signal twice does nothing.

from blinker import signal

order_placed = signal("order-placed")
lw.watch_signal(order_placed)  # or lw.watch_signal("order-placed")
lw.watch_signal(anonymous_signal, name="cart-emptied")  # a name for a Signal() without one


@app.post("/orders")
def create_order():
    order_placed.send(current_user, order_id=7, total=99.5)

Flask's own signals are picked by name with LEEUWENHOEK_SIGNALS (message_flashed, request_started, request_finished, request_tearing_down, appcontext_pushed, appcontext_popped, appcontext_tearing_down, template_rendered, before_render_template, got_request_exception), and only the ones the app sends are recorded. The default is just message_flashed: the request and app context signals fire on every request and repeat what the request entry already shows. request_started fires before the request's batch opens and request_tearing_down and the app context ones after it closes, so they are stored as standalone entries.

A signal that another active watcher already records is never recorded twice: template_rendered and before_render_template while the template watcher is on, got_request_exception while the exception watcher is on, and Flask-Mail's email_dispatched while the mail watcher is on. Signals sent while the panel is served, for ignored paths, while recording is paused, or by Leeuwenhoek itself are never recorded.

Redis

Every command sent through redis.Redis or redis.asyncio.Redis (and their subclasses, such as fakeredis) is recorded, whichever client sent it, and appears in the Redis section and as a card, with the count and total time, in the detail of the request that sent it. Commands sent outside a request are standalone entries. A pipeline (r.pipeline(), with or without a transaction) is one PIPELINE entry listing its commands, the error of each one that failed with raise_on_error=False, and the time of the whole round trip; its queued commands are not recorded again. Commands that a pipeline in WATCH mode sends right away are recorded on their own. Return values are never stored.

Arguments are decoded as UTF-8 (binary values are shown as [binary: N bytes]); at most 20 are kept per command, each cut at 200 characters, and at most 50 commands per pipeline. Passwords are never stored: the arguments of AUTH, the password of HELLO ... AUTH and MIGRATE ... AUTH/AUTH2, CONFIG SET requirepass/masterauth and the password rules of ACL SETUSER are replaced by ********. When a key contains a name in LEEUWENHOEK_HIDDEN_PARAMS (user:1:token), every argument after it is masked, and a hash field or argument that contains one (HSET user:1 password ...) masks the value that follows it. Commands that raised are tagged failed.

Libraries that use Redis are recorded at the command level too. With Flask-Caching's RedisCache, a cache.get() gives a cache entry and the GET it sent as a redis entry, in the same batch. RQ's commands are recorded as well: those sent while enqueueing join the request, and those a worker sends between jobs (polling, heartbeats) are standalone; drop them with lw.filter() or leave redis out of LEEUWENHOEK_WATCHERS if they are noise. Redis Cluster clients (RedisCluster) and Pub/Sub connections are not covered.

dump()

from leeuwenhoek import dump


@app.get("/checkout")
def checkout():
    dump(cart)  # shows up in the Dumps section and in this request's batch
    return "ok"

CLI commands

Every command you add to app.cli (with @app.cli.command(), a nested group or a blueprint's cli) gets its own batch, so the queries, logs and exceptions it produces are grouped under it, as they are under a request. Commands registered after init_app are recorded too.

@app.cli.command("seed")
@click.argument("count", type=int)
@click.option("--token")
def seed(count, token):
    app.logger.info("seeding %s users", count)
    db.session.execute(...)

flask seed 10 --token=abc records a command entry with {"count": 10} as arguments, {"token": "********"} as options (values are redacted with LEEUWENHOEK_HIDDEN_PARAMS), the exit code and the duration, plus the log line and the query, all in one batch. The exit code is 0, the code given to sys.exit() or ctx.exit(), the code of a click.ClickException (2 for a usage error), or 1 when the command raises. An exception raised by the command is recorded in its batch, and the batch is saved even though the command failed.

Like requests, commands are only recorded while Leeuwenhoek is enabled: run them with flask --debug ... or set LEEUWENHOEK_ENABLED. Skip some with LEEUWENHOEK_IGNORE_COMMANDS: an entry matches a command by its full name, by a group prefix ("db" skips db upgrade) or as a glob ("seed-*"). Nothing that happens inside an ignored command is recorded. flask leeuwenhoek ..., flask run and flask shell never get a command entry or a batch: each request served by flask run gets its own batch as usual, and a shell session, which can last for hours and isn't one unit of work, stores what you run in it as standalone entries.

Scheduled jobs

Every run of an APScheduler 3.x job gets its own batch, with a schedule entry as its root, so the queries, logs and exceptions of one run are grouped under it, as they are under a request. It works with every scheduler (BackgroundScheduler, BlockingScheduler, AsyncIOScheduler...), every job store and coroutine jobs.

scheduler = BackgroundScheduler()


@scheduler.scheduled_job("interval", minutes=5, id="sync")
def sync_users():
    with app.app_context():  # needed to use Flask-SQLAlchemy and friends
        app.logger.info("syncing users")
        db.session.execute(...)

Leeuwenhoek never pushes an app context for a job: a job that needs one must push it itself, as it would without Leeuwenhoek, so the job behaves the same whether it is installed or not. Recording needs no app context.

A run records the job id and name, the function, the trigger (interval[0:05:00], cron[...]), the executor, the time it was scheduled for and the next run time, the arguments, its duration, and its repr()'d result (cut at 1000 characters) or the exception it raised, which is also recorded in the batch by the exception watcher (as handled, since APScheduler catches it). Runs missed by more than the job's misfire_grace_time and runs skipped because the job already had max_instances running are recorded as missed and skipped entries, without a batch of their own. A run started while a batch is current in the scheduler's thread (a flask command running a BlockingScheduler, for example) keeps that batch as its parent.

APScheduler logs Running job ... before a run and Job ... raised an exception after it, outside the run, so the log watcher stores these lines as standalone entries.

With the ProcessPoolExecutor, the job runs in another process and is not recorded: Leeuwenhoek's wrapper does not cross the process boundary, and the child process gets the original job. Use a thread pool (the default) or record from inside the job. Celery beat is not part of this watcher: scheduled Celery tasks are recorded by the Celery watcher.

RQ jobs

Jobs are recorded twice, as RQ sees them:

  • The dispatch: queue.enqueue(...) (or enqueue_call, or enqueue_job with a job built by Job.create) records an rq entry with event: "enqueue" in the current batch, so a request's detail lists the jobs it enqueued, and stores the batch id in job.meta["leeuwenhoek_parent_batch_id"]. Its status is queued, deferred for a job waiting on a dependency, or the outcome for a synchronous queue. Outside a batch (a script), it is stored as standalone.
  • Each execution: the worker runs the job in a new batch (kind="job") whose parent_batch_id is the dispatching batch. Its root entry (event: "perform") has the status finished or failed, the duration and, on failure, the exception class and message. Queries, logs, HTTP calls and the exception raised by the job join this batch, and the job's detail links back to the request. A retried job gets one batch per attempt.

The rq_job:<id> tag ties the dispatch and every execution of one job together: the job detail links to them.

queue = Queue(connection=redis)


@app.post("/reports")
def create_report():
    # Recorded kwargs: {"report_id": 7, "token": "********"}
    queue.enqueue(build_report, report_id=7, token="abc")
    return "", 202

The worker must create the app (and so call Leeuwenhoek.init_app) before it starts, for example with a small script that builds the app and then runs Worker([queue], connection=redis).work(). Forking workers (the default Worker), SimpleWorker and Queue(is_async=False) are all covered. A forked work horse ends with os._exit(), which skips the atexit flush, so the watcher writes the job's batch before handing control back to RQ (waiting up to LEEUWENHOEK_FLUSH_TIMEOUT). The worker's own work between jobs (polling, heartbeats, registry cleanup, its log lines) is never recorded. A work horse killed by RQ (SIGKILL after a hard timeout) records nothing. Jobs that RQ moves on its own (the scheduler enqueuing a job scheduled with enqueue_at, a dependency becoming ready, a retry) are not recorded again as dispatches, only when they run.

args and kwargs hold the job's original arguments when args_redacted is false: nothing was masked, truncated or turned into a string to fit in JSON. When it is true, they are only for display. job.description is not stored, because it repeats the arguments unredacted.

Celery jobs

Pass your Celery app to init_app (or to the constructor):

celery = Celery(__name__, broker="redis://localhost:6379/0")
lw = Leeuwenhoek()
lw.init_app(app, celery=celery)

Every run of one of its tasks gets its own batch (kind="job"), so the queries, logs and exceptions of the task are grouped under it, and the batch page shows the job on top. Without celery=, the tasks of every Celery app in the process are recorded (Celery's signals are global). The worker must create the Flask app and call init_app too (the usual Flask + Celery setup does), and Leeuwenhoek must be enabled there (LEEUWENHOEK_ENABLED).

When a task is dispatched while a request (or a command, or another job) is being recorded, the id of that batch travels in the message headers, and the job's entries get it as parent_batch_id. The dispatching batch also gets a job entry with the status dispatched, shown in the Jobs card of the request detail with a View job link to the job's batch; the job's page links back with Dispatched by. A retry (self.retry()) is a new run with its own batch and the same parent. The Jobs listing shows the runs only.

In eager mode (task_always_eager, or task.apply()), Celery doesn't publish a message, so there are no headers: the task runs inside the batch that dispatched it, and that batch becomes its parent. Tasks dispatched outside any batch (a script, celery beat) get a job batch without a parent.

Args and kwargs are redacted with LEEUWENHOEK_HIDDEN_PARAMS (by key, at any depth) and cut at LEEUWENHOEK_MAX_BODY_KB, like the result. content["args_redacted"] is false only when the stored args and kwargs are exactly the ones the task received (nothing masked, truncated or unserializable), so the job could be dispatched again with them.

Peak memory

Set LEEUWENHOEK_TRACK_MEMORY=true to record how much memory each request allocated at its peak. It is shown as Peak memory in the request detail and stored as memory_peak_kb in the entry.

It uses tracemalloc, which is off by default because it slows down every allocation while it runs (about 1 ms more per request in the benchmark). Leeuwenhoek starts tracing when a recorded request begins and stops it when none is running. If tracing was already started by something else (python -X tracemalloc, a profiler), it is left running, and only memory allocated during the request counts.

tracemalloc measures the whole process, not one thread. With concurrent requests (a threaded server, gevent), each peak includes the other requests in flight, so treat the numbers as approximate there. They are exact when requests are handled one at a time.

The panel

URL Shows
/leeuwenhoek/requests Requests (also /queries, /exceptions, /logs, /dumps, /commands, /http, /cache, /models, /events, /redis, /schedule, /rq, /jobs), newest first, with search (?q=) and tag filter (?tag=); /exceptions?resolved=1 includes resolved families
/leeuwenhoek/batches One row per batch
/leeuwenhoek/<type>/<uuid> One entry, plus everything else from its batch grouped by type
/leeuwenhoek/batch/<batch_id> A whole batch, with its request, command or scheduled job run, RQ job or Celery task on top
/leeuwenhoek/exceptions/family/<hash> Every occurrence of one exception, with a Resolve / Reopen button
POST /leeuwenhoek/exceptions/family/<hash>/resolve (/reopen) Marks the family as resolved (or open again); needs the CSRF token
POST /leeuwenhoek/jobs/<uuid>/retry (/rq/<uuid>/retry) Dispatches a failed Celery task (or RQ job) again with its original arguments; needs the CSRF token, see Retrying failed jobs
/leeuwenhoek/<type>/rows?after=<id> HTML <tr> rows of the entries newer than <id>, with the same filters as the listing (used by live polling)
/leeuwenhoek/api/status JSON: recording state and counts per type

Listings update live: every LEEUWENHOEK_POLL_SECONDS the panel fetches the new rows and puts them at the top of the table, and refreshes the counters in the menu. Polling pauses while the tab is hidden and doesn't add rows on an older page (?before=).

The navbar has buttons to pause or resume recording and to clear all entries. They, like the Resolve / Reopen buttons below, use POST with the panel's own CSRF token. Apps using Flask-WTF don't need any changes, because the panel is exempted from Flask-WTF's CSRF check.

Resolving exceptions

Once a bug is fixed, mark its exception family as resolved with the Resolve button on the Exceptions listing, an exception's detail page or its family page. Resolved families are hidden from the Exceptions listing (and from its live polling); Show resolved (/leeuwenhoek/exceptions?resolved=1) lists them again, marked as resolved and with a Reopen button.

A new occurrence after the resolve time reopens the family by itself: it shows up in the listing again with all its occurrences. The state lives in the settings table (resolved:<family_hash> with the resolve time), so every instance sharing the database sees it.

Retrying failed jobs

The detail page of a failed run, a Celery task with the state FAILURE or an RQ execution with the status failed, has a Retry button that dispatches the job again with the arguments it received. Dispatch entries and successful runs don't get one. Like the other panel buttons, it uses POST with the panel's CSRF token and follows the panel's access control.

The button is disabled, with the reason next to it, when the run can't be dispatched again:

  • The arguments were not kept exactly (args_redacted is true): something was masked with LEEUWENHOEK_HIDDEN_PARAMS, cut at LEEUWENHOEK_MAX_BODY_KB or was not plain JSON. Leeuwenhoek never dispatches a job with the arguments it only shows.
  • Retrying is turned off: LEEUWENHOEK_ALLOW_RETRY defaults to app.debug. Re-running a job is a side effect (a second email, a second charge), so outside debug mode it stays off until you set LEEUWENHOEK_ALLOW_RETRY=true.
  • RQ only: there is no Redis connection to enqueue on. Leeuwenhoek doesn't store the original job's connection, so give it one: lw.init_app(app, rq_connection=redis) or LEEUWENHOEK_RQ_REDIS_URL=redis://localhost:6379/0.

A POST that is refused gets the detail page back with the reason: 409 when the run can't be retried, 403 when retrying is turned off, 502 when the broker or Redis rejects the dispatch. On success the panel goes back to the failed run with a notice, the id of the new task or job and a link to its run.

How the job is dispatched again:

  • Celery: on the app passed as celery= (or celery.current_app), to the queue the run was recorded on, with a leeuwenhoek_retry_of message header. When the task is registered in the panel's process it goes through task.apply_async(), so task_always_eager and the task's options apply; otherwise it is published by name with send_task(), for tasks defined only on the workers.
  • RQ: function is enqueued by its dotted path on Queue(<queue>, connection=...), as a new job with its own id, and job.meta["leeuwenhoek_retry_of"] set.

Either way, the new run stores the uuid of the failed entry as content["retry_of"] and gets the tag retry_of:<uuid>: its detail page links back with Retry of, and the failed run's Retries of this run link lists every retry. The panel's own requests are not recorded, so there is no dispatch entry and the new run has no parent batch.

Configuration

Each setting is read from app.config["LEEUWENHOEK_<KEY>"] first, then from an environment variable with the same name. In environment variables, booleans accept 1/true/yes/on, lists accept JSON or comma-separated values, and BASIC_AUTH uses the form user:password.

Key Default Description
LEEUWENHOEK_ENABLED app.debug Master switch
LEEUWENHOEK_URL_PREFIX /leeuwenhoek Where the panel is mounted. Requests under it are never recorded.
LEEUWENHOEK_WATCHERS every watcher whose library is installed Watchers to enable. A listed watcher whose library is missing is skipped with a warning.
LEEUWENHOEK_IGNORE_PATHS ["/static*", "/favicon.ico", "/health*"] Glob patterns of paths that are not recorded
LEEUWENHOEK_IGNORE_COMMANDS [] CLI commands that are not recorded: full names, group prefixes or glob patterns, see CLI commands
LEEUWENHOEK_HIDDEN_HEADERS ["authorization", "cookie", "x-api-key"] Headers replaced by ********
LEEUWENHOEK_HIDDEN_PARAMS ["password", "password_confirmation", "token", "secret"] Keys replaced by ******** in bodies, query strings and sessions, at any depth
LEEUWENHOEK_MAX_BODY_KB 64 Bodies larger than this are truncated
LEEUWENHOEK_MAX_ENTRIES_PER_BATCH 500 Maximum entries per batch. The request entry itself is always kept.
LEEUWENHOEK_SLOW_QUERY_MS 100 Queries that take longer than this many milliseconds are flagged as slow and tagged slow
LEEUWENHOEK_CACHE_VALUES False Store the values the cache watcher reads and writes, truncated to LEEUWENHOEK_MAX_BODY_KB. Fields named in LEEUWENHOEK_HIDDEN_PARAMS are redacted inside values, and the whole value is replaced by ******** when its key contains one of those names (user:1:token).
LEEUWENHOEK_N_PLUS_ONE_THRESHOLD 5 Queries with the same normalized SQL repeated this many times in a batch are tagged n+1, see N+1 queries. 0 disables it.
LEEUWENHOEK_SIGNALS ["message_flashed"] Flask signals the event watcher records, by name, see Signals
LEEUWENHOEK_REQUEST_TAGS {} Tags built from request data, see Tags
LEEUWENHOEK_SAMPLE_RATE 1.0 Fraction of requests to record (0–1)
LEEUWENHOEK_ONLY_ERRORS False Keep only batches with an exception or a 5xx response
LEEUWENHOEK_TRACK_MEMORY False Record each request's peak memory with tracemalloc, see Peak memory
LEEUWENHOEK_HANDLED_EXCEPTION_MIN_STATUS 500 Record exceptions handled by your error handlers when the response status is at least this
LEEUWENHOEK_STORAGE sqlalchemy sqlalchemy or memory (for tests)
LEEUWENHOEK_DATABASE_URI sqlite:///<instance_path>/leeuwenhoek.db Storage database, separate from your app's
LEEUWENHOEK_DB_DRIVER, _USERNAME, _PASSWORD, _HOST, _PORT, _DATABASE, _SOCKET — Build the URI from parts instead. The driver defaults to postgresql+psycopg. _SOCKET takes a Cloud SQL instance connection name (project:region:instance).
LEEUWENHOEK_ENGINE_OPTIONS {} Extra create_engine() arguments (pool size, etc.)
LEEUWENHOEK_AUTO_INSTALL True Create the tables in init_app
LEEUWENHOEK_ASYNC_WRITE True Write batches in a background thread
LEEUWENHOEK_QUEUE_MAX_BATCHES 1000 Pending batches kept in memory. When the queue is full, new batches are dropped (with a warning) and requests are never blocked.
LEEUWENHOEK_FLUSH_TIMEOUT 5.0 Seconds to wait for pending writes at exit
LEEUWENHOEK_PRUNE_HOURS 24 Retention period
LEEUWENHOEK_AUTO_PRUNE True Prune old entries from the background writer
LEEUWENHOEK_PRUNE_INTERVAL_MINUTES 30 How often to prune
LEEUWENHOEK_ALLOW_RETRY app.debug Enable the Retry button on failed Celery and RQ runs, see Retrying failed jobs
LEEUWENHOEK_RQ_REDIS_URL None Redis that RQ retries are enqueued on, when init_app got no rq_connection=
LEEUWENHOEK_BASIC_AUTH None ("user", "password") to protect the panel
LEEUWENHOEK_PAGE_SIZE 50 Rows per page in the panel
LEEUWENHOEK_POLL_SECONDS 4 How often the panel polls for new rows and menu counts. 0 disables live polling.
LEEUWENHOEK_BULMA_CDN False Load Bulma from a CDN instead of the bundled copy

Access control

  • Debug mode: the panel is open.

  • Outside debug mode: the panel returns 403 unless you configure access. Use either a rule:

    from flask_login import current_user
    
    lw.authorize(lambda: current_user.is_authenticated and current_user.is_admin)
    

    or basic auth:

    LEEUWENHOEK_BASIC_AUTH=admin:s3cret
    

    If both are set, both must pass.

  • When Leeuwenhoek is disabled (LEEUWENHOEK_ENABLED is false), the panel returns 404.

Tags

Tags make entries searchable from the panel with ?tag=. There are three ways to add them:

# 1. From request data, with dotted paths (lists are traversed, and JSON or base64 JSON strings are parsed)
app.config["LEEUWENHOEK_REQUEST_TAGS"] = {
    "order": "body.order_id",  # -> order:1234
    "sku": "body.items.sku",  # -> one tag per item
    "tenant": ["headers.X-Tenant-Id", "query.tenant"],
    "event": "body.message.data.event_type",  # a Pub/Sub push message
}


# 2. A tagger callback, run for every entry before it is saved
@lw.tag
def tenant_tag(entry):
    return [f"tenant:{g.tenant_id}"] if hasattr(g, "tenant_id") else []


# 3. From anywhere while a request is being handled; the tags reach every entry in its batch
from leeuwenhoek import add_tags

add_tags(f"customer:{customer.id}")

Filters

A filter decides which entries are stored. It receives an entry and returns True to keep it. Several can be registered, and lw.filter works as a decorator too:

# Skip "304 Not Modified" responses
lw.filter(lambda e: not (e.type == "request" and e.content["status"] == 304))


@lw.filter
def no_debug_logs(entry):
    return not (entry.type == "log" and entry.content["level"] == "DEBUG")

Filters run when a batch is saved, before tagging. If a filter drops the root entry of a batch (the request entry, for example), the whole batch is dropped. A filter that raises is logged and the entry is kept.

Running in production or staging

Leeuwenhoek is built for development first. For shared environments, a reasonable setup is:

LEEUWENHOEK_ENABLED=true
LEEUWENHOEK_SAMPLE_RATE=0.1          # or LEEUWENHOEK_ONLY_ERRORS=true
LEEUWENHOEK_BASIC_AUTH=admin:...     # or lw.authorize(...)
LEEUWENHOEK_DB_HOST=10.0.0.5         # Postgres instead of SQLite
LEEUWENHOEK_DB_USERNAME=leeuwenhoek
LEEUWENHOEK_DB_PASSWORD=...
LEEUWENHOEK_DB_DATABASE=leeuwenhoek

For Postgres, install a driver too, for example psycopg[binary].

Things to know:

  • Writes never block a request. If the database is slow or down, batches are dropped and a warning is logged.
  • The Retry button on failed jobs is off outside debug mode; turn it on with LEEUWENHOEK_ALLOW_RETRY=true only if the people with panel access may re-run jobs.
  • Every instance prunes old entries on its own schedule, with jitter so they don't all prune at once.
  • Designed for gunicorn (including --preload, since the writer thread is restarted after a fork), uWSGI, the Werkzeug dev server and async views.

CLI

flask leeuwenhoek install            # create the tables (done automatically unless AUTO_INSTALL is off)
flask leeuwenhoek prune --hours=48   # delete entries older than 48 hours
flask leeuwenhoek clear              # delete everything
flask leeuwenhoek pause              # stop recording (shared by all instances)
flask leeuwenhoek resume

Testing your app with Leeuwenhoek

Use the in-memory storage and synchronous writes, so entries can be checked right after a request:

app.config.update(
    LEEUWENHOEK_ENABLED=True,
    LEEUWENHOEK_STORAGE="memory",
    LEEUWENHOEK_ASYNC_WRITE=False,
)
lw = Leeuwenhoek(app)
...
lw.storage.list("request")
lw.close()  # remove global handlers and listeners

Roadmap

Work is tracked in GitHub issues and grouped into milestones.

Shipped (on main, not yet released to PyPI):

  • v0.2 (complete): watchers for outgoing HTTP, templates, CLI commands, mail, cache and Celery jobs; slow-query and N+1 detection; lw.filter(); capture of exceptions outside requests (sys.excepthook); live polling; dark mode; auto-detection of available watchers; a test that the wheel ships templates and static files; a request overhead benchmark; mypy and ruff format in CI.
  • v0.3 (in progress): watchers for RQ jobs, ORM models, Redis, scheduled tasks and blinker events; peak memory per request; resolving exception families.

Upcoming in v0.3:

  • Retrying failed jobs from the panel (#21).
  • Publishing to PyPI (#23).

Out of scope: distributed tracing or APM (use OpenTelemetry), aggregated metrics, and alerting.

Development

uv sync                      # install dependencies and dev tools
uv run pytest                # tests
uv run ruff check .          # lint
uv run ruff format .         # format
uv build                     # sdist + wheel

See docs/architecture.md for how it works inside.

License

MIT

Metadata

Release files for leeuwenhoek 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for leeuwenhoek 0.1.0
File Size Uploaded Provenance Actions
leeuwenhoek-0.1.0.tar.gz 168.9 kB With provenance Details for leeuwenhoek-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for leeuwenhoek 0.1.0
File Interpreter ABI Platform Actions
leeuwenhoek-0.1.0-py3-none-any.whl Python 3 none any Details for leeuwenhoek-0.1.0-py3-none-any.whl

Total release size: 350.4 kB

Release files / leeuwenhoek-0.1.0.tar.gz

Download URL leeuwenhoek-0.1.0.tar.gz
Size 168.9 kB
Tags Source
SHA-256 checksum
How to use checksums
cba629e83611c1733e3006a7795940818deba5f76b8271db6fbfe0813719cb73
BLAKE2b-256 checksum
How to use checksums
5217c59153cbc194b2955f986aa9f3efc8f40d835fd39ddefb08cb9dbc310ff5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / leeuwenhoek-0.1.0-py3-none-any.whl

Download URL leeuwenhoek-0.1.0-py3-none-any.whl
Size 181.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fa4390f484f8a3b55a0effaa91cee7bc1961ca3aa8d1cf2e8dd701f2e30d441f
BLAKE2b-256 checksum
How to use checksums
be249734587e56b063985690a42a9bf4a537a2442fda2403bd4947ad2882d615
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for leeuwenhoek 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for leeuwenhoek-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page