leeuwenhoek
A debugging microscope that lives inside your Flask app. Leeuwenhoek records what happens during each request, CLI command and background job (the request itself, SQL queries, exceptions, log lines, ad-hoc dumps, outgoing HTTP calls, cache and Redis operations, mail, ORM changes and more), groups it all by batch, and shows it in a panel at /leeuwenhoek.
- No separate service. It is a Blueprint plus a few hooks. The panel is rendered on the server with Jinja and Bulma, with no frontend build and no CDN.
- No changes to your code. Everything is captured through Flask hooks, signals, SQLAlchemy events and a logging handler.
- Grouped by batch. Every entry produced while handling one request shares a
batch_id, so you can open a request and see every query, log line and exception it produced. - Its own database. Entries go to a separate SQLAlchemy engine (SQLite by default, Postgres in production), so they never mix with your app's data or record themselves.
- Safe enough for staging. Sensitive values are redacted, payloads are truncated, and writes happen in a background thread with a bounded queue. Sampling, an only-errors mode and access control are built in.
Status: early (v0.1, not on PyPI yet). See the roadmap for what has shipped and what's coming.
Installation
Requires Python 3.10+, Flask 2.3+ and SQLAlchemy 2.x.
uv add leeuwenhoek
# or
pip install leeuwenhoek
Add the login extra (leeuwenhoek[login]) to record the current Flask-Login user on each request. If Flask-Login is already installed, this happens automatically.
Add the cache extra (leeuwenhoek[cache]) to record Flask-Caching operations. As with Flask-Login, the cache watcher turns on by itself when Flask-Caching is installed.
Add the mail extra (leeuwenhoek[mail]) to record emails sent with Flask-Mail. As with every integration watcher, it turns on by itself when Flask-Mail is installed.
Add the redis extra (leeuwenhoek[redis]) to record the commands sent with redis-py. The Redis watcher turns on by itself when redis is installed.
Add the celery extra (leeuwenhoek[celery]) to record Celery tasks, see Celery jobs. The watcher turns on by itself when Celery is installed.
Add the http extra to install requests and HTTPX for the outgoing HTTP watcher. Either one is enough, and the watcher turns on by itself if your app already uses one of them.
Add the sched extra (leeuwenhoek[sched]) to record APScheduler job runs. Only APScheduler 3.x is supported (apscheduler>=3.10,<4); version 4 has a different API. The watcher turns on by itself when APScheduler is installed.
Add the rq extra (leeuwenhoek[rq]) to record RQ jobs (RQ 1.15 or newer). The watcher turns on by itself when RQ is installed.
Quickstart
from flask import Flask
from leeuwenhoek import Leeuwenhoek
app = Flask(__name__)
Leeuwenhoek(app)
Or with the application factory pattern:
lw = Leeuwenhoek()
def create_app():
app = Flask(__name__)
lw.init_app(app, engine=db.engine) # engine: only record queries from this engine
return app
Run the app in debug mode (flask run --debug), make a few requests, then open http://localhost:5000/leeuwenhoek.
By default Leeuwenhoek is enabled only when app.debug is True, and the panel is only reachable in debug mode. See Access control to use it anywhere else.
Try the demo
git clone https://github.com/Manuel-Espinosa/leeuwenhoek
cd leeuwenhoek
uv sync
uv run flask --app examples/demo run --debug
Visit /users, /boom and /dump, then open /leeuwenhoek.
What it records
| Watcher | Captures |
|---|---|
request |
Method, URL, URL rule, endpoint, blueprint, view args, IP, headers, query string, body, session, user, status, response headers, response body (JSON/text only), duration, and optionally peak memory |
query |
SQL, parameters, duration, and the file and line in your code that ran the query. Queries slower than LEEUWENHOEK_SLOW_QUERY_MS are flagged as slow, and queries repeated many times in one request as N+1. |
exception |
Class, message, full traceback with a code snippet for each frame. Repeated exceptions are grouped into families (same class, file and line). Also captures exceptions turned into 5xx responses by your own error handlers, and uncaught exceptions in scripts, CLI commands and threads (through sys.excepthook and threading.excepthook, which are chained and restored by lw.close()). |
log |
Level, message, logger name, extra fields and exc_info, from a handler on the root logger |
dump |
Whatever you pass to dump(), with the file and line it was called from |
mail |
Emails sent with Flask-Mail (its email_dispatched signal, which also fires when sending is suppressed with MAIL_SUPPRESS_SEND or TESTING): sender, to/cc/bcc, reply-to, subject, extra headers (redacted with LEEUWENHOEK_HIDDEN_HEADERS), HTML and text bodies (truncated to LEEUWENHOEK_MAX_BODY_KB) and attachment names and types, never their content. The panel previews the HTML body in a fully sandboxed <iframe> (no scripts, no same-origin). Needs the mail extra. |
template |
Each Jinja template rendered with render_template, render_template_string or stream_template: name, file, context keys (never their values, and without Flask's own g, request and session) and render duration. Templates pulled in with {% extends %} or {% include %} are part of the top-level render, not separate entries. Leeuwenhoek's own templates are never recorded. |
command |
flask CLI commands of app.cli: full name (db seed for subcommands), arguments, options, exit code and duration. Each command gets its own batch, see CLI commands. |
job |
Celery task runs: task name and id, args and kwargs, queue, attempt, state (SUCCESS, FAILURE, RETRY), duration, result, exception and retry reason. Each run gets its own batch, linked to the request that dispatched it, see Celery jobs. |
http |
Outgoing HTTP calls made with requests or HTTPX (sync and async clients): method, URL, request headers, payload, status, response headers, response body (JSON/text only), duration, redirects, and the error if the call raised. Enabled when either library is installed. |
cache |
Flask-Caching operations (get, get_many, get_dict, has, set, add, set_many, delete, delete_many, unlink, clear), including the ones made by @cache.cached and @cache.memoize (with Flask-Caching 2.5 or newer; older versions, the last ones for Python 3.10, make the decorators talk to the backend directly and only the direct calls are recorded): keys, hit or miss for reads, TTL for writes, backend and duration. Values are stored only with LEEUWENHOEK_CACHE_VALUES. The Cache card of a request shows its hit and miss counts. |
model |
Inserts, updates and deletes made through the SQLAlchemy ORM, in any Session (Flask-SQLAlchemy's or your own): model, table, action (created, updated, deleted), primary key and the changed columns with their old and new values. Recorded when the transaction commits, see ORM models. |
event |
Blinker signals: the Flask signals in LEEUWENHOEK_SIGNALS (by default message_flashed) and your own signals added with lw.watch_signal(): signal name, sender (class name or truncated repr) and keyword arguments (redacted with LEEUWENHOEK_HIDDEN_PARAMS, truncated to LEEUWENHOEK_MAX_BODY_KB), tagged event:<name>. See Signals. |
redis |
Commands sent with redis-py (redis.Redis and redis.asyncio.Redis): command, arguments (redacted and truncated), server, database, duration and the error if the command raised. A pipeline is one entry with its list of commands. See Redis. |
schedule |
Each run of an APScheduler 3.x job: id, name, function, trigger, executor, scheduled and next run time, arguments (redacted with LEEUWENHOEK_HIDDEN_PARAMS), result (repr, truncated) or exception, and duration; also runs missed past their misfire grace time and runs skipped because the job reached max_instances. Each run gets its own batch, see Scheduled jobs. Needs the sched extra. |
rq |
RQ jobs: function path, queue, job id, status, arguments and keyword arguments (redacted with LEEUWENHOEK_HIDDEN_PARAMS, truncated to LEEUWENHOEK_MAX_BODY_KB), duration and exception. Each execution gets its own batch, linked to the request that enqueued the job, see RQ jobs. Needs the rq extra. |
By default every watcher whose library is installed is enabled, so integration watchers (mail, Celery, Redis...) turn on by themselves when you install the package they observe. Set LEEUWENHOEK_WATCHERS to pick them explicitly, e.g. LEEUWENHOEK_WATCHERS=request,exception. Only enabled watchers appear in the panel menu.
Requests automatically get the tags endpoint:<name>, status:<code> and user:<id>. Commands get command:<name> and exit_code:<code>, Celery tasks task:<name> and state:<STATE>, RQ entries rq_job:<id>, rq_queue:<name> and rq_status:<status>, scheduled job runs job:<id> and job_status:<success|error|missed|skipped>. Log entries get level:<LEVEL>, templates template:<name> and ORM changes model:<Name>. Slow queries get slow, show a yellow slow badge in the panel, and the Queries card of a request shows how many of its queries were slow. Repeated queries get n+1 (see N+1 queries).
N+1 queries
When a batch closes, its queries are grouped by their normalized SQL: string and number literals and bound parameters (?, %s, :name, %(name)s, $1) become ?, lists such as IN (1, 2, 3) collapse to IN (?), whitespace is collapsed and keywords are upper-cased. So select * from users where id = 1 and SELECT * FROM users WHERE id = 2 are the same statement.
Every query in a group of at least LEEUWENHOEK_N_PLUS_ONE_THRESHOLD (default 5) gets content["n_plus_one"] = True and the n+1 tag, and shows a red n+1 badge in the panel. The Queries card of the request lists each repeated statement with its count, for example SELECT * FROM users WHERE id = ? × 12. Set the threshold to 0 to turn detection off.
The detection runs as an on_finish callback, before the batch is saved, so filters and taggers already see the flag and the tag (lw.filter(lambda e: not e.content.get("n_plus_one")) works, for example). Queries a filter drops afterwards were still counted.
Outgoing HTTP
Every call made through requests or httpx (including httpx.AsyncClient) is recorded, whichever client or session made it, and appears in the HTTP client section and as a card in the detail of the request that made it. Calls made outside a request are stored as standalone entries. The URL's query parameters and the payload and JSON response keys listed in LEEUWENHOEK_HIDDEN_PARAMS are masked, as are the headers in LEEUWENHOEK_HIDDEN_HEADERS; bodies are cut at LEEUWENHOEK_MAX_BODY_KB. Only JSON and text response bodies are kept, and never for streamed responses (stream=True), which are left unread. A requests redirect chain is one entry with the final URL and the number of redirects. Entries are tagged host:<hostname>, and calls that raised are tagged failed.
ORM models
The model watcher listens to after_flush on SQLAlchemy's Session class, so every session is covered without configuration. For each object a flush inserts, updates or deletes it records the model class, the table, the action, the primary key (also for new rows, whose key the database just assigned) and the changes, as {"column": {"old": ..., "new": ...}}:
- created: every loaded column, from
null. - updated: only the columns that really changed (an object in
session.dirtywhose values are the same is skipped). An old value that was never loaded isnull. - deleted: every loaded column, to
null.
Only columns are recorded; relationships are skipped, since their foreign key columns already show the link. Values are made JSON-safe (datetimes as ISO strings, Decimal as a string, binary data as <N bytes>), columns named in LEEUWENHOEK_HIDDEN_PARAMS are masked, and each value is cut at LEEUWENHOEK_MAX_BODY_KB. Nothing is ever loaded from the database to fill a value.
Changes are captured at flush time, while their history still exists, but only recorded when the transaction commits. Changes flushed and then rolled back (including a rolled-back begin_nested() savepoint) or left uncommitted when the session closes are dropped, so the panel only shows what reached the database. A commit made after the request ended is stored as standalone. Leeuwenhoek's own storage (its engine and its tables) is never recorded.
Signals
Watch your own blinker signals with lw.watch_signal(), before or after init_app. Every send is recorded as an event entry in the Events section and in the batch of the request, command or job that sent it, or as a standalone entry outside one. Watching a signal twice does nothing.
from blinker import signal
order_placed = signal("order-placed")
lw.watch_signal(order_placed) # or lw.watch_signal("order-placed")
lw.watch_signal(anonymous_signal, name="cart-emptied") # a name for a Signal() without one
@app.post("/orders")
def create_order():
order_placed.send(current_user, order_id=7, total=99.5)
Flask's own signals are picked by name with LEEUWENHOEK_SIGNALS (message_flashed, request_started, request_finished, request_tearing_down, appcontext_pushed, appcontext_popped, appcontext_tearing_down, template_rendered, before_render_template, got_request_exception), and only the ones the app sends are recorded. The default is just message_flashed: the request and app context signals fire on every request and repeat what the request entry already shows. request_started fires before the request's batch opens and request_tearing_down and the app context ones after it closes, so they are stored as standalone entries.
A signal that another active watcher already records is never recorded twice: template_rendered and before_render_template while the template watcher is on, got_request_exception while the exception watcher is on, and Flask-Mail's email_dispatched while the mail watcher is on. Signals sent while the panel is served, for ignored paths, while recording is paused, or by Leeuwenhoek itself are never recorded.
Redis
Every command sent through redis.Redis or redis.asyncio.Redis (and their subclasses, such as fakeredis) is recorded, whichever client sent it, and appears in the Redis section and as a card, with the count and total time, in the detail of the request that sent it. Commands sent outside a request are standalone entries. A pipeline (r.pipeline(), with or without a transaction) is one PIPELINE entry listing its commands, the error of each one that failed with raise_on_error=False, and the time of the whole round trip; its queued commands are not recorded again. Commands that a pipeline in WATCH mode sends right away are recorded on their own. Return values are never stored.
Arguments are decoded as UTF-8 (binary values are shown as [binary: N bytes]); at most 20 are kept per command, each cut at 200 characters, and at most 50 commands per pipeline. Passwords are never stored: the arguments of AUTH, the password of HELLO ... AUTH and MIGRATE ... AUTH/AUTH2, CONFIG SET requirepass/masterauth and the password rules of ACL SETUSER are replaced by ********. When a key contains a name in LEEUWENHOEK_HIDDEN_PARAMS (user:1:token), every argument after it is masked, and a hash field or argument that contains one (HSET user:1 password ...) masks the value that follows it. Commands that raised are tagged failed.
Libraries that use Redis are recorded at the command level too. With Flask-Caching's RedisCache, a cache.get() gives a cache entry and the GET it sent as a redis entry, in the same batch. RQ's commands are recorded as well: those sent while enqueueing join the request, and those a worker sends between jobs (polling, heartbeats) are standalone; drop them with lw.filter() or leave redis out of LEEUWENHOEK_WATCHERS if they are noise. Redis Cluster clients (RedisCluster) and Pub/Sub connections are not covered.
dump()
from leeuwenhoek import dump
@app.get("/checkout")
def checkout():
dump(cart) # shows up in the Dumps section and in this request's batch
return "ok"
CLI commands
Every command you add to app.cli (with @app.cli.command(), a nested group or a blueprint's cli) gets its own batch, so the queries, logs and exceptions it produces are grouped under it, as they are under a request. Commands registered after init_app are recorded too.
@app.cli.command("seed")
@click.argument("count", type=int)
@click.option("--token")
def seed(count, token):
app.logger.info("seeding %s users", count)
db.session.execute(...)
flask seed 10 --token=abc records a command entry with {"count": 10} as arguments, {"token": "********"} as options (values are redacted with LEEUWENHOEK_HIDDEN_PARAMS), the exit code and the duration, plus the log line and the query, all in one batch. The exit code is 0, the code given to sys.exit() or ctx.exit(), the code of a click.ClickException (2 for a usage error), or 1 when the command raises. An exception raised by the command is recorded in its batch, and the batch is saved even though the command failed.
Like requests, commands are only recorded while Leeuwenhoek is enabled: run them with flask --debug ... or set LEEUWENHOEK_ENABLED. Skip some with LEEUWENHOEK_IGNORE_COMMANDS: an entry matches a command by its full name, by a group prefix ("db" skips db upgrade) or as a glob ("seed-*"). Nothing that happens inside an ignored command is recorded. flask leeuwenhoek ..., flask run and flask shell never get a command entry or a batch: each request served by flask run gets its own batch as usual, and a shell session, which can last for hours and isn't one unit of work, stores what you run in it as standalone entries.
Scheduled jobs
Every run of an APScheduler 3.x job gets its own batch, with a schedule entry as its root, so the queries, logs and exceptions of one run are grouped under it, as they are under a request. It works with every scheduler (BackgroundScheduler, BlockingScheduler, AsyncIOScheduler...), every job store and coroutine jobs.
scheduler = BackgroundScheduler()
@scheduler.scheduled_job("interval", minutes=5, id="sync")
def sync_users():
with app.app_context(): # needed to use Flask-SQLAlchemy and friends
app.logger.info("syncing users")
db.session.execute(...)
Leeuwenhoek never pushes an app context for a job: a job that needs one must push it itself, as it would without Leeuwenhoek, so the job behaves the same whether it is installed or not. Recording needs no app context.
A run records the job id and name, the function, the trigger (interval[0:05:00], cron[...]), the executor, the time it was scheduled for and the next run time, the arguments, its duration, and its repr()'d result (cut at 1000 characters) or the exception it raised, which is also recorded in the batch by the exception watcher (as handled, since APScheduler catches it). Runs missed by more than the job's misfire_grace_time and runs skipped because the job already had max_instances running are recorded as missed and skipped entries, without a batch of their own. A run started while a batch is current in the scheduler's thread (a flask command running a BlockingScheduler, for example) keeps that batch as its parent.
APScheduler logs Running job ... before a run and Job ... raised an exception after it, outside the run, so the log watcher stores these lines as standalone entries.
With the ProcessPoolExecutor, the job runs in another process and is not recorded: Leeuwenhoek's wrapper does not cross the process boundary, and the child process gets the original job. Use a thread pool (the default) or record from inside the job. Celery beat is not part of this watcher: scheduled Celery tasks are recorded by the Celery watcher.
RQ jobs
Jobs are recorded twice, as RQ sees them:
- The dispatch:
queue.enqueue(...)(orenqueue_call, orenqueue_jobwith a job built byJob.create) records anrqentry withevent: "enqueue"in the current batch, so a request's detail lists the jobs it enqueued, and stores the batch id injob.meta["leeuwenhoek_parent_batch_id"]. Its status isqueued,deferredfor a job waiting on a dependency, or the outcome for a synchronous queue. Outside a batch (a script), it is stored asstandalone. - Each execution: the worker runs the job in a new batch (
kind="job") whoseparent_batch_idis the dispatching batch. Its root entry (event: "perform") has the statusfinishedorfailed, the duration and, on failure, the exception class and message. Queries, logs, HTTP calls and the exception raised by the job join this batch, and the job's detail links back to the request. A retried job gets one batch per attempt.
The rq_job:<id> tag ties the dispatch and every execution of one job together: the job detail links to them.
queue = Queue(connection=redis)
@app.post("/reports")
def create_report():
# Recorded kwargs: {"report_id": 7, "token": "********"}
queue.enqueue(build_report, report_id=7, token="abc")
return "", 202
The worker must create the app (and so call Leeuwenhoek.init_app) before it starts, for example with a small script that builds the app and then runs Worker([queue], connection=redis).work(). Forking workers (the default Worker), SimpleWorker and Queue(is_async=False) are all covered. A forked work horse ends with os._exit(), which skips the atexit flush, so the watcher writes the job's batch before handing control back to RQ (waiting up to LEEUWENHOEK_FLUSH_TIMEOUT). The worker's own work between jobs (polling, heartbeats, registry cleanup, its log lines) is never recorded. A work horse killed by RQ (SIGKILL after a hard timeout) records nothing. Jobs that RQ moves on its own (the scheduler enqueuing a job scheduled with enqueue_at, a dependency becoming ready, a retry) are not recorded again as dispatches, only when they run.
args and kwargs hold the job's original arguments when args_redacted is false: nothing was masked, truncated or turned into a string to fit in JSON. When it is true, they are only for display. job.description is not stored, because it repeats the arguments unredacted.
Celery jobs
Pass your Celery app to init_app (or to the constructor):
celery = Celery(__name__, broker="redis://localhost:6379/0")
lw = Leeuwenhoek()
lw.init_app(app, celery=celery)
Every run of one of its tasks gets its own batch (kind="job"), so the queries, logs and exceptions of the task are grouped under it, and the batch page shows the job on top. Without celery=, the tasks of every Celery app in the process are recorded (Celery's signals are global). The worker must create the Flask app and call init_app too (the usual Flask + Celery setup does), and Leeuwenhoek must be enabled there (LEEUWENHOEK_ENABLED).
When a task is dispatched while a request (or a command, or another job) is being recorded, the id of that batch travels in the message headers, and the job's entries get it as parent_batch_id. The dispatching batch also gets a job entry with the status dispatched, shown in the Jobs card of the request detail with a View job link to the job's batch; the job's page links back with Dispatched by. A retry (self.retry()) is a new run with its own batch and the same parent. The Jobs listing shows the runs only.
In eager mode (task_always_eager, or task.apply()), Celery doesn't publish a message, so there are no headers: the task runs inside the batch that dispatched it, and that batch becomes its parent. Tasks dispatched outside any batch (a script, celery beat) get a job batch without a parent.
Args and kwargs are redacted with LEEUWENHOEK_HIDDEN_PARAMS (by key, at any depth) and cut at LEEUWENHOEK_MAX_BODY_KB, like the result. content["args_redacted"] is false only when the stored args and kwargs are exactly the ones the task received (nothing masked, truncated or unserializable), so the job could be dispatched again with them.
Peak memory
Set LEEUWENHOEK_TRACK_MEMORY=true to record how much memory each request allocated at its peak. It is shown as Peak memory in the request detail and stored as memory_peak_kb in the entry.
It uses tracemalloc, which is off by default because it slows down every allocation while it runs (about 1 ms more per request in the benchmark). Leeuwenhoek starts tracing when a recorded request begins and stops it when none is running. If tracing was already started by something else (python -X tracemalloc, a profiler), it is left running, and only memory allocated during the request counts.
tracemalloc measures the whole process, not one thread. With concurrent requests (a threaded server, gevent), each peak includes the other requests in flight, so treat the numbers as approximate there. They are exact when requests are handled one at a time.
The panel
| URL | Shows |
|---|---|
/leeuwenhoek/requests |
Requests (also /queries, /exceptions, /logs, /dumps, /commands, /http, /cache, /models, /events, /redis, /schedule, /rq, /jobs), newest first, with search (?q=) and tag filter (?tag=); /exceptions?resolved=1 includes resolved families |
/leeuwenhoek/batches |
One row per batch |
/leeuwenhoek/<type>/<uuid> |
One entry, plus everything else from its batch grouped by type |
/leeuwenhoek/batch/<batch_id> |
A whole batch, with its request, command or scheduled job run, RQ job or Celery task on top |
/leeuwenhoek/exceptions/family/<hash> |
Every occurrence of one exception, with a Resolve / Reopen button |
POST /leeuwenhoek/exceptions/family/<hash>/resolve (/reopen) |
Marks the family as resolved (or open again); needs the CSRF token |
POST /leeuwenhoek/jobs/<uuid>/retry (/rq/<uuid>/retry) |
Dispatches a failed Celery task (or RQ job) again with its original arguments; needs the CSRF token, see Retrying failed jobs |
/leeuwenhoek/<type>/rows?after=<id> |
HTML <tr> rows of the entries newer than <id>, with the same filters as the listing (used by live polling) |
/leeuwenhoek/api/status |
JSON: recording state and counts per type |
Listings update live: every LEEUWENHOEK_POLL_SECONDS the panel fetches the new rows and puts them at the top of the table, and refreshes the counters in the menu. Polling pauses while the tab is hidden and doesn't add rows on an older page (?before=).
The navbar has buttons to pause or resume recording and to clear all entries. They, like the Resolve / Reopen buttons below, use POST with the panel's own CSRF token. Apps using Flask-WTF don't need any changes, because the panel is exempted from Flask-WTF's CSRF check.
Resolving exceptions
Once a bug is fixed, mark its exception family as resolved with the Resolve button on the Exceptions listing, an exception's detail page or its family page. Resolved families are hidden from the Exceptions listing (and from its live polling); Show resolved (/leeuwenhoek/exceptions?resolved=1) lists them again, marked as resolved and with a Reopen button.
A new occurrence after the resolve time reopens the family by itself: it shows up in the listing again with all its occurrences. The state lives in the settings table (resolved:<family_hash> with the resolve time), so every instance sharing the database sees it.
Retrying failed jobs
The detail page of a failed run, a Celery task with the state FAILURE or an RQ execution with the status failed, has a Retry button that dispatches the job again with the arguments it received. Dispatch entries and successful runs don't get one. Like the other panel buttons, it uses POST with the panel's CSRF token and follows the panel's access control.
The button is disabled, with the reason next to it, when the run can't be dispatched again:
- The arguments were not kept exactly (
args_redactedistrue): something was masked withLEEUWENHOEK_HIDDEN_PARAMS, cut atLEEUWENHOEK_MAX_BODY_KBor was not plain JSON. Leeuwenhoek never dispatches a job with the arguments it only shows. - Retrying is turned off:
LEEUWENHOEK_ALLOW_RETRYdefaults toapp.debug. Re-running a job is a side effect (a second email, a second charge), so outside debug mode it stays off until you setLEEUWENHOEK_ALLOW_RETRY=true. - RQ only: there is no Redis connection to enqueue on. Leeuwenhoek doesn't store the original job's connection, so give it one:
lw.init_app(app, rq_connection=redis)orLEEUWENHOEK_RQ_REDIS_URL=redis://localhost:6379/0.
A POST that is refused gets the detail page back with the reason: 409 when the run can't be retried, 403 when retrying is turned off, 502 when the broker or Redis rejects the dispatch. On success the panel goes back to the failed run with a notice, the id of the new task or job and a link to its run.
How the job is dispatched again:
- Celery: on the app passed as
celery=(orcelery.current_app), to the queue the run was recorded on, with aleeuwenhoek_retry_ofmessage header. When the task is registered in the panel's process it goes throughtask.apply_async(), sotask_always_eagerand the task's options apply; otherwise it is published by name withsend_task(), for tasks defined only on the workers. - RQ:
functionis enqueued by its dotted path onQueue(<queue>, connection=...), as a new job with its own id, andjob.meta["leeuwenhoek_retry_of"]set.
Either way, the new run stores the uuid of the failed entry as content["retry_of"] and gets the tag retry_of:<uuid>: its detail page links back with Retry of, and the failed run's Retries of this run link lists every retry. The panel's own requests are not recorded, so there is no dispatch entry and the new run has no parent batch.
Configuration
Each setting is read from app.config["LEEUWENHOEK_<KEY>"] first, then from an environment variable with the same name. In environment variables, booleans accept 1/true/yes/on, lists accept JSON or comma-separated values, and BASIC_AUTH uses the form user:password.
| Key | Default | Description |
|---|---|---|
LEEUWENHOEK_ENABLED |
app.debug |
Master switch |
LEEUWENHOEK_URL_PREFIX |
/leeuwenhoek |
Where the panel is mounted. Requests under it are never recorded. |
LEEUWENHOEK_WATCHERS |
every watcher whose library is installed | Watchers to enable. A listed watcher whose library is missing is skipped with a warning. |
LEEUWENHOEK_IGNORE_PATHS |
["/static*", "/favicon.ico", "/health*"] |
Glob patterns of paths that are not recorded |
LEEUWENHOEK_IGNORE_COMMANDS |
[] |
CLI commands that are not recorded: full names, group prefixes or glob patterns, see CLI commands |
LEEUWENHOEK_HIDDEN_HEADERS |
["authorization", "cookie", "x-api-key"] |
Headers replaced by ******** |
LEEUWENHOEK_HIDDEN_PARAMS |
["password", "password_confirmation", "token", "secret"] |
Keys replaced by ******** in bodies, query strings and sessions, at any depth |
LEEUWENHOEK_MAX_BODY_KB |
64 |
Bodies larger than this are truncated |
LEEUWENHOEK_MAX_ENTRIES_PER_BATCH |
500 |
Maximum entries per batch. The request entry itself is always kept. |
LEEUWENHOEK_SLOW_QUERY_MS |
100 |
Queries that take longer than this many milliseconds are flagged as slow and tagged slow |
LEEUWENHOEK_CACHE_VALUES |
False |
Store the values the cache watcher reads and writes, truncated to LEEUWENHOEK_MAX_BODY_KB. Fields named in LEEUWENHOEK_HIDDEN_PARAMS are redacted inside values, and the whole value is replaced by ******** when its key contains one of those names (user:1:token). |
LEEUWENHOEK_N_PLUS_ONE_THRESHOLD |
5 |
Queries with the same normalized SQL repeated this many times in a batch are tagged n+1, see N+1 queries. 0 disables it. |
LEEUWENHOEK_SIGNALS |
["message_flashed"] |
Flask signals the event watcher records, by name, see Signals |
LEEUWENHOEK_REQUEST_TAGS |
{} |
Tags built from request data, see Tags |
LEEUWENHOEK_SAMPLE_RATE |
1.0 |
Fraction of requests to record (0–1) |
LEEUWENHOEK_ONLY_ERRORS |
False |
Keep only batches with an exception or a 5xx response |
LEEUWENHOEK_TRACK_MEMORY |
False |
Record each request's peak memory with tracemalloc, see Peak memory |
LEEUWENHOEK_HANDLED_EXCEPTION_MIN_STATUS |
500 |
Record exceptions handled by your error handlers when the response status is at least this |
LEEUWENHOEK_STORAGE |
sqlalchemy |
sqlalchemy or memory (for tests) |
LEEUWENHOEK_DATABASE_URI |
sqlite:///<instance_path>/leeuwenhoek.db |
Storage database, separate from your app's |
LEEUWENHOEK_DB_DRIVER, _USERNAME, _PASSWORD, _HOST, _PORT, _DATABASE, _SOCKET |
— | Build the URI from parts instead. The driver defaults to postgresql+psycopg. _SOCKET takes a Cloud SQL instance connection name (project:region:instance). |
LEEUWENHOEK_ENGINE_OPTIONS |
{} |
Extra create_engine() arguments (pool size, etc.) |
LEEUWENHOEK_AUTO_INSTALL |
True |
Create the tables in init_app |
LEEUWENHOEK_ASYNC_WRITE |
True |
Write batches in a background thread |
LEEUWENHOEK_QUEUE_MAX_BATCHES |
1000 |
Pending batches kept in memory. When the queue is full, new batches are dropped (with a warning) and requests are never blocked. |
LEEUWENHOEK_FLUSH_TIMEOUT |
5.0 |
Seconds to wait for pending writes at exit |
LEEUWENHOEK_PRUNE_HOURS |
24 |
Retention period |
LEEUWENHOEK_AUTO_PRUNE |
True |
Prune old entries from the background writer |
LEEUWENHOEK_PRUNE_INTERVAL_MINUTES |
30 |
How often to prune |
LEEUWENHOEK_ALLOW_RETRY |
app.debug |
Enable the Retry button on failed Celery and RQ runs, see Retrying failed jobs |
LEEUWENHOEK_RQ_REDIS_URL |
None |
Redis that RQ retries are enqueued on, when init_app got no rq_connection= |
LEEUWENHOEK_BASIC_AUTH |
None |
("user", "password") to protect the panel |
LEEUWENHOEK_PAGE_SIZE |
50 |
Rows per page in the panel |
LEEUWENHOEK_POLL_SECONDS |
4 |
How often the panel polls for new rows and menu counts. 0 disables live polling. |
LEEUWENHOEK_BULMA_CDN |
False |
Load Bulma from a CDN instead of the bundled copy |
Access control
-
Debug mode: the panel is open.
-
Outside debug mode: the panel returns 403 unless you configure access. Use either a rule:
from flask_login import current_user lw.authorize(lambda: current_user.is_authenticated and current_user.is_admin)
or basic auth:
LEEUWENHOEK_BASIC_AUTH=admin:s3cret
If both are set, both must pass.
-
When Leeuwenhoek is disabled (
LEEUWENHOEK_ENABLEDis false), the panel returns 404.
Tags
Tags make entries searchable from the panel with ?tag=. There are three ways to add them:
# 1. From request data, with dotted paths (lists are traversed, and JSON or base64 JSON strings are parsed)
app.config["LEEUWENHOEK_REQUEST_TAGS"] = {
"order": "body.order_id", # -> order:1234
"sku": "body.items.sku", # -> one tag per item
"tenant": ["headers.X-Tenant-Id", "query.tenant"],
"event": "body.message.data.event_type", # a Pub/Sub push message
}
# 2. A tagger callback, run for every entry before it is saved
@lw.tag
def tenant_tag(entry):
return [f"tenant:{g.tenant_id}"] if hasattr(g, "tenant_id") else []
# 3. From anywhere while a request is being handled; the tags reach every entry in its batch
from leeuwenhoek import add_tags
add_tags(f"customer:{customer.id}")
Filters
A filter decides which entries are stored. It receives an entry and returns True to keep it. Several can be registered, and lw.filter works as a decorator too:
# Skip "304 Not Modified" responses
lw.filter(lambda e: not (e.type == "request" and e.content["status"] == 304))
@lw.filter
def no_debug_logs(entry):
return not (entry.type == "log" and entry.content["level"] == "DEBUG")
Filters run when a batch is saved, before tagging. If a filter drops the root entry of a batch (the request entry, for example), the whole batch is dropped. A filter that raises is logged and the entry is kept.
Running in production or staging
Leeuwenhoek is built for development first. For shared environments, a reasonable setup is:
LEEUWENHOEK_ENABLED=true
LEEUWENHOEK_SAMPLE_RATE=0.1 # or LEEUWENHOEK_ONLY_ERRORS=true
LEEUWENHOEK_BASIC_AUTH=admin:... # or lw.authorize(...)
LEEUWENHOEK_DB_HOST=10.0.0.5 # Postgres instead of SQLite
LEEUWENHOEK_DB_USERNAME=leeuwenhoek
LEEUWENHOEK_DB_PASSWORD=...
LEEUWENHOEK_DB_DATABASE=leeuwenhoek
For Postgres, install a driver too, for example psycopg[binary].
Things to know:
- Writes never block a request. If the database is slow or down, batches are dropped and a warning is logged.
- The Retry button on failed jobs is off outside debug mode; turn it on with
LEEUWENHOEK_ALLOW_RETRY=trueonly if the people with panel access may re-run jobs. - Every instance prunes old entries on its own schedule, with jitter so they don't all prune at once.
- Designed for gunicorn (including
--preload, since the writer thread is restarted after a fork), uWSGI, the Werkzeug dev server and async views.
CLI
flask leeuwenhoek install # create the tables (done automatically unless AUTO_INSTALL is off)
flask leeuwenhoek prune --hours=48 # delete entries older than 48 hours
flask leeuwenhoek clear # delete everything
flask leeuwenhoek pause # stop recording (shared by all instances)
flask leeuwenhoek resume
Testing your app with Leeuwenhoek
Use the in-memory storage and synchronous writes, so entries can be checked right after a request:
app.config.update(
LEEUWENHOEK_ENABLED=True,
LEEUWENHOEK_STORAGE="memory",
LEEUWENHOEK_ASYNC_WRITE=False,
)
lw = Leeuwenhoek(app)
...
lw.storage.list("request")
lw.close() # remove global handlers and listeners
Roadmap
Work is tracked in GitHub issues and grouped into milestones.
Shipped (on main, not yet released to PyPI):
- v0.2 (complete): watchers for outgoing HTTP, templates, CLI commands, mail, cache and Celery jobs; slow-query and N+1 detection;
lw.filter(); capture of exceptions outside requests (sys.excepthook); live polling; dark mode; auto-detection of available watchers; a test that the wheel ships templates and static files; a request overhead benchmark; mypy and ruff format in CI. - v0.3 (in progress): watchers for RQ jobs, ORM models, Redis, scheduled tasks and blinker events; peak memory per request; resolving exception families.
Upcoming in v0.3:
Out of scope: distributed tracing or APM (use OpenTelemetry), aggregated metrics, and alerting.
Development
uv sync # install dependencies and dev tools
uv run pytest # tests
uv run ruff check . # lint
uv run ruff format . # format
uv build # sdist + wheel
See docs/architecture.md for how it works inside.
License
MIT
Metadata
Release files for leeuwenhoek 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| leeuwenhoek-0.1.0.tar.gz | 168.9 kB | With provenance | Details for leeuwenhoek-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| leeuwenhoek-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for leeuwenhoek-0.1.0-py3-none-any.whl |
Total release size: 350.4 kB
Release files / leeuwenhoek-0.1.0.tar.gz
| Download URL | leeuwenhoek-0.1.0.tar.gz |
|---|---|
| Size | 168.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cba629e83611c1733e3006a7795940818deba5f76b8271db6fbfe0813719cb73
|
|
BLAKE2b-256 checksum How to use checksums |
5217c59153cbc194b2955f986aa9f3efc8f40d835fd39ddefb08cb9dbc310ff5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / leeuwenhoek-0.1.0-py3-none-any.whl
| Download URL | leeuwenhoek-0.1.0-py3-none-any.whl |
|---|---|
| Size | 181.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fa4390f484f8a3b55a0effaa91cee7bc1961ca3aa8d1cf2e8dd701f2e30d441f
|
|
BLAKE2b-256 checksum How to use checksums |
be249734587e56b063985690a42a9bf4a537a2442fda2403bd4947ad2882d615
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logSecurity for leeuwenhoek 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for leeuwenhoek-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.