visionary-sdk (Python)

The Python collector for Visionary. It runs inside your web service and writes:

• one row per HTTP request into api_logs (method, route pattern, status, timing, user, redacted headers and bodies, error codes, and how many database and outbound HTTP calls it made),
• the database and HTTP calls of slow or failed requests into spans,
• unhandled and reported exceptions into errors,
• one releases row per deploy (when the release is known), and
• a periodic host / process / runtime sample into system_metrics.

The Visionary server reads these tables for its dashboards. All slow work happens on background threads, so requests are never blocked or slowed. If the database is down, the SDK only writes log lines.

Every Visionary SDK behaves the same way. The contract is sdks/SPEC.md.

Install

pip install visionary-sdk                 # core: stdlib only (D1, SQLite)
pip install "visionary-sdk[system]"       # + psutil: host CPU/memory/network, process CPU/RSS/threads/fds
pip install "visionary-sdk[postgres]"     # + psycopg 3
pip install "visionary-sdk[mysql]"        # + PyMySQL

Requires Python 3.9+. Without the system extra, Linux hosts read every host and process column from /proc instead. On macOS and Windows most of them are NULL; disk usage, thread count, GC and HTTP throughput are still recorded.

Quickstart

First configure a database (see Choosing the database). For example:

export VISIONARY_DATABASE_URL=postgres://user:pass@localhost:5432/app

Django

# settings.py
MIDDLEWARE = [
    "visionary.integrations.django.VisionaryMiddleware",  # first, so it sees the final response
    # ...
]

The middleware calls visionary.init() the first time it loads, so nothing else is needed. It works under WSGI and ASGI, with sync and async views.

• Route: request.resolver_match.route with a leading /, e.g. /users/<int:id>/.
• User: if you don't call set_user, an authenticated request.user is used (pk as the id, get_username() as the handle).

FastAPI / Starlette (any ASGI app)

from fastapi import FastAPI
from visionary.integrations.asgi import VisionaryASGIMiddleware

app = FastAPI()
app.add_middleware(VisionaryASGIMiddleware)
# or, for any ASGI app: app = VisionaryASGIMiddleware(app)

The route is the matched route's path (/users/{user_id}). Streaming responses keep streaming. Only the first VISIONARY_LOG_MAX_BODY_BYTES of a body are copied.

Flask

from flask import Flask
from visionary.integrations.flask import Visionary

app = Flask(__name__)
Visionary(app)            # or: ext = Visionary(); ext.init_app(app)

The route is request.url_rule.rule (/users/<int:user_id>).

Anything else

import visionary

agent = visionary.init()   # reads the environment, starts, flushes at exit

init() never raises. If no database is configured, it logs one warning and returns a no-op agent, and the integrations pass requests straight through.

For explicit control (tests, several agents, custom config):

cfg = visionary.from_env()          # or visionary.Config(...)
cfg.path_prefix = "/api"
agent = visionary.Agent(cfg).start()
app.add_middleware(VisionaryASGIMiddleware, agent=agent)
...
agent.close(timeout=10)             # drains the queue and flushes samples

Choosing the database

The SDK checks these in order:

1. Your own connection. Use this when the app already has a driver:

   import psycopg, visionary
   
   cfg = visionary.from_env()
   cfg.connection_factory = lambda: psycopg.connect(DSN)
   cfg.dialect = "postgres"            # "postgres", "mysql" or "sqlite"
   visionary.init(cfg)
   

   The factory must return a new DB-API connection. Visionary keeps that connection for its own use:

   • PostgreSQL and MySQL: one shared connection.
   • SQLite: one connection per worker thread.

   After an error it reconnects through the factory, and it closes the connection at shutdown. Each batch runs as one transaction. If you hand it an autocommit connection, batches are not atomic.

2. VISIONARY_DATABASE_URL, chosen by scheme:

   URL	Backend
   postgres://user:pass@host:5432/db (or postgresql://)	PostgreSQL via psycopg 3
   mysql://user:pass@host:3306/db	MySQL 8+ / MariaDB via PyMySQL
   sqlite:///absolute/path.db, sqlite:///C:/data/x.db, sqlite://relative.db	SQLite via sqlite3
   d1://	Cloudflare D1, credentials from D1_*

3. Cloudflare D1, when D1_ACCOUNT_ID, D1_DATABASE_ID and D1_API_TOKEN are set. D1_API_BASE overrides the API URL.

Tables and indexes are created at startup from the schema files shipped in the package (visionary/schema/<dialect>/). This happens on the background threads, so a slow database never delays boot. If the database is down at startup, creation is retried before each write.

Configuration

Every setting has an environment variable, and visionary.Config has the same settings as attributes. The full list and semantics are in SPEC.md, "Environment".

Booleans accept true/false/1/0/yes/no.

Telemetry (spans, errors, releases)

Service and release

VISIONARY_SERVICE names the service, so several services can share one database. The release (version) is VISIONARY_RELEASE, else the first of RAILWAY_GIT_COMMIT_SHA, RENDER_GIT_COMMIT, VERCEL_GIT_COMMIT_SHA, HEROKU_SLUG_COMMIT, SOURCE_VERSION, GIT_COMMIT, GITHUB_SHA, K_REVISION; commit hashes are cut to 12 characters. Both go on every api_logs, system_metrics and errors row (spans: service only). When the release is known, one releases row (sdk = python, runtime such as CPython 3.13.1) is written at start; the dashboards mark deploys with it.

Spans

Every call the SDK can see is counted into db_count, db_ms, http_count and http_ms on the request's api_logs row (NULL for a kind of call that is not instrumented). When the request took at least VISIONARY_SPAN_THRESHOLD_MS or answered 5xx, its first VISIONARY_SPAN_MAX calls are also stored as spans rows, in start order. Spans outside a request are stored when they take at least the threshold themselves, with request_id NULL.

The request is found through a contextvar, so spans from sync views, thread pools that copy the context (FastAPI sync endpoints, asgiref, contextvars.copy_context().run) and asyncio tasks all land on the right request.

engine = visionary.instrument_sqlalchemy(create_engine(DSN))

with visionary.span("queue", "publish order.created", target="rabbitmq") as s:
    channel.basic_publish(...)
    s.set_row_count(1)       # optional; s.set_error(...) marks it failed

Database spans are named after the first keyword and main table (SELECT users, INSERT bookings, BEGIN), with the target system/database (postgresql/app, sqlite/app.db). The statement is stored sanitized: string and number literals become ?, IN (?, ?, ...) becomes IN (?), whitespace is collapsed. Parameter values are never stored. HTTP spans are named METHOD host; the detail is the URL without credentials, query or fragment. An HTTP span fails on a transport error or a 5xx answer. A call through requests makes one span, not a second one for urllib3.

The SDK's own writes are never recorded as spans.

Errors

The Django, ASGI and Flask integrations record every exception the framework turns into a 500 (handled = 0) and leave the exception and the framework's response unchanged. 4xx exceptions (Http404, HTTPException, abort(404)) are not errors. To record one you handled:

try:
    charge(card)
except PaymentError:
    visionary.capture_exception()      # or capture_exception(exc)

Inside a request the row carries its request_id, method, route, path and user. type is the class name, qualified with its module unless it is a builtin (ValueError, psycopg.errors.UndefinedTable); stack is traceback.format_exception (the last 16 KB).

fingerprint groups the same error: the first 16 hex characters of sha1(type + "\n" + frames), where frames are the 5 innermost frames of your own code (not the standard library, site-packages or this SDK), each written function@path with the path relative to its sys.path entry (shop/views.py), joined with \n. If no frame is your code, the 5 innermost frames are used; with no frames at all, the message with digit runs replaced by 0. At most 20 rows per fingerprint per minute are kept.

Retention: spans and errors follow VISIONARY_LOG_RETENTION_DAYS; releases are kept.

Users

import visionary

visionary.set_user(user.id, handle=user.email, role="admin")

Call set_user from auth middleware or a view once the user is known. Only id is required. It is contextvar-based, so it works from sync views, thread pools (FastAPI sync endpoints) and asyncio tasks. Outside a request handled by an integration, it does nothing.

visionary.set_route("/custom/{pattern}") works the same way for the route, for routers the integration can't see.

Hooks

cfg = visionary.from_env()

def lookup_user(user_id):
    """Fill in handle/role for ids logged without them. Runs on the writer
    thread, once per distinct user per batch; keep it cheap (cache)."""
    u = USER_CACHE.get(user_id)
    return (u.email, u.role) if u else None

def db_stats():
    """Your pool's stats for the db_* columns. Counters are cumulative;
    Visionary stores per-sample deltas."""
    s = pool.get_stats()                     # e.g. psycopg_pool
    return {
        "total": s["pool_size"], "idle": s["pool_available"],
        "in_use": s["pool_size"] - s["pool_available"], "max": s["pool_max"],
        "acquires": s.get("requests_num", 0), "empty_acquires": s.get("requests_queued", 0),
        "acquire_wait_ms": s.get("requests_wait_ms", 0),
    }

cfg.lookup_user = lookup_user
cfg.db_stats = db_stats
visionary.init(cfg)

If db_stats() leaves out a key, that column is stored as NULL.

With Django 5.1+ PostgreSQL pooling ("OPTIONS": {"pool": ...}) you don't need the hook: when db_stats is unset, the Django middleware sums the stats of Django's pools the same way. Without a pool the db_* columns stay NULL.

What is captured

Each request (api_logs) records:

• Request id: the incoming X-Request-ID if it is at most 128 characters, otherwise a random 32-hex id. It is echoed on the response.
• Timing: start time (2026-10-17T17:34:00.123Z) and duration in ms.
• Request line: method, path, route pattern and query string.
• Status: the final status (500 when the handler raised).
• Client: IP (first X-Forwarded-For hop, otherwise the peer address) and user agent.
• User: id, handle and role.
• Headers: all request headers as JSON.
• Bodies: content types, bodies up to the cap with *_truncated flags, and full body sizes.
• Errors: for status >= 400, the error code and key from the JSON response body.

Bodies are not captured for multipart/*, image/*, video/*, audio/*, application/octet-stream, application/zip and application/pdf.

Streaming responses are counted, and their first bytes are copied without buffering. File responses (FileResponse, Flask send_file) keep their sendfile path, so only their Content-Length is recorded.

Each sample (system_metrics) records:

• Host: CPU, load average (Unix only), memory, disk usage of the working directory, and network rates.
• Process: CPU, RSS, threads and open files.
• Runtime: live Python threads (in the goroutines column), the tracemalloc heap when tracing is on, and GC collections and pause time (via gc.callbacks).
• Database pool: whatever your db_stats hook returns, or Django's PostgreSQL pools when it isn't set.
• HTTP: requests, 5xx responses, requests per second and in-flight requests seen by the integrations. These are counted even when request logging is off.
• Node (node-exporter style):
  • CPU: count, user / system / iowait / steal share.
  • Memory: available and page-cache memory, swap.
  • Disk: total size, inode use, read/write KB/s and IOPS summed over whole disks (partitions skipped), average I/O wait (await), and busy % of the busiest disk.
  • Network: packets/s, errors and drops.
  • TCP: connections established and retransmits.
  • Processes on the host, running and blocked.
  • Context switches/s.
  • This process: read/write KB/s and its open-file limit.

Sources that don't exist on the platform are NULL:

• Linux only: iowait, steal, disk busy, the TCP columns, procs_running / procs_blocked.
• Not on Windows: page cache, inodes and the open-file limit.
• Windows resolution: disk await comes from whole-second counters there, so it is coarse.

On Linux the host, process and node columns are read from /proc when psutil isn't installed, so all of them are filled either way.

Older api_logs and system_metrics tables are upgraded in place at startup: missing columns are added with ALTER TABLE ... ADD COLUMN, and existing rows keep NULL in them.

Redaction

Redaction runs on the writer thread before anything is stored, and all matching is case-insensitive.

• Headers: Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-Api-Key and X-Auth-Token (plus VISIONARY_LOG_REDACT_HEADERS) become [REDACTED].
• JSON keys: password, passwd, secret, client_secret, token, access_token, refresh_token, id_token, api_key, apikey, otp, pin, card_number, cvv and private_key (plus VISIONARY_LOG_REDACT_FIELDS) are redacted in request and response bodies. VISIONARY_LOG_REDACT_REQUEST_FIELDS adds keys redacted in request bodies only.
• Truncated bodies: matching uses a regex, so bodies cut at the cap are still scrubbed.
• Form bodies: application/x-www-form-urlencoded request bodies are scrubbed the same way (password=[REDACTED]).

Operational notes

• Request queue: holds 5000 entries. When it is full, new entries are dropped and the count is logged.
• Request writes: batches of up to 50 are written every 3 s. A failed batch is retried up to 3 times (1 s, then 2 s backoff), then dropped with an error log.
• System writes: samples are written 4 at a time. On failure they are kept (at most an hour's worth) and retried with the next write.
• Retention: runs 60 s after start, then daily, deleting in chunks of 5000.
• Telemetry writes: spans, errors and the release row share the request queue and batches (one transaction, or one D1 batch call, per flush).
• Logging: all messages go to the visionary logger.
• Forking servers: with servers like gunicorn --preload, the background threads are restarted in each worker after fork.