viclix-sdk 0.0.2

Traffic logging (IPs, users, sessions, endpoints) for FastAPI apps, with a built-in viewer, ready for Viclix

viclix-sdk

Traffic logging for FastAPI apps, ready to visualize in Viclix.

Installs an ASGI middleware that records every request (IP, user, endpoint, time, duration, status…) into a dedicated JSONL file. Writing happens on a background thread with the file kept open, so the request path never does disk I/O. Those logs then feed the animated Viclix visualization: who came in, from which IP, to which pages and in what order.

Installation

pip install viclix-sdk

Minimal usage

from fastapi import FastAPI
from viclix_sdk import MiddlewareLogger

app = FastAPI()
app.add_middleware(MiddlewareLogger)

That already creates the .viclix/middleware/ folder and writes traffic-YYYY-MM-DD.jsonl there.

Configuration

app.add_middleware(
    MiddlewareLogger,
    log_path=".viclix/middleware/traffic.jsonl",  # base path (the date is appended)
    trust_proxy=True,                       # real IP from X-Forwarded-For (PythonAnywhere)
    rotation="daily",                       # "daily" or None
    include_query=False,                    # include the query string (careful with sensitive data)
    ignore_paths=["/static", "/health"],    # prefixes that are not logged
)

Parameter	Default	Description
log_path	.viclix/middleware/traffic.jsonl	Base path of the log.
trust_proxy	True	Reads the IP from X-Forwarded-For / X-Real-IP.
user_resolver	reads request.state.user	Callable[[scope], id] to extract the user.
rotation	"daily"	Daily file rotation, or None.
include_query	False	Includes the query string in the event.
ignore_paths	[]	Path prefixes to ignore.
track_sessions	True	Anonymous session cookie (viclix_sid) per visitor.
session_cookie	"viclix_sid"	Name of the session cookie.
session_ttl_days	365	Lifetime of the session cookie.

Sessions (trace a visitor's flow)

With track_sessions=True (the default), each visitor gets an anonymous viclix_sid cookie and every event carries a session field. This lets you follow a single visitor's full sequence of pages even without a login — for example, to see the flow that led a user to a 4xx/5xx error:

GET /            -> 200   sess=44ef64b0
GET /checkout    -> 200   sess=44ef64b0
GET /boom        -> 500   sess=44ef64b0   ← the flow that ended in an error

Identifying the user

By default request.state.user is read (whatever your auth layer leaves there). If you store an object, username, email, id, pk are tried in order. For custom logic:

def resolver(scope):
    state = scope.get("state") or {}
    user = state.get("user")
    return getattr(user, "email", None)

app.add_middleware(MiddlewareLogger, user_resolver=resolver)

Rotation and retention (log size)

By default logs rotate hourly, closed files are compressed to .gz and deleted after 24 h:

app.add_middleware(
    MiddlewareLogger,
    rotation="hourly",       # "hourly" (default) | "daily" | None
    retention_hours=24,      # delete files older than this (None = no limit)
    max_total_mb=200,        # hard size cap: deletes the oldest (None = no cap)
    compress=True,           # gzip already-rotated files
)

• retention_hours bounds by time; max_total_mb is the safety net by size (a traffic spike or a bot can fill the disk within the retention window).
• Cleanup and compression run on the background thread at rotation time — never on the request path — and only touch the log's own files.
• The viewer reads .jsonl and .jsonl.gz transparently.

The biggest saving is usually not logging static assets:

app.add_middleware(MiddlewareLogger, ignore_paths=["/static", "/assets", "/favicon.ico"])

Background tasks accept the same parameters:

track_background_tasks(name="checkout", rotation="hourly", retention_hours=24)

SQL queries (SQLAlchemy)

From the outside you see that a request took 2 s; never why. With one line every event starts carrying the detail of what it did against the database:

from viclix_sdk import track_db_queries

track_db_queries()              # every engine in the process
track_db_queries(engine)        # or just one

No query is touched: it hooks SQLAlchemy's before/after_cursor_execute events. It does not write a line per query — that would multiply the volume. It aggregates per request (and per background task) and attaches it as the db field of the event that was already being emitted:

"db": {"n": 21, "ms": 312.4, "slow": 2,
       "slowest_ms": 40.1, "slowest": "SELECT id FROM projects",
       "repeat": {"sql": "SELECT name FROM projects WHERE id = ?", "n": 20, "ms": 180.2}}

repeat is the N+1 detector: the same query —normalized, without values— repeated many times within the same request. In the viewer, endpoints with N+1 are flagged with ⚠ and the average number of queries.

Only the text of the statement is stored, never the parameters: in SQLAlchemy the values travel separately (bind params), so no data leaks.

Options: slow_ms (slow-query threshold, 100 by default), repeat_threshold (repeats needed to flag N+1, 5) and max_distinct (cap on distinct queries tracked per request, 200).

Exceptions

A 500 seen from the outside is an identical black box every time. The middleware captures unhandled exceptions —the only place where the traceback still exists— and adds them to the event, attributed to the endpoint, the IP and the user:

"error": {"type": "ValueError", "msg": "something broke badly",
          "where": "routes.py:142 in checkout", "traceback": "Traceback ..."}

Enabled by default (capture_errors=True) and it does not alter the flow: the exception is re-raised as-is. With error_traceback=False only the type, message and file:line are stored. Background tasks that fail log the same in their own log (and are not duplicated in the request that created them).

No local variables are captured, only the formatted traceback.

External services (httpx / requests)

From the outside you can't attribute "checkout was slow because Stripe took 3 s": the proxy only sees a slow request. With one line:

from viclix_sdk import track_outbound_calls

track_outbound_calls(slow_ms=500)

Wraps httpx.Client.send, httpx.AsyncClient.send and requests.Session.send. Like the SQL one, it aggregates per request/task and travels as the out field:

"out": {"n": 2, "ms": 3200.5, "slow": 1,
        "hosts": [{"host": "api.stripe.com", "n": 1, "ms": 3000.2}],
        "slowest": {"host": "api.stripe.com", "method": "POST",
                    "path": "/v1/charges", "ms": 3000.2, "status": 200}}

From the URL, host and path are kept; the query string is dropped because it usually carries tokens and API keys. In the viewer there is a port (🛰) where the little person waits as long as each third party took.

With streaming responses the timing reaches the headers, not the last byte of the body.

Process health

This one does go to its own log (.viclix/health/<name>.jsonl), because it doesn't hang off any request: it's periodic sampling.

from viclix_sdk import track_health

track_health(name="www", interval=15)

{"kind": "health", "app": "www", "loop_lag_ms": 3.4,
 "rss_mb": 182.5, "threads": 12, "in_flight": 2,
 "pool": {"in_use": 3, "size": 5, "overflow": 0}}

The key figure is loop_lag_ms: how long the event loop takes to attend to something that was already ready. From the outside you only see "high latency"; the lag tells you there's a synchronous call blocking the async loop. It's measured from the sampling thread with loop.call_soon_threadsafe, so it takes no room in the loop itself.

rss_mb uses psutil if installed, otherwise /proc/self/statm (Linux). pool comes from the first SQLAlchemy engine seen in use, so it needs track_db_queries() active. In the viewer this is the city's weather: it clouds over and rains when the process is suffering.

Hung requests and who is blocking the loop

Two parameters of track_health, no new lines:

track_health(name="www", stuck_after=30, profile_on_lag=200)

stuck_after (seconds) reports requests that are still in flight. It's the only way to see a request that never finishes: the normal event is written when it ends, so a deadlock or a third party with no timeout left no trace at all. They show up in the health sample while they're still hung:

"stuck": [{"method": "GET", "path": "/pay/charge", "ip": "1.2.3.4", "age_s": 47.2}]

profile_on_lag (ms) takes the snapshot of the stacks when the loop stalls. The trick is in the ordering: the probe waits only that long and, if the ping hasn't come back, the loop is blocked right now — that's the moment to photograph. Waiting for it to return would give the stack of the loop already at rest, which says nothing.

"blocked_by": "billing.py:61 in summarize",
"stacks": [{"thread": "MainThread", "stack": ["...", "billing.py:61 in summarize"]}]

Cache (Redis)

from viclix_sdk import track_cache
track_cache()               # redis.Redis and redis.asyncio.Redis
track_cache(my_client)      # or just that client

cache field on the event: {"n": 10, "hits": 8, "misses": 2, "ms": 4.1}. From the outside you can't tell a cached response from a recomputed one. Only the command is counted, never the key or the value.

Authentication and security

from viclix_sdk import track_security
track_security(name="www", login_paths=["/login", "/signup"])

Its own log (.viclix/security/<name>.jsonl), 30-day retention. Detects without touching your code: unauthorized (401), denied (403), login.failed and login.ok on the login paths.

The important part is that it can be grouped by account: an attacker rotating IPs against a single user is invisible grouped by IP and obvious by account.

Honest limit: from the middleware only the status code is visible. The reason (wrong password vs. locked account vs. expired token) is only known by your auth layer. For that there's an optional call wherever it matters:

from viclix_sdk import record_auth
record_auth("login.failed", user=email, reason="bad_password")

When used, the automatic detection stays quiet for that request, so the same attempt isn't logged twice.

Startup, shutdown and deploy

from viclix_sdk import track_lifecycle
track_lifecycle(name="www", version=settings.release, config=settings)

Its own tiny log: one or two lines per deploy. From the outside a redeploy and a crash-loop look identical; here they're told apart because a start without its preceding stop means the previous process died dirty (SIGKILL, OOM). No signal handlers are installed, so as not to step on uvicorn's.

From the config, the names of non-sensitive keys and a fingerprint of all values are stored — never the values. If the fingerprint changes between two starts of the same version, someone touched the configuration.

Response sizes

Runs on its own, with no configuration: a bytes field on every event. Detects the endpoint returning 4 MB of unpaginated JSON. Disable with MiddlewareLogger(track_bytes=False).

Unlike the rest of this README, this one isn't exclusive to the inside: your proxy sees the bytes too. It's here because it's free.

Viewer inside your app (no CLI)

Two lines and you have the viewer on your own domain, without starting anything:

from viclix_sdk import mount_viewer
mount_viewer(app, "/_viclix", token=settings.viclix_token)

You get yourdomain.com/_viclix and yourdomain.com/_viclix/experimental.

Only your admins, no token

mount_viewer(app, "/_viclix", authorize=my_admin_check)

authorize receives the ASGI scope and returns True/False; from there you read the session cookie just like in user_resolver. If you pass both, whoever satisfies either gets in.

What it does for security

• The token disappears from the URL. When you arrive with a valid ?token=… it's stored in an HttpOnly; SameSite=Strict; Path=<viewer> cookie and you're redirected to the same path without the query. The token appears once and is not left in history, nor in the Referer of later requests, nor in the proxy logs.
• It responds 404, not 401. Whoever lacks the key doesn't know the viewer exists.
• It fails closed. With no token and no authorize, or with a token shorter than 16 characters, mount_viewer raises ValueError at startup instead of serving the logs to anyone. Generate the token with secrets.token_urlsafe(32).
• Constant-time comparison (hmac.compare_digest).
• Failed attempts are logged as viewer.denied in the security log, if track_security() is active.
• The viewer does not log itself: browsing it does not generate the events you're looking at.

What it does NOT do

It doesn't encrypt anything or rate-limit attempts. And above all: these logs carry emails, tracebacks, SQL statements, IPs and config key names. Mount it only behind HTTPS and treat it like an admin panel, because that's exactly what it is.

Multiple apps (same venv / server)

If you have several FastAPI apps in the same directory/venv, give each one a distinct name and each writes to its own file (.viclix/middleware/<name>.jsonl), instead of sharing a single one:

app.add_middleware(MiddlewareLogger, name="checkout")   # → checkout.jsonl
app.add_middleware(MiddlewareLogger, name="auth")       # → auth.jsonl
track_background_tasks(name="checkout")                 # tasks separated too

In addition, every event carries an app field. In the viewer:

viclix-monitor                  # aggregates ALL apps (tells you which ones)
viclix-monitor --app checkout   # only that app

Without name, all share traffic.jsonl (previous behavior).

Viewer (CLI)

The package installs the viclix-monitor command. Running it in your terminal:

viclix-monitor

1. Reads the logs from .viclix/middleware/ (all days aggregated). If that folder doesn't exist, it searches recursively (up to 3 levels) as a fallback.
2. Starts a local server and opens the browser with an interactive visualization.
3. Shows stats, a timeline per session/IP (with traffic animation), the most active endpoints and IPs, and a table filterable by IP, path, user or status class.

Living world (experimental)

At /experimental (linked from the dashboard) there's an isometric representation where each IP is a little person who leaves the "entrance" and walks toward the endpoint they requested. Endpoints are grouped by class (first path segment) into districts — /products, /products/42 → products district — and the map starts at 0 and grows as IPs and endpoints are discovered. Root-level files probed by scanners (/config.json, /.env, /.github, /values.yaml…) all fall into a single root-files district (labeled / archivos) instead of creating one district per probe. The pulse color indicates the status class of the last request (green/yellow/orange/red). While walking, each person carries a thread hooked to their shadow: dotted for what they've already walked, solid with an arrowhead for what's left. The color says the errand — neutral toward an endpoint, green to the warehouse, amber if that trip is an N+1, blue to the third-party port. Toggle it with the ➶ button next to the flow arrows.

Map controls: play, speed, drag to move, wheel to zoom, Fit to reframe. The character engine is POKER's (sprites.js).

The speed control has two halves with 1× right in the middle. 1× is real time (one second of log = one second of wall clock). The left half goes from 0.1× to 1× in 0.1 steps (slow-motion, to inspect a burst with a magnifying glass); the right half from 1× up to 1000× in 1-2-5 steps, to fast-forward through long logs (a full day plays in ~86 s at 1000×).

Live mode (🔴): follow the log in real time. The SDK polls for new events and appends them to the world without resetting it, keeping the playhead pinned to now. Taking manual control (seek, apply a range, reset) turns it off.

Time panel. Drag it by the ⠿ handle and it stays where you leave it (remembered in localStorage); double-click the handle to re-center it.

Over the bar:

gesture	effect
click / drag	jump to that instant
wheel	zoom anchored to the cursor: the instant you point at doesn't move and the bar stretches around it
Shift + drag	pan the visible window
double click	back to 1× zoom
⟚ + drag	select a range (loops within it and proposes that range below)

The zoom only stretches the bar: useful to see bursts of seconds within a whole day. When zoomed in, a rail appears on top showing which slice of the total you're viewing, and the window follows the playhead while playing.

Load range (bottom row): day + from-hour to-hour, 24 h maximum. This is not a zoom: it actually trims the simulated window — it recomputes the world, the tasks, the security events and the clock. If to is less than from, it crosses midnight (22:00 → 06:00 = 8 h). The counter next to it previews how many requests fall in the range before applying it, and an empty range is not applied: it warns you. Todo returns to the full log.

Counters with lists. The counters at the bottom left (IPs, districts, endpoints, requests, jobs and the weather) open, on hover, the list of what they count; a click on the counter pins it until the next click.

menu	what it lists	on clicking an item
IPs	each IP with the accounts it has used (or anonymous), its origin, whether it's a bot and whether it has auth rejections	opens its card and the camera follows it
Districts	the neighborhoods with their endpoints, hits and errors	district card (aggregate of the whole neighborhood)
Endpoints	by hits, with status, errors and SQL load (⚠ if N+1)	endpoint card and the camera goes to it
Requests	the last 30 fired, with IP, user, duration, SQL and exception	jumps to whoever made it
Jobs	the latest runs and, below, the task types	the run's card or the bench's
Weather	loop lag, memory, in-flight, threads, pool, who blocked and the hung requests	process card (the hung ones jump to their IP)

The same in the ranking on the right (endpoints, IPs, errors and origins — an origin filters the world) and inside the cards: the endpoints listed in an IP, a district or the warehouse are links.

This view is self-contained and public: it's the basis of what will later be shown in Viclix.

Options:

viclix-monitor [LOG]         # path to a specific .jsonl (optional)
  --dir .                    # root search directory
  --depth 3                  # maximum search depth
  --port 8787                # local server port
  --all-days                 # aggregate all daily files of the same base
  --no-browser               # don't open the browser automatically

Log format (JSONL)

One JSON line per request:

{"v":1,"ts":"2026-07-24T16:50:55.349029+00:00","ip":"1.2.3.4","method":"GET","path":"/dashboard","status":200,"duration_ms":3.06,"user":"[e-mail]","ua":"Mozilla/5.0","referer":null}

Field	Description
v	Schema version.
ts	ISO-8601 timestamp in UTC.
ip	Client IP.
method	HTTP method.
path	Requested path.
status	Response code.
duration_ms	Request duration in ms.
user	User identifier (or null).
session	Anonymous session ID (viclix_sid cookie).
ua	User-Agent.
referer	Referer.
query	Query string (only if include_query=True).

Performance notes

• Events are queued in memory and written by a daemon thread; the request does not wait on disk.
• If the queue saturates (queue_maxsize, 10k by default), new events are dropped instead of blocking the app. The writer keeps written and dropped counters.
• The file is flushed every second and closed cleanly when the process ends (atexit).