gradexp 0.1.0

Gradient Explorer Client Library

source ./venv/bin/activate

This repo is intended to reimplement some key features of wandb for our own purposes. This is the pythonic frontend for our wandb-for-tensors service.

Repo features: pip install gradexp gradexp login -> opens webpage -> gets token and stores in in permanent context In python file:

import gradexp
gradexp.init("project name")

...

# Automatically starts tracking run id, step, etc
gradexp.log(TODO ... )

...

App-wide features:

1. Local staging + asynchronous streaming When you call wandb.log() in your training script, the client SDK:

collects the metric/media payload locally (in memory and also writes to local run files).

hands it off to a separate streaming thread/process that runs alongside your main process.

this uploader process asynchronously batches and sends events over the network. This avoids blocking your training process.

events are queued in memory and written to disk if needed (e.g., offline mode) and then eventually synced.

if you set WANDB_MODE=offline, nothing is sent to the server until a sync is triggered. This behavior effectively decouples ingestion from your training loop. It’s not “direct write to bucket” from your app — it goes through this upload pipeline first. Weights & Biases Documentation

2. Upload endpoints and storage target On the server side (hosted or self-managed):

incoming streaming events (metrics, history tuples) arrive via W&B’s application backend.

metadata and small event records (like per-step metrics) are stored relationally (MySQL in self-managed reference architecture).

larger blobs (logs, media files, artifacts) are written out to object storage buckets (e.g., S3 or your own BYOB bucket). You don’t stream directly into the bucket in small per-log increments — the backend receives the event first and the service layer persists the data. Weights & Biases Documentation +1

3. How “appending” works Object storage (S3/compatible) isn’t a traditional file system — you can’t literally open and append to an existing file like with a local file. Instead:

W&B will write each piece of logged data as a separate object or part of a structured object.

for metric history exports (e.g., after a run completes), W&B creates Parquet exports and pushes those into the bucket as artifacts/history files.

this is a batch write rather than incremental byte-level append. The UI and service layer then stitch these pieces together logically for history views. (This pattern is common across object-store backed systems — you don’t append bytes to objects at every metric call in practice.) Weights & Biases Documentation

4. Back-end buffering & batching Even though metrics are “live” in the UI, there’s batching:

the SDK batches small updates and flushes over HTTP.

on the server side, those are ingested through API layers and persisted into the database or stored as discrete objects.

visualization updates pull from the latest ingested state. The “real-time” aspect is achieved by frequent flushes and update propagation — not by writing to a monolithic streaming file. W&B’s process model means it tolerates network latency/outages by buffering locally and then syncing when available.