datashader-browser

Run Datashader-powered visualizations entirely in a browser with Python and WebAssembly—without assuming that the complete dataset fits in browser memory.

This is an early, independent project. It is not yet an official HoloViz project and is not affiliated with or endorsed by the Datashader maintainers.

Try the live browser demo. Open streaming-points.ipynb and run all cells.

What works now

The first vertical slice is a bounded-memory point aggregator:

import datashader as ds
import datashader.transfer_functions as tf

from datashader_browser import StreamingCanvas, iter_csv_chunks, iter_parquet_batches

source = iter_csv_chunks(
    "points.csv",
    chunksize=250_000,
    columns=["longitude", "latitude", "value"],
)

canvas = StreamingCanvas(
    plot_width=1000,
    plot_height=600,
    x_range=(-180, 180),
    y_range=(-90, 90),
)
aggregate = canvas.points(source, "longitude", "latitude", agg=ds.mean("value"))
image = tf.shade(aggregate)

# A local Parquet file can yield projected Arrow record batches directly.
parquet_source = iter_parquet_batches(
    "points.parquet", columns=["longitude", "latitude"], batch_size=250_000
)
aggregate = canvas.points(parquet_source, "longitude", "latitude", agg=ds.count())

Only one input chunk and Datashader's fixed-size aggregate buffers are live at once. The implementation calls Datashader's own compiler, so point binning and reductions match native Datashader instead of being reimplemented.

Currently verified reductions are count, sum, mean, min, and max. Explicit x_range and y_range values are required to preserve a true single-pass pipeline.

Interactive HoloViews integration

The optional HoloViews adapter connects Bokeh viewport events to repeatable, bounded Parquet scans:

pip install "datashader-browser[data,holoviews]"

from datashader_browser import ParquetSource
from datashader_browser.holoviews import datashade

source = ParquetSource("points.parquet", batch_size=250_000)
plot = datashade(
    source,
    "longitude",
    "latitude",
    x_range=(-180, 180),
    y_range=(-90, 90),
)
plot

HoloViews and Bokeh supply axes, zoom and pan tools, and dynamic plot updates. On each viewport change, the adapter calls StreamingCanvas and returns a new RGB element. The current PyArrow backend applies viewport predicates within each bounded batch but still scans the file sequentially. Spatially pruned reads remain a data-plane optimization rather than a current performance claim.

Instant zoom with tile pyramids

Rescanning the source on every pan and zoom does not scale in a browser. A tile pyramid stores Datashader's reduction state (counts, sums, minima, maxima) at several resolutions. You build it once with ordinary Python and host it as static files. Each viewport then reads only the few small tiles it needs.

datashader-browser build-tiles nyc_taxi_wide.parq --x pickup_x --y pickup_y \
    --x-range -8256000 -8209000 --y-range 4964000 4990000 \
    --max-level 8 --values fare_amount --workers 8 --out tiles/

import datashader as ds
from datashader_browser import ParquetSource, TilePyramid
from datashader_browser.holoviews import datashade

pyramid = TilePyramid("https://example.org/nyc/tiles")   # or a local folder
datashade(pyramid, agg=ds.mean("fare_amount"),
          fallback=ParquetSource("nyc_taxi_wide.parq"))  # raw scan past the deepest level

Same results as Datashader. Every level is identical to ds.Canvas(plot_width=n, plot_height=n, ...).points(...) over the full extent. Counts, minima and maxima match bit for bit, and sums and means match up to floating-point summation order. The test suite checks every level and reduction, and the NYC benchmark repeats the check on all 11.8 million real rows. This works because the binning reuses Datashader's own arithmetic, and doubling the grid scales it exactly (see Architecture).

Browser results, NYC Taxi (11.8M rows), measured in headless Chrome 150 with the Xeus-Python WebAssembly kernel. Each row is an 800×500 render, and tile times are cold, fetched over HTTP:

The scan columns also required downloading the 367 MiB Parquet file into the browser (2.3 s on localhost). The pyramid needed only manifest.json plus the visible tiles. Building that 9-level pyramid took 24 s natively with 8 threads and produced 25,245 tiles (176 MiB, including the four fare_amount fields). A 7-level build is 3,208 tiles (52 MiB) and takes 6 s. Raw results are in docs/benchmarks/. Reproduce them with examples/benchmark_zoom.py (native) and examples/browser_benchmark.py (browser).

Pyramids support count, sum, mean, min and max on points. Other reductions still use a scan. Try tile-pyramid.ipynb in the live demo.

NYC Taxi real-data verification

The streaming accumulator successfully rendered all 11,842,094 pickup locations in HoloViz's transformed January 2015 NYC Taxi dataset. The 385 MB Parquet file was decoded as 48 Arrow/Pandas batches of at most 250,000 rows; the complete table was never materialized as one DataFrame.

One warm-cache Linux run produced the 1200-by-700 image in 3.24 seconds with 514 MiB peak process RSS. Runtime and memory will vary by machine. Reproduce it with:

uv sync --extra data
uv run python examples/nyc_taxi.py --download

This benchmark validates StreamingCanvas against a real large dataset on native Python. The same file has also been streamed end to end inside the browser: the full 367 MiB Parquet scan of all 11.8 million rows completed in 28.6 s in the Xeus-Python WebAssembly kernel (see the table above). Browser peak memory has not yet been measured.

Browser runtime

The initial deployment target is JupyterLite with the Xeus-Python WebAssembly kernel. Its environment is described in environment.yml and uses the experimental Numba and llvmlite packages from emscripten-forge. The kernel executes in a Web Worker, keeping aggregation off the browser UI thread. The Parquet-capable build is currently about 256 MB because PyArrow brings compiled Arrow and compression dependencies; this is an engineering baseline, not the final delivery architecture. The first visit therefore needs a substantial download, while subsequent use benefits from the browser cache.

Browser verification: streaming-parquet.ipynb has been run end-to-end in the Xeus-Python WebAssembly kernel. It created a 57.2 MB, two-million-row Parquet file in the browser filesystem, streamed it through 20 Arrow batches, checked the final row count, and rendered the shaded aggregate with no browser console errors. The full NYC Taxi file has since been scanned in the browser as well (see the tile pyramid section). interactive-parquet.ipynb is the HoloViews integration example; browser interaction validation is tracked separately from the existing static render.

uv sync --extra test
uv run pytest

# Build the browser site (requires the build dependencies below).
uv pip install -r requirements-build.txt
uv run jupyter lite build \
  --XeusAddon.environment_file=environment.yml \
  --contents content \
  --output-dir dist

Large-data strategy

“In the browser” does not mean that an unlimited dataset can be copied into a Pandas DataFrame. The project uses three execution levels:

1. Stream: scan CSV or local Parquet/Arrow batches through a bounded-memory Datashader accumulator. Total row count affects time, not peak aggregate memory.
2. Prune: use Parquet row groups, spatial partitions, and DuckDB-Wasm to read only the viewport and required columns.
3. Tile (implemented): fetch precomputed multi-resolution aggregate tiles instead of rescanning the source on every zoom.

See Architecture, Research, and the roadmap for the engineering decisions and current limitations.

Progressive rendering

StreamingCanvas.points(..., on_partial=callback, partial_every=4) passes the callback a finalized aggregate of everything scanned so far, so a long scan can show an image that fills in while it runs.

Status

This repository is an alpha-stage research implementation. The WebAssembly Numba/llvmlite stack works in current demonstrations but its patches have not yet been fully upstreamed. Production use should wait for reproducible browser builds, compatibility CI, and measured browser memory limits.

License

BSD-3-Clause. Datashader is a separate BSD-3-Clause project owned by its respective copyright holders and contributors.