datashader-browser

Run Datashader-powered visualizations entirely in a browser with Python and WebAssembly—without assuming that the complete dataset fits in browser memory.

This is an early, independent project. It is not yet an official HoloViz project and is not affiliated with or endorsed by the Datashader maintainers.

Try the live browser demo. Open streaming-points.ipynb and run all cells.

What works now

The first vertical slice is a bounded-memory point aggregator:

import datashader as ds
import datashader.transfer_functions as tf

from datashader_browser import StreamingCanvas, iter_csv_chunks, iter_parquet_batches

source = iter_csv_chunks(
    "points.csv",
    chunksize=250_000,
    columns=["longitude", "latitude", "value"],
)

canvas = StreamingCanvas(
    plot_width=1000,
    plot_height=600,
    x_range=(-180, 180),
    y_range=(-90, 90),
)
aggregate = canvas.points(source, "longitude", "latitude", agg=ds.mean("value"))
image = tf.shade(aggregate)

# A local Parquet file can yield projected Arrow record batches directly.
parquet_source = iter_parquet_batches(
    "points.parquet", columns=["longitude", "latitude"], batch_size=250_000
)
aggregate = canvas.points(parquet_source, "longitude", "latitude", agg=ds.count())

Only one input chunk and Datashader's fixed-size aggregate buffers are live at once. The implementation calls Datashader's own compiler, so point binning and reductions match native Datashader instead of being reimplemented.

Currently verified reductions are count, sum, mean, min, and max. Explicit x_range and y_range values are required to preserve a true single-pass pipeline.

NYC Taxi real-data verification

The streaming accumulator successfully rendered all 11,842,094 pickup locations in HoloViz's transformed January 2015 NYC Taxi dataset. The 385 MB Parquet file was decoded as 48 Arrow/Pandas batches of at most 250,000 rows; the complete table was never materialized as one DataFrame.

One warm-cache Linux run produced the 1200-by-700 image in 3.24 seconds with 514 MiB peak process RSS. Runtime and memory will vary by machine. Reproduce it with:

uv sync --extra data
uv run python examples/nyc_taxi.py --download

This benchmark validates StreamingCanvas against a real large dataset on native Python. The browser build now includes an experimental PyArrow Parquet reader: open nyc-taxi-browser.ipynb, place the source file in the browser filesystem, and it streams projected Arrow record batches through the same accumulator. Full 385 MB end-to-end browser timing and memory results are still an explicit validation task, not a claim made by this repository.

Browser runtime

The initial deployment target is JupyterLite with the Xeus-Python WebAssembly kernel. Its environment is described in environment.yml and uses the experimental Numba and llvmlite packages from emscripten-forge. The kernel executes in a Web Worker, keeping aggregation off the browser UI thread. The Parquet-capable build is currently about 256 MB because PyArrow brings compiled Arrow and compression dependencies; this is an engineering baseline, not the final delivery architecture. The first visit therefore needs a substantial download, while subsequent use benefits from the browser cache.

Browser verification: streaming-parquet.ipynb has been run end-to-end in the Xeus-Python WebAssembly kernel. It created a 57.2 MB, two-million-row Parquet file in the browser filesystem, streamed it through 20 Arrow batches, checked the final row count, and rendered the shaded aggregate with no browser console errors. The full 385 MB NYC Taxi browser benchmark remains a separate task.

uv sync --extra test
uv run pytest

# Build the browser site (requires the build dependencies below).
uv pip install -r requirements-build.txt
uv run jupyter lite build \
  --XeusAddon.environment_file=environment.yml \
  --contents content \
  --output-dir dist

Large-data strategy

“In the browser” does not mean that an unlimited dataset can be copied into a Pandas DataFrame. The project uses three execution levels:

1. Stream: scan CSV or local Parquet/Arrow batches through a bounded-memory Datashader accumulator. Total row count affects time, not peak aggregate memory.
2. Prune: use Parquet row groups, spatial partitions, and DuckDB-Wasm to read only the viewport and required columns.
3. Tile: for billion-row interactive datasets, fetch spatial partitions or precomputed multiresolution aggregates instead of rescanning the source on every zoom.

See Architecture, Research, and the roadmap for the engineering decisions and current limitations.

Status

This repository is an alpha-stage research implementation. The WebAssembly Numba/llvmlite stack works in current demonstrations but its patches have not yet been fully upstreamed. Production use should wait for reproducible browser builds, compatibility CI, and measured browser memory limits.

License

BSD-3-Clause. Datashader is a separate BSD-3-Clause project owned by its respective copyright holders and contributors.