s3view

A fast, lightweight S3 browser for astronomy. It behaves like a file manager — click through prefixes, preview things, hit space bar — but it streams instead of downloading.

Compatible with: FITS, ASDF, movies, standard image formats, PDFs, standard text formats.

Run from command line:

s3view                          # open your default location
s3view s3://bucket/prefix/      # open somewhere specific
s3view --set-start s3://bucket/prefix/    # remember that as the default

It starts a local server, opens your browser and prints a URL. Ctrl-C quits. On first run, with nothing configured, it lists your buckets and lets you pick.

Installation

uv tool install 's3view[all]'

One command, no clone, and nothing added to the environments you work in. [all] brings the preview stack with it — astropy, numpy, Pillow, matplotlib, PyYAML — so every feature is on from the first run. Python 3.10+; no boto3, no web framework, no build step, no npm install.

To try it without installing anything at all:

uvx 's3view[all]' s3://bucket/prefix/

Or use whichever packaging tool you already have:

pipx install 's3view[all]'
pip install 's3view[all]'

Leave the [all] off for a minimal install. botocore is the only hard dependency; everything the extras would have added degrades gracefully, and s3view tells you at launch what is live (thumbnails:on fits:on). The one combination to avoid is an isolated install without the extras: a pipx or uv tool environment cannot see the astropy and Pillow in your normal environment, so FITS, ASDF and thumbnails quietly switch off.

Running from a checkout

For development, or if you would rather s3view used the scientific stack you have already built than carry its own copy of it:

git clone git@github.com:lgbouma/s3view.git /my/preferred/dir/s3view
echo 'export PATH="/my/preferred/dir/s3view/bin:$PATH"' >> ~/.bashrc   # or ~/.zshrc
exec $SHELL

bin/s3view runs the checkout under whichever python3 is first on your PATH. That is the point — it inherits your stack — but it also means that environment needs botocore, and a per-project virtualenv frequently does not even when your usual environment does. If the wrong interpreter wins, name the right one:

export S3VIEW_PYTHON="$HOME/envs/astro/bin/python"

pip install -e /my/preferred/dir/s3view is the same idea with a console script instead of a PATH entry, and edits still take effect immediately.

Optional dependencies

Everything below degrades gracefully, and the launch banner says which of them are live.

Credentials and endpoints

Standard botocore resolution: environment variables, ~/.aws/credentials, AWS_PROFILE, SSO, instance roles. Override per run with --profile and --region. Non-AWS S3-compatible stores work via --endpoint-url (MinIO, Ceph, Cloudflare R2, Wasabi).

Buckets you can read but do not own. S3's ListBuckets returns only the buckets owned by the calling account. A bucket shared with you cross-account through a bucket policy is fully readable yet never appears in that list — a common arrangement for shared project data. s3view therefore adds any bucket it knows you can reach (bookmarked, or the one you are in) to the sidebar and marks it with a ·. To reach one for the first time, either pass it on the command line or press ⌘L and type the s3:// path, then bookmark it with ☆.

What it previews

Array previews share one interface: choose the HDU or array, a stretch (zscale, asinh, log, 99.5%, minmax), a colormap, and a resolution from 256 to 1024 px. The footer always reports what it actually read — e.g. 2.1 s · read ~8.4 MB of 191 MB (4.2%) — so the cost is never hidden from you.

Containers the browser cannot decode (mkv, avi) offer Open in player, which hands the presigned URL to IINA or VLC — still streaming, never downloading.

Configuration

~/.config/s3view/config.json, written on first run. Set start to open somewhere by default:

{
  "start": "s3://your-bucket/your/prefix/",
  "bookmarks": [
    {"name": "your-prefix", "uri": "s3://your-bucket/your/prefix/"}
  ],
  "profile": null,
  "region": null,
  "endpoint_url": null,
  "page_size": 1000,
  "presign_expires": 3600,
  "external_player": "IINA"
}

start accepts any s3://bucket/prefix/; leave it null for the bucket picker. s3view --set-start s3://bucket/prefix/ writes it for you, and ☆ manages bookmarks. No bucket names are baked into the source.

Security

The server binds 127.0.0.1 only, and every API call requires a token generated fresh at startup and carried in the URL it opens. Without this, any web page you happened to have open could quietly read your buckets through localhost. Requests arriving with a foreign Origin header are rejected.

Presigned URLs default to one-hour expiry and are minted only for objects you actually open. Copy URL puts one on your clipboard deliberately — treat it as a password for that object until it expires.

Development

pip install -e ".[test]"
pytest

The suite needs no AWS credentials and no network. It builds synthetic FITS and ASDF files in memory and serves them through a fake S3 that records every ranged read, so the tests can assert on how much was fetched — the property the whole program exists to protect. Both the contiguous and strided read paths are exercised and checked against each other for identical pixels.

CI runs on Python 3.10–3.13 on Linux plus macOS, and a separate job installs only botocore to prove the optional dependencies really do degrade gracefully rather than crashing.

Releasing

Version lives in one place, s3view/__init__.py; pyproject.toml reads it from there. To cut a release, bump it, commit, then:

git tag v0.1.1 && git push origin v0.1.1

The release workflow builds the sdist and wheel, refuses the tag if it disagrees with s3view.__version__, installs the wheel into a clean environment and starts the CLI from it, then uploads to PyPI through trusted publishing — there is no API token anywhere in the repository or its secrets. Running the workflow by hand (workflow_dispatch) does everything except the upload, which is the way to rehearse a release: PyPI never allows a version number to be reused, even after the file is deleted.

To re-record the README animation (needs playwright and ffmpeg, neither of them a runtime dependency):

pip install playwright && playwright install chromium
python tools/record_demo.py            # -> docs/demo.gif

Why it is fast

Video and audio never pass through this program. The page is handed a short-lived presigned S3 URL and the browser's own media stack range-requests it directly, so seeking is cheap no matter how large the file. Measured from inside the browser against a 731 MB mp4, on a ~2.4 MB/s link:

Array images are read by byte range, not downloaded. Both FITS and ASDF are self-describing: a small ranged read of the metadata is enough to compute the exact byte offset of every row of every array. s3view then fetches only the rows the preview needs, in parallel, and bins the columns it already has in memory. For a 200 MB, 4088×4088 float32 detector frame:

Ranged reads bypass botocore's request machinery: the object is presigned once and every range is pulled over a pooled HTTPS connection, because signing 512 separate requests costs more CPU than the transfer costs bandwidth.

Listings are paginated and virtualized. A prefix of 1000 objects renders about 60 DOM nodes; the next page is prefetched in the background while you read the current one. A prefix holding 187 GB across 1000 files opens as fast as an empty one.

Thumbnails are lazy and rate-limited. Only tiles actually on screen are requested, three at a time, cached on disk under ~/.cache/s3view. Array thumbnails in gallery view cost megabytes each, so they sit behind a toggle.

Known limitations

• Array previews sample rows rather than averaging them vertically, so a decimated view of a crowded field aliases. Columns are averaged. Raise the resolution to sample more rows.
• ASDF blocks compressed with lz4 or blosc cannot be read by range and are not supported; uncompressed, zlib and bzip2 blocks are.
• GetBucketLocation is frequently denied on cross-account buckets, so the region is taken from the x-amz-bucket-region header on HeadBucket instead. If both are denied, s3view falls back to your configured default region.
• Chrome will not load video in a hidden or background tab; if a movie sits on a spinner, bring the window to the front.
• Read-only. There is no upload, rename, or delete.