UniSky

UniSky turns heterogeneous astronomical survey data — FITS images on arbitrary WCS grids, catalogs, contours, DS9 regions, and strong-lensing models — into an interactive, WCS-aware web map, in the spirit of the DESI Legacy Survey viewer and as a more capable replacement for FitsMap.

Two commands define the product, and two more keep a built site current without paying for it twice:

unisky build config.yaml -o build/   # ingest data, emit a self-contained site
unisky serve build/                  # serve the static site + science API
unisky add config.yaml -o build/ ID  # build ONE more layer into that site
unisky remove -o build/ ID           # ...and take one back out again

build uses several cores by default (-j N to choose, -j 1 for a strictly serial build, -j 0 for all of them) — see Parallel builds.

A fifth builds nothing and reads no data. It shows — and can rewrite — what the deployment serving a site declares it can offer, which is two booleans in manifest.json and nothing else. See Deployment:

unisky capabilities build/           # show it; --no-cutout / --static set it

Features

• Client-side stretch: image tiles are quantized 16-bit data tiles; black/white point, stretch function (linear, sqrt, log, asinh, power) and colormap change instantly in the browser — no tile refetch, no server.
• Any WCS in, one map out: images on TAN/SIN/CAR/… grids are reprojected onto a common gnomonic display grid for display only; science operations always use the native pixels.
• Native-grid cutouts: the cutout API returns FITS whose pixels are bit-identical to the source file, with correctly shifted WCS.
• Rich catalogs: configurable popups, join a second table (e.g. photo-z onto photometry), color markers by any column, per-source cutout links.
• Contours & regions: σ-scaled contour levels traced in the native pixel grid; DS9 region files rendered as vector overlays.
• Strong lensing, live: critical curves and caustics (separately switchable) with a source-redshift slider, magnification at the cursor, counter-image prediction, κ and μ as map layers (glafic-convention models). Give the model a lensing potential (maps.psi, glafic's _pot dump) and every predicted counter-image is labelled with its time delay relative to the first image to arrive.
• Composite RGB base layers: each channel is one band or a weighted sum of several (g: 0.5*f322w2 + 0.5*f150w2), combined one at a time or by Lupton's rule — written in exactly the grammar the viewer's RGB panel reads and prints, so a composite designed in the browser can be pasted into the config and built.
• URL-hash permalinks, coordinate readout and goto box (decimal or sexagesimal), optional basic auth.
• Incremental builds: unisky add puts one more layer into a finished site without touching the rest of it, reusing the display grid the site was built with and refusing rather than move it; unisky remove takes one back out, deleting exactly what the manifest says that layer owned. The two are a byte-for-byte round trip — see Adding and removing layers of a built site.

Quickstart

Requirements: Python ≥ 3.10, and nothing else in order to use UniSky — the built viewer is committed under src/unisky/static/ and ships inside the wheel, so this section needs neither Node ≥ 18 nor just. Testing and Development further down need both.

git clone <this-repo> unisky && cd unisky

# NAME the interpreter, and let the CHECK be the thing that fails. A bare
# `python3` is 3.9 on stock macOS and on several LTS distributions, and a 3.9
# venv builds fine - it fails later, about setup.py, for no visible reason.
PY=python3.12   # 3.10, 3.11, 3.13 ... any of them; `python3` only if it is one
"$PY" -c 'import sys; sys.exit(0 if sys.version_info >= (3, 10) else f"{sys.executable} is Python {sys.version.split()[0]}, and UniSky needs >= 3.10")' && "$PY" -m venv .venv && source .venv/bin/activate

pip install -U pip     # a venv from an old interpreter bootstraps an old pip
pip install -e ".[dev]"

# Prove the install on the committed test field: no data volume, no network,
# a few seconds, ~3 MB. If this builds and serves, UniSky works here.
unisky build tests/fixtures/rxcj2211_tiny/config.yaml -o build-tiny
unisky serve build-tiny --host 127.0.0.1 --port 8000

Then open http://127.0.0.1:8000/.

Why the version check is a command, and why pip install -U pip is there. Neither line is decoration. Skip both, and Python 3.9 does not fail on the version at all: the venv bootstraps pip 21.2.4, which cannot do an editable install of a project that has no setup.py, so what you get is File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode, followed by editable mode currently requires a setuptools-based build. That sends a reader off to install setuptools or write a setup.py, and never mentions Python, because requires-python = ">=3.10" is not consulted on that path at all. Upgrade pip and it is consulted: Package 'unisky' requires a different Python: 3.9.6 not in '>=3.10', which is the truth. The check on the line above only says it earlier, before there is a venv to throw away. (pip's exact words move between releases; they are here because they are what you would paste into a search engine.)

The bundled full field is the real thing, and needs the real data:

# ~10-40 min, ~2-4 GB. Needs the RXCJ2211 data volume mounted AND readable,
# and network for the handful of layers the config downloads. With no
# readable volume it refuses in seconds rather than building half a site.
unisky build examples/rxcj2211.yaml -o build-rxcj2211
unisky serve build-rxcj2211 --host 127.0.0.1 --port 8000

For your own data, copy examples/rxcj2211.yaml and edit: every layer is a type: + paths + options entry; grid, stretch limits and contour σ are derived automatically when omitted.

Keep that environment activated for everything below — the just recipes call plain python and pip, and the E2E suite runs the unisky console script.

To work on the viewer itself, just setup adds the frontend dependencies (npm install), and every change under frontend/ is rebuilt with just build-fe — never a bare npm run build, which empties src/unisky/static/ and leaves it without the freshness stamp the test suite checks. See docs/frontend-bundle.md.

Parallel builds

Layers are independent, so unisky build builds several at once:

unisky build config.yaml -o build/          # default: see "Choosing -j" below
unisky build config.yaml -o build/ -j 4     # exactly 4
unisky build config.yaml -o build/ -j 0     # every core
unisky build config.yaml -o build/ -j 1     # strictly serial

-j N is a total core budget, not just a layer count: images, catalogs, contours, regions and lens models run in parallel processes (a composite waits only for its three channels), and whatever the layers cannot use — a single-layer config, or the tail of a build — goes to threads inside a layer, parallelizing the reprojection bands and tile encoding.

The output does not depend on -j. Every choice writes byte-identical tiles and the same manifest, whose layers stay in config order however the build finishes; the test suite asserts this by building the committed real-data fixture at -j 1 and -j 4 and comparing every file's hash.

Choosing -j

The default is the smaller of 75% of the cores and as many layers as fit in half of RAM (~1.5 GB per concurrent layer), so it stays sane on a 64-core box with 64 GB as well as on a laptop. Measured on a 10-core / 32 GB machine building a 12-layer config (examples/rxcj2211_core.yaml as it stood then — it has since grown to 18 layers) on a 4000×4000 grid, 2805 tiles, median of 3 runs:

-j	wall time	speedup	peak RSS (whole process tree)
1	68.4 s	1.00×	2.1 GB
5	29.4 s	2.32×	5.2 GB
6	24.8 s	2.75×	5.3 GB
7	22.5 s	3.04×	5.9 GB
8	22.8 s	3.00×	6.6 GB
10	22.7 s	3.02×	7.1 GB

Wall time is flat past -j 7 (= the default here) because the tail of a build is a single long layer — a composite waiting on its three channels, or a lens model — and no number of jobs shortens it. So the extra jobs buy nothing measurable while costing RAM and the last free cores; raise -j above the default only for a config with many independent image layers.

Memory scales with concurrent layers rather than with cores. On the real 14435×19223 production grid, five full-resolution JWST layers at -j 5 peaked at 7.4 GB across the whole process tree (~1.5 GB per layer). One layer is a hard floor that -j cannot lower: on the same field a composite alone peaks near 8.5 GB whatever -j says, because it holds three full-grid channels plus its RGB stack and is serialized behind its channels anyway. Budget for that floor first and for -j second; lower -j if a build starts swapping. Set TMPDIR to a filesystem with room for the display-grid scratch arrays (one float32 array of the display grid per in-flight layer — 1.1 GB each at production scale — plus one per composite channel until its composite is done).

Adding and removing layers of a built site

A full-field build is minutes and gigabytes. When one more file turns up — a region file, a new catalog, a band that finished reducing this morning — unisky add builds only the layers you name, into the site that is already there:

unisky add config.yaml -o build-rxcj2211 lrd_msa          # a layer you appended
unisky add config.yaml -o build-rxcj2211 f410m            # re-build one layer
unisky add config.yaml -o build-rxcj2211 f410m --dry-run  # just the checks

What you name are layer id:s out of the config, never filenames: lrd_msa is the id given to a newly appended region layer in docs/incremental.md, and f410m is one examples/rxcj2211.yaml already has. (A layer that feeds a composite — every channels: entry — is refused on its own, so that the composite cannot be left showing the pixels its channel used to have.)

Every other layer's tiles are the same bytes afterwards, and the new entries are spliced into manifest.json in config order, not in the order they were added. Measured on a 4000×4000 px field of six JWST layers (2107 tiles, 298 MB), 10-core laptop at -j 4:

	wall time	tiles written
unisky build — the whole site	20.3 s	2107
unisky add — one 1200² layer	2.3 s	61

The display grid is why this is its own command. Layers are reprojected onto one grid derived from the footprint of every layer that has one — the image layers and the prerendered ones — and every tile on disk is addressed by it, so adding one of those to the config can move the grid, which would silently make all those tiles describe different sky. unisky add therefore reads the grid out of the existing manifest.json, reuses it exactly, and refuses rather than build against a config that no longer describes it. The refusal names every way the two disagree, one line each, naming the setting, what the config says and what the site was actually built with — and then gives both ways out: rebuilding the whole site, and pinning the grid, the second as a field:/grid: block already filled in with the site's own numbers, ready to paste into the config. (A grid that follows a reference image cannot be pinned at all, and that case says why instead of offering a block that would not work.)

This README no longer reproduces that text. It was quoted here verbatim once and was wrong inside a fortnight — the message had grown the pin block and moved to the project's four-part refusal layout while the quote sat still — and the rebuild half of it changed again while this paragraph was being written. Ask your own copy instead: one command, and it cannot rot.

unisky add <config> -o <site> <layer-id> --dry-run

--dry-run runs every check and writes nothing, so it prints exactly the refusal a real add would print, against the site you actually have.

It also refuses a layer that misses the grid entirely, and warns — loudly, because it is real data going missing — about one that is clipped by it or is finer than it. Catalogs, contours, regions, footprints and lens models contribute no footprint to the grid at all, so adding those is always safe: the test suite adds one to the real-data fixture and asserts the result is byte-identical to a site built from the complete config in one go.

Taking one back out

unisky remove is the inverse, with the same guarantees — one staging swap, the manifest written last and atomically, and a refusal that has not touched a byte:

unisky remove -o build-rxcj2211 lrd_msa            # delete a layer
unisky remove -o build-rxcj2211 f410m --dry-run    # what exactly would go

It takes no config: what a layer owns, and which composites are rendered from it, are read from the site's own manifest.json — so a layer can leave the site before, after, or instead of leaving the config. Layers you do not name are not touched, and manifest.json is the only file it writes.

It refuses an id the site does not have, a layer a composite still needs as one of its channels (naming every dependent composite, and weighted channels resolved from the recipe the manifest recorded), and the last image layer — which unisky build will not create a site without either. --dry-run lists precisely what would be deleted, down to the file count and the bytes.

The display grid does not shrink. It was derived when the site was built and every tile on disk is addressed by it, so removing even the widest layer leaves it exactly as it was; only a full rebuild re-derives one, and the command says so. Remove a layer and add it straight back and you get the same site byte for byte.

Full story, including how to pin a grid so it can never move again: docs/incremental.md.

Build progress

A full-field build is minutes of work, so unisky build shows what it is doing while it does it. On a terminal (one frame of one run — the shape is what tests/test_progress.py holds down, not these numbers):

████████████████░░░░░░░░░░░░  57%  3/9 layers  9.1s elapsed  ~6.7s left
  f356w              reproject           1/4 ██░░░░░░░░  16%    2.1s
  f444w              tile               6/30 ███████░░░  74%    6.6s
  irac1              stats                2/2 ██████░░░░  67%    1.2s

One in-place bar for the build, one line per layer in flight — its id, its current stage, its progress within that stage and how long it has been running — so with -j 7 all seven concurrent layers are visible at once. Finished layers scroll off as ordinary log lines ( built image 'f200w' in 7.8s), so the scrollback is still a build log, and the run ends with

built 9 layers in 19s (194.7 MB) -> build/
  slowest: f444w 9.0s  f356w 8.3s  f090w 8.2s  f150w 8.1s  f200w 7.8s

The slowest few are the number worth having when tuning -j: they are the tail no extra job can shorten.

Progress is work-weighted, not layer-counted. Layers differ by more than an order of magnitude — a full-resolution JWST layer writes ~1000 tiles where a native-scale-capped IRAC layer writes ~80 — so "3 of 25 layers" would be a bar that stalls and an ETA that lurches. Each layer's cost is estimated before the build from the decimated grid it will actually reproject and tile (the same layer_max_zoom cap described above), from headers the build already reads to derive the display grid; the bar and the ETA are driven off that. Layers whose cost is not grid-sized (catalogs, contours, regions) get one small flat weight.

The ETA waits until it means something. A remaining-time estimate divides by the fraction done, so it multiplies any error in that fraction by (1-f)/f — 99x when a hundredth of the work is in, 9x at a tenth. So it is withheld (eta pending) until a tenth of the weighted work has actually been measured, and the rate is measured from the moment the first worker reported in, not from process start: the seconds spent deriving the grid and spawning the pool are a fixed cost, and charging them to the per-unit rate is what makes an early estimate absurd. On a 41s four-layer build the first number shown is within ~15%, and it stays inside ~10% from half way on.

It degrades to plain lines. Redirect the build and you get a log, not a smear of escape sequences:

$ unisky build config.yaml -o build/ -j 3 > build.log
$ cat build.log
display grid: 524x524 px @ 0.03"/px, max zoom 2
building 9 layers with 3 jobs
  built image 'f090w' in 0.2s
  ...
  built lensmodel 'lens' in 0.5s
built 9 layers (11 map layers) in 1.8s (3.1 MB) -> build/
  slowest: lens 0.5s  f090w 0.2s  f444w 0.2s  f200w 0.2s  mulimg 0.1s

(Nine configured layers, eleven in the manifest: a lens model publishes its κ and μ maps as image layers of their own, and the summary reports both counts whenever they differ.)

No ANSI, no cursor movement, one line per layer. That is automatic whenever stdout is not a terminal (a pipe, a log file, CI) or TERM=dumb; --no-progress forces it on a terminal too. NO_COLOR is honoured separately and means what it says: the live display stays, the colour goes. Lines are clamped to the real terminal width rather than wrapped, and a Ctrl-C, a failing layer or any other exception erases the frame and restores the cursor without swallowing the error.

Nothing about the display reaches the output. Workers report through a multiprocessing queue at a bounded rate and the terminal is redrawn at most ~10×/s; a redirected build installs no reporting sink at all. A build with the live display writes byte-identical files to one with it disabled (there is a test), and on the 9-layer, 3673×3673 synthetic field at -j 3 the two are indistinguishable in wall time (medians 17.19 s vs 17.19 s over 6 alternating runs each).

Testing

just test            # python unit tests
just test-fe         # frontend unit tests
just e2e             # Playwright end-to-end suite on a committed tiny real-data field
just check-fe-fresh  # is the committed viewer bundle the one frontend/ builds?
just check           # rebuilds the frontend bundle, then all of the above

just e2e needs two things just setup does not give it. The unisky console script has to be on PATH — the suite builds and serves the fixture with it, so run it with the project environment activated — and Playwright's browsers have to be downloaded, which is a separate step from npm install:

cd frontend && npx playwright install chromium

Chromium alone, because playwright.config.ts declares no projects and so runs the default one; a bare npx playwright install would also fetch Firefox and WebKit, which nothing here uses. Skip the step and all twelve E2E tests fail together, each with Playwright's own "Looks like Playwright was just installed or updated" banner, and just check fails with them after everything else in it has passed.

UNISKY_RXCJ2211, and what to expect of it

Some tests want the real RXCJ2211 dataset. UNISKY_RXCJ2211 says WHERE that dataset is. Setting it is not a promise that anything will run: pointing at a dataset and being able to read one are different questions.

UNISKY_RXCJ2211=/path/to/RXCJ2211 just test

Each of those tests decides for itself, by opening a file it needs, and skips with a reason naming the path it tried when it cannot. That is a deliberately harder question than "is the variable set", or even "is the file there": a volume can be mounted with stat() allowed and open() denied — macOS does exactly that when a removable volume's privacy permission has been revoked — and on such a machine every path exists and nothing can be read. Setting the variable on such a machine is how this very line used to turn a clean skip into five errors and a failed run.

So what to expect is: just test passes with or without it. What the variable changes is how many tests run, and pytest will say which ones did not, and why:

python -m pytest -rs     # -rs prints one line per skip, with its reason

If the variable is set and the real-data tests skip anyway, that reason line is the answer — usually that the files cannot be opened rather than that they are missing. Point it at a copy this machine can read.

just test also skips ~20 tests without frontend/node_modules. The composite tests that check the builder's colour arithmetic against the viewer's own run the TypeScript side under node; with no npm install in frontend/ they skip, saying so, and only the Python half is covered. just setup installs them; pytest -rs names them if you are unsure.

Tests that reach a real archive are not part of the default run. pytest alone must pass on a machine with no network at all, so the network marker is deselected in pyproject.toml; opt in with python -m pytest -m network. Those really do reach live services, so they can fail for reasons that have nothing to do with this repository — an archive being down is the ordinary case there, not a regression. See docs/remote-data.md.

Deployment

A built site is self-contained; the science API is not, and only half of it matters. /api/cutout re-opens the ORIGINAL FITS at request time, by the absolute path in each layer's native block, so it needs the dataset on the serving machine — often terabytes that were never part of the build folder. /api/lens/* reads only the .npy grids the build already wrote inside the folder, so it needs a Python process and nothing else.

Two booleans in manifest.json say which of the two a deployment offers, and three flags choose them at build time without editing the config:

unisky build config.yaml -o site/                # both on: the full server
unisky build config.yaml -o site/ --no-cutout    # has the folder, not the data
unisky build config.yaml -o site/ --static       # --no-cutout --no-lensing

The flags only subtract — your config's serve: block is the ceiling. unisky capabilities shows or rewrites the two on a site that is already built, with no config, no rebuild and not one tile read or written:

unisky capabilities site/                 # with no flags: what does it offer?
unisky capabilities site/ --static        # about to be uploaded to R2
unisky capabilities site/ --lensing       # ...and back up again, a month later

unisky serve --no-cutout lowers a full build for one process and can never raise one; unisky add floors the config against what the site already declares, so rebuilding one layer cannot re-arm something you turned off. The viewer reads the same two booleans, and a feature the deployment cannot serve is absent — no panel, no dock button, no menu item, no keystroke.

Both deployment stories are in docs/deploy.md: a collaboration server (uvicorn under systemd, nginx serving tiles with immutable cache headers and proxying /api, basic auth, rsync updates), and Cloudflare — tiles in R2 because a real field clears Pages' file cap, /api/* on one hostname through a Tunnel to a machine that holds the build folder and not the FITS, the cache headers that keep manifest.json fresh, and Access instead of basic auth.

Development

just --list shows all dev targets (setup, test, test-fe, build-fe, check-fe-fresh, e2e, check, build-example, serve-example).

src/unisky/static/ is a committed build product: unisky build copies it into every site and the wheel ships it, so a frontend change that is not rebuilt reaches every user as the old viewer. Rebuild with just build-fe and commit it in the same commit as the source change — just check-fe-fresh, and the ordinary pytest run, fail when it is stale. The whole story, including what counts as frontend source, is in docs/frontend-bundle.md.

The rest of docs/:

• incremental.md — unisky add and unisky remove: one more layer into a site that is already built, and one back out.
• remote-data.md — layers UniSky downloads for you (Legacy Survey, HiPS, TAP), and the cache they land in.
• composites.md — type: composite: weighted channels and Lupton colour, and the agreement with the viewer.
• prerendered.md — type: prerendered: a colour image made outside the pipeline, placed by borrowing the WCS of the FITS it was rendered from.
• alignment.md — per-layer align: offsets, for mosaics built on astrometric solutions that disagree.
• lensing-normalization.md — how a lens model's maps are rescaled from their own source plane to any other.
• uploads.md — dropping your own region files and catalogues onto the map, read and drawn in the browser.
• deploy.md — the two deployments: a collaboration server (uvicorn, nginx, basic auth, rsync) and Cloudflare (R2, a Tunnel for /api, cache headers, Access) — and serve.cutout / serve.lensing, which say what a deployment offers.

Release history lives in CHANGELOG.md.

Upgrading from AnyMap

This project was called AnyMap. The rename to UniSky is complete — package, command, environment variables, manifest key — and nothing is aliased, so an existing setup needs three things:

pip uninstall -y anymap && pip install -e ".[dev]"   # the console script moved
unisky build config.yaml -o build/                   # rebuild each built site

(-y is load-bearing. Without it pip uninstall stops to ask, and anywhere that is not an interactive terminal — a script, CI, a block pasted into a pipe — it dies on the prompt with an EOFError traceback and exit 2, so the && never installs UniSky.)

The third is the one worth not skipping: merge the download cache. ~/.cache/anymap holds gigabytes of archive downloads (DESI cutouts, HiPS tiles, TAP answers), no code looks at the old path, and re-fetching them means going back to services that are not always up.

# Merge old into new, keeping whichever copy of a file is NEWER, and printing
# one line per file it brings across. Works whether or not ~/.cache/unisky
# already exists, and is safe to run twice.
rsync -a --update --itemize-changes ~/.cache/anymap/ ~/.cache/unisky/

# Then check nothing was left behind before you delete the old one yourself:
diff -rq ~/.cache/anymap ~/.cache/unisky | grep "^Only in $HOME/.cache/anymap" \
  || echo "nothing left in the old cache"

Both trailing slashes are load-bearing: rsync src/ dst/ merges the contents of src into dst, while rsync src dst/ would nest src inside it.

Do not use mv for this — which is what this README said until now. mv ~/.cache/anymap ~/.cache/unisky is right exactly once, on a machine that has never run the new build. The moment ~/.cache/unisky exists — which it does from the first unisky build onwards — mv puts the old directory inside the new one, at ~/.cache/unisky/anymap/, where cache_root() never looks. It silently strands the very data it exists to preserve, and reports success. On the machine this correction was written on, ~/.cache/anymap and ~/.cache/unisky both exist and the old one holds 66 files, 68 MiB, that are in no other copy — measured, not estimated:

# what an `mv` would strand, before you do anything to either directory
diff -rq ~/.cache/anymap ~/.cache/unisky | grep -c "^Only in $HOME/.cache/anymap"

Nothing inside the cache needs converting: the key is the canonical request URL, which the rename did not touch. $ANYMAP_CACHE, $ANYMAP_OFFLINE and $ANYMAP_SCRATCH are now $UNISKY_*. Full detail — including what a pre-rename site does when it reaches the new viewer — is in CHANGELOG.md, and the cache itself is in docs/remote-data.md.

What this file promises, and what it does not

Every command in this file was last run in order, from a fresh clone on a machine with no prior state — the ones that need the full RXCJ2211 volume as far as a machine without it can take them, and the ones whose result depends on the machine both ways. That is not a habit this file came by early: it had been wrong on a real machine four separate times before that pass, and the pass itself turned up two more. They were all the same failure — a line true the day it was written, about something that moved underneath it — so the rules now are:

• A command whose outcome depends on the machine carries its own check. The interpreter check fails naming the version it found; the cache merge ends in a diff that names anything left behind; the E2E section states the two prerequisites that are missing rather than claiming it works.
• Program output is quoted only where a test pins its shape. The progress display, its fallbacks and the build summary are quoted, and tests/test_progress.py holds them to that shape. The refusals are not quoted: nothing pins their wording, and the one that was quoted here rotted inside a fortnight. In its place is the --dry-run that prints the current one.
• An outcome that depends on your permissions is not predicted. UNISKY_RXCJ2211 says where data is; whether a test can open it is the test's question to ask, and this file says so rather than guess the answer.
• A document that cannot be run end to end says which half was. The Deployment commands were run against the committed test field, not a fresh clone's full build; docs/deploy.md ends with a section naming, line by line, what was executed there and what needed an account this machine does not have.

Still trusted rather than checked, and best read as measurements from one machine on one day: the two benchmark tables, and everything said about the full RXCJ2211 field, which no CI ever builds.

License

BSD 3-Clause. See LICENSE.