SORE: Simons Observatory REplicator

SORE is a tool to ensure that a copy of your critical data is stored on all of our HPC machines. You can upload a file to SORE, and within an hour it should be replicated across NERSC, Princeton, and SO:UK. The system works by leveraging S3: we use Versity's S3 Gateway to allow you to upload data to the NERSC filesystem from anywhere in the world, and then a background Globus job ensures data is replicated to the other systems. You can only add and delete data that you have uploaded; no 'edit' functionality exists. You should also not attempt to directly modify data on disk at any of the sites (and this will be impossible anyway).

To upload data, all you need to do is use the sore command-line utility:

sore -i local_file.fits -o testing/2026-01-01/local_file.fits

The -i input file is the path on your local filesystem. The -o output filename is the position in the global namespace that your file will take. Your file will then be uploaded, over the internet, to NERSC, where it will appear at sore/$GITHUB_USERNAME/testing/2026-01-01/local_file.fits. It will shortly be copied to other sites.

You can also upload a directory recursively:

sore -i local_observation -o testing/2026-01-01/local_observation

Each regular file is stored under the output prefix with its relative path preserved. Symlinked files and directories are skipped, and empty directories do not create objects. An input tree containing no regular files is rejected.

Developed with the Codex/OpenAI 5.6 generation models.

Getting set up

To get set up, you will first need to install the soreplicator python package:

uv pip install soreplicator

To authenticate with the system, you will need a username and password. These are secrets and should never be shared with anyone else. To get your username and password, you will need to log into the web interface at https://sore.simonsobservatory.org/setup. Here, you will be able to get your username and password for the sore server. Your username will always be the same as your GitHub username.

sore includes a utility to store these secrets for re-use in a safe location. We recommend using this. To set up sore you should call:

sore --setup

This will allow you to paste in your username and password, which will be saved in your config location (usually ~/.config/sore.json) with 600 permissions.

Deleting Files

Sometimes you will want to remove files from the system, because they are old and no longer used. You can do this with the sore command-line tool:

sore -d testing/2026-01-01/local-file.fits

Which will ask you to input y/n for whether you are sure you want to delete this file in the global namespace. Note that you can only change your own files, not anyone elses. If you are sure and want to avoid the prompt, you can use

sore -d testing/2026-01-01/local-file.fits -f

to force-delete without needing to respond to the prompt.

The same flag deletes an uploaded directory recursively:

sore -d testing/2026-01-01/local-observation

If the path names an existing object, only that object is deleted. Otherwise, SORE deletes every object beneath the matching path followed by /; similarly named paths such as local-observation-old are not affected. Large directories are listed in pages of up to 1,000 objects and deleted with four concurrent requests.

Administrator Information

Setting up sore is easy. We provide a containerized version of the server. The most important thing to realize is that we provide one bucket per user, with the bucket name the same as their GitHub username. This allows us to set bucket-level permissions for each user, allowing them only PUT, GET, and DELETE access to these buckets, with GET permissions for all buckets.

Included with this repository is a nginx configuration file that allows you to mount:

• Versity's S3 Gateway on /
• The provided sore server on /setup.
• Sharing of directories for .well-known challenges.

We authenticate users using the Simons Observatory authentication framework soauth.

To set up sore, you will need an administrator credential (ideally not the root credential) that allows the creation of buckets and users with various levels of permissions. You should set the following environment variables:

• SORE_GATEWAY_USERNAME: the username of the account that can create other accounts
• SORE_GATEWAY_PASSWORD: the password of the account that can create other accounts
• SORE_GATEWAY_REMOTE_LOCATION: the location of the gateway on the open internet
• SORE_GATEWAY_LOCAL_LOCATION: the location of the gateway on the local network
• SORE_ADMIN_GRANT: the soauth grant for administrator priviliges (sore:admin)
• SORE_USER_GRANT: the soauth grant required for users to create accounts (sore:user)

We will only communicate with the gateway on the local network and never over the open internet. The REMOTE_LOCATION is just for providing information in the UI.

Administrator Panels

Within the Web UI, you can see information on all of the user accounts that have been created to date, and the buckets along with their sizes. This information is pulled dynamically from the S3 gateway and is not stored separately for the sore server so the sore server is effectively stateless. You will require the SORE_ADMIN_GRANT to view this page.

Implementation

This repository contains the complete gateway edge, account service, and client:

• docker-compose.yml runs VersityGW with a POSIX directory, sidecar metadata, IAM, its WebGUI, the SORE server, and nginx.
• soreplicator/server/ is a stateless FastAPI service. It authenticates with soauth, creates or rotates a VersityGW account, creates the matching bucket, and maintains a bucket policy that gives every provisioned account read access while preserving owner write/delete access.
• soreplicator/client/ supplies the sore command, including recursive directory uploads that preserve relative paths and skip symlinks. Large files use four concurrent S3 multipart transfers with a terminal progress bar; smaller files use a regular PUT. Both paths write conditionally, so an existing object cannot be overwritten. Configuration is written atomically with mode 600 and insecure remote HTTP endpoints are rejected.
• The /setup/admin view queries users, buckets, object counts, and sizes directly from VersityGW. No credentials or inventory are stored in an application database.

The downstream Globus replication job is intentionally outside this repository's scope; it can consume the gateway's shared POSIX directory independently.

Local stack

The included compose topology is a loopback-only development environment:

cp .env.example .env
# Change the secrets and SORE_DEVELOPMENT_USERNAME in .env first.
docker compose up -d --build

Open the following local endpoints:

• SORE setup: http://sore.localhost:8080/setup/
• VersityGW WebGUI through nginx: http://gateway.localhost:8080/
• S3 through nginx: http://s3.localhost:8080/

The WebGUI is preconfigured to use s3.localhost for S3 and admin.localhost for its admin API. Log in with the gateway administrator values from .env. Persistent objects, versions, IAM data, and metadata are stored under ./data by default. Set SORE_DATA_DIRECTORY to share another host directory.

The compose stack deliberately uses the fixed development identity named by SORE_DEVELOPMENT_USERNAME; nginx publishes only on 127.0.0.1. Development authentication cannot start unless SORE_DEBUG=true. Never expose this mode to a network.

Production server configuration

The Docker image runs as an unprivileged user with a read-only-compatible root filesystem. For production, set SORE_AUTH_MODE=soauth, SORE_DEBUG=false, and provide these settings to the server container:

Setting	Purpose
SORE_GATEWAY_USERNAME / SORE_GATEWAY_PASSWORD	Versity administrator credential
SORE_GATEWAY_REMOTE_LOCATION	Public S3 URL displayed to clients
SORE_GATEWAY_LOCAL_LOCATION	Private S3 URL used by the server
SORE_GATEWAY_ADMIN_LOCATION	Private Versity admin API URL
SORE_GATEWAY_REGION	SigV4 region; defaults to us-east-1
SORE_APP_BASE_URL	Public application URL, including /setup
SORE_AUTHENTICATION_BASE_URL	SOAuth service URL
SORE_APP_ID / SORE_CLIENT_SECRET	SOAuth application credential
SORE_PUBLIC_KEY_FILE	Mounted SOAuth public key path
SORE_KEY_PAIR_TYPE	SOAuth key type; defaults to Ed25519
SORE_ADMIN_GRANT / SORE_USER_GRANT	Required grants

Terminate TLS at nginx or an upstream load balancer, keep the admin API private, replace every development secret, and adapt nginx/nginx.conf to the production hostnames. The .well-known directory is mounted read-only for ACME challenges.

Development and verification

uv sync --extra dev
uv run pytest
uv run ruff check .
npm install
npm run build:css
docker compose config

Tailwind output and htmx are vendored under soreplicator/server/static, so the setup UI does not depend on third-party CDNs. The Tailwind palette is the colorblind-friendly SO palette from socolors.

Releases

GitHub Actions runs Ruff and the test suite on Python 3.11–3.13 for pull requests and pushes to main, and verifies that the committed Tailwind stylesheet is current. To publish a release, update the version in pyproject.toml, merge it to main, and publish a GitHub Release.

The publishing workflow uses PyPI Trusted Publishing instead of a stored API token. Before the first release, configure a GitHub Actions Trusted Publisher for this repository and the pypi environment in the soreplicator PyPI project.