Refget

User-facing documentation is hosted at refgenie.org/refget.

This repository includes:

1. /refget: The refget Python package, which provides a Python interface to both remote and local use of refget standards. It has clients and functions for both refget sequences and refget sequence collections (seqcol).
2. /refget/seqcolapi: Sequence collections API software, a FastAPI wrapper built on top of the refget package. It provides a bare-bones Sequence Collections API service. It ships in the refget wheel, but its dependencies do not — see Installation. Nothing on the plain import refget path imports this subpackage, so a base install never pays for fastapi/uvicorn/sqlmodel/psycopg2.
3. /seqcolapi: A thin compatibility shim re-exporting refget.seqcolapi, kept so existing deployments that run uvicorn seqcolapi.main:app (or :store_app) keep working. This shim is deliberately excluded from the wheel — it is importable only from a checkout of this repository (or from a deployment that COPYs the directory), never from pip install refget. Every instruction below therefore uses refget.seqcolapi.main, which works everywhere.
4. /deployment: Server configurations for demo instances and public deployed instances. There are also github workflows (in .github/workflows) that deploy the demo server instance from this repository.
5. /test_fasta and /test_api: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
6. /frontend: a React seqcolapi front-end.

Installation

The base install is deliberately light — a client library and CLI, no web server and no ORM. Everything heavier is an extra, and the extras compose:

The importable module path is refget.seqcolapi.main, not seqcolapi.main. The bare seqcolapi package in this repository is a compatibility shim that is not shipped in the wheel; uvicorn seqcolapi.main:store_app resolves only from a checkout of this repository, and raises ModuleNotFoundError: No module named 'seqcolapi' in a pip-installed environment.

The two service extras correspond to the two deployment modes described under Development and deployment: Backend. The store-backed mode is the common case and the cheaper one; it needs no database dependencies at all.

These boundaries are enforced by module structure, not convention: the database code lives in refget/models.py, refget/agents.py and refget/seqcolapi/dbapp.py, and nothing else imports them at module level. The router's response bodies live in refget/response_models.py (plain pydantic) precisely so that serving the API does not require an ORM. Importing a module without its extra raises an error naming the extra to install, rather than a bare ModuleNotFoundError. tests/local/test_import_gating.py is the tripwire.

Deploy to AWS ECS

To deploy the public demo instance, you can either:

1. Create a GitHub release - This triggers the deploy_release_software.yml workflow, which builds and pushes the Docker image to DockerHub. After that completes, it automatically triggers deploy_primary.yml to deploy to AWS ECS.

2. Manual dispatch - You can manually trigger either workflow from the GitHub Actions tab.

This builds seqcolapi, pushes to DockerHub, and deploys to ECS.

Testing

Unit tests

pytest

Integration tests (requires Docker)

Integration tests run against an ephemeral PostgreSQL database in Docker:

./scripts/test-integration.sh

This starts the test database, runs tests, and cleans up automatically.

Development and deployment: Backend

Store-backed (no database)

The store-backed seqcolapi uses a RefgetStore (local files) instead of PostgreSQL. This is the simplest way to run the API, and it needs only pip install 'refget[seqcolapi]' — fastapi and uvicorn, no sqlmodel, no sqlalchemy, no psycopg2.

For safe concurrent serving, the store is fully loaded and converted to a read-only store (RefgetStore.into_readonly()) before serving, so HTTP reads borrow immutably across request threads. The refget store serve CLI does this by default; pass --lazy to serve directly from the mutable, lazy-loading store instead (single-reader-oriented, not recommended for concurrent production serving).

Quick start

Requires a checkout of this repository (the script and the demo FASTA files live here):

bash deployment/store_demo_up.sh

This will:

• Build a local RefgetStore from test FASTA files
• Run the store-backed seqcolapi with uvicorn
• Block the terminal until you press Ctrl+C, which cleans up

No Docker or database required.

Step-by-step

1. Build a store from FASTA files. data_loaders/ is not part of the wheel, so this step needs a checkout of this repository:

python data_loaders/demo_build_store.py test_fasta /tmp/refget_demo_store

From a pip install, build a store with the CLI instead:

refget store init -p /tmp/refget_demo_store
refget store add -p /tmp/refget_demo_store <your.fa>

2. Start the store-backed API. This works anywhere refget[seqcolapi] is installed:

REFGET_STORE_PATH=/tmp/refget_demo_store uvicorn refget.seqcolapi.main:store_app --reload --port 8100

Remote store

To run against a remote (S3) store:

REFGET_STORE_URL=https://example.com/store uvicorn refget.seqcolapi.main:store_app --port 8100

DB-backed (PostgreSQL)

If you need a database-backed instance (e.g., for mutable data, advanced queries), use the DB-backed workflow. This one needs pip install 'refget[seqcolapi-db]'. In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way to get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):

bash deployment/demo_up.sh

This will:

• populate env vars
• launch postgres container with docker
• run the refget service with uvicorn
• load up the demo data
• block the terminal until you press Ctrl+C, which will shut down all services.

Step-by-step process (DB-backed)

Alternatively, if you want to run each step separately to see what's really going on, start here.

Setting up a database connection

First configure a database connection through environment variables. Choose one of these:

source deployment/local_demo/local_demo.env # local demo (see below to create the database using docker)
source deployment/seqcolapi.databio.org/production.env # connect to production database

If you're using the local_demo, then use docker to launch a local postgres database service like this:

docker run --rm --name refget-postgres -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD \
  -e POSTGRES_USER \
  -e POSTGRES_DB \
  -e POSTGRES_HOST \
  postgres:17.0

If you need to load test data into your server, then you have to install gtars (with pip install gtars), a Python package for computing GA4GH digests. You can then load test data like this:

PYTHONPATH=. python data_loaders/load_demo_seqcols.py

or, with the CLI (refget add-fasta was replaced by refget admin load):

refget admin load --pep test_fasta/test_fasta_metadata.csv --fa-root test_fasta

Running the seqcolapi API backend

Run the demo seqcolapi service like this:

uvicorn refget.seqcolapi.main:app --reload --port 8100

Running with docker

To build the docker file, first build the image from a checkout of this repository (the build context is the seqcolapi/ compatibility shim, which exists only here):

docker build -f deployment/dockerhub/Dockerfile -t databio/seqcolapi seqcolapi

To run in container:

source deployment/seqcolapi.databio.org/production.env
docker run --rm -p 8000:80 --name seqcolapi \
  --env "POSTGRES_USER" \
  --env "POSTGRES_DB" \
  --env "POSTGRES_PASSWORD" \
  --env "POSTGRES_HOST" \
  databio/seqcolapi

Deploying container to dockerhub

Use the github action in this repo which deploys on release, or through manual dispatch.

Running the frontend

Once you have a backend running, you can run a frontend to interact with it

Local client with local server

cd frontend
npm i
VITE_API_BASE="http://localhost:8100" npm run dev

Local client with production server

cd frontend
npm i
VITE_API_BASE="https://seqcolapi.databio.org" npm run dev

Development with local WASM

The /digest feature uses @databio/gtars for WASM-based FASTA processing. To use a local gtars-wasm build instead of the npm package:

LOCAL_GTARS=../../gtars/gtars-wasm/pkg npm run dev

The LOCAL_GTARS env var should point to the pkg/ directory of a built gtars-wasm package (run wasm-pack build --target web in gtars-wasm to build it).

gtars WASM API Reference

The streaming API handles files of any size:

import * as gtars from '@databio/gtars';
await gtars.default();  // Initialize WASM

// Streaming API (for large files)
const handle = gtars.fastaHasherNew();
gtars.fastaHasherUpdate(handle, chunk);  // Feed Uint8Array chunks
const result = gtars.fastaHasherFinish(handle);  // Get SeqColResult

// Batch API (for small files)
const result = gtars.digestSeqcol(fastaBytes);

Result object:

interface SeqColResult {
  digest: string;           // Collection digest (SHA512t24u)
  names_digest: string;
  sequences_digest: string;
  lengths_digest: string;
  n_sequences: number;
  sequences: Array<{
    name: string;
    length: number;
    alphabet: string;       // dna2bit, dna3bit, etc.
    sha512t24u: string;
    md5: string;
    description?: string;
  }>;
}

Deploying

1. Ensure the refget package master branch is as you want it.
2. Deploy the updated secqolapi app to dockerhub (using manual dispatch, or deploy on github release).
3. Finally, deploy the instance with manual dispatch using the included GitHub action.

Developer notes

Models

The database objects and attributes are represented as SQLModel objects in refget/models.py (requires refget[db]). To add a new attribute:

1. create a new model. This will create a table for that model, etc.
2. change the function that creates the objects, to populate the new attribute.

HTTP response bodies are not defined there. They are plain pydantic models in refget/response_models.py, so that refget.router — and therefore the store-backed service — can be imported without an ORM. Put a new response schema there unless it genuinely maps to a database table.

Example of loading reference fasta datasets:

Needs pip install 'refget[seqcolapi-db]' and a configured PostgreSQL connection (see DB-backed (PostgreSQL)):

refget admin load --pep ref_fasta.csv --fa-root $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta