Skip to main content

Refget

Run pytests

User-facing documentation is hosted at docs.refgenie.org/refget.

This repository includes:

  1. /refget: The refget Python package, which provides a Python interface to both remote and local use of refget standards. It has clients and functions for both refget sequences and refget sequence collections (seqcol).
  2. /refget/seqcolapi: Sequence collections API software, a FastAPI wrapper built on top of the refget package. It provides a bare-bones Sequence Collections API service. It ships in the refget wheel, but its dependencies do not — see Installation. Nothing on the plain import refget path imports this subpackage, so a base install never pays for fastapi/uvicorn/sqlmodel/psycopg2.
  3. /seqcolapi: A thin compatibility shim re-exporting refget.seqcolapi, kept so existing deployments that run uvicorn seqcolapi.main:app (or :store_app) keep working. This shim is deliberately excluded from the wheel — it is importable only from a checkout of this repository (or from a deployment that COPYs the directory), never from pip install refget. Every instruction below therefore uses refget.seqcolapi.main, which works everywhere.
  4. /deployment: Server configurations for demo instances and public deployed instances. There are also github workflows (in .github/workflows) that deploy the demo server instance from this repository.
  5. /test_fasta and /test_api: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
  6. /frontend: a React seqcolapi front-end.

Installation

The base install is deliberately light — a client library and CLI, no web server and no ORM. Everything heavier is an extra, and the extras compose:

Install Adds Use it for
pip install refget — The Python library and the refget CLI: digests, RefgetStore, API clients, compliance. No fastapi, no sqlalchemy.
pip install 'refget[db]' sqlmodel, psycopg2-binary The SQLModel layer: refget.models, refget.agents.RefgetDBAgent, refget admin. A library capability — you can want the ORM without wanting a server.
pip install 'refget[seqcolapi]' fastapi, uvicorn Serving a RefgetStore. uvicorn refget.seqcolapi.main:store_app, refget store serve, refget.seqcolapi.create_seqcol_app. No database of any kind.
pip install 'refget[seqcolapi-db]' both of the above The PostgreSQL-backed service (uvicorn refget.seqcolapi.main:app), i.e. what runs seqcolapi.databio.org.

The importable module path is refget.seqcolapi.main, not seqcolapi.main. The bare seqcolapi package in this repository is a compatibility shim that is not shipped in the wheel; uvicorn seqcolapi.main:store_app resolves only from a checkout of this repository, and raises ModuleNotFoundError: No module named 'seqcolapi' in a pip-installed environment.

The two service extras correspond to the two deployment modes described under Development and deployment: Backend. The store-backed mode is the common case and the cheaper one; it needs no database dependencies at all.

These boundaries are enforced by module structure, not convention: the database code lives in refget/models.py, refget/agents.py and refget/seqcolapi/dbapp.py, and nothing else imports them at module level. The router's response bodies live in refget/response_models.py (plain pydantic) precisely so that serving the API does not require an ORM. Importing a module without its extra raises an error naming the extra to install, rather than a bare ModuleNotFoundError. tests/local/test_import_gating.py is the tripwire.

Deploy to AWS ECS

To deploy the public demo instance, you can either:

  1. Create a GitHub release - This triggers the deploy_release_software.yml workflow, which builds and pushes the Docker image to DockerHub. After that completes, it automatically triggers deploy_primary.yml to deploy to AWS ECS.

  2. Manual dispatch - You can manually trigger either workflow from the GitHub Actions tab.

This builds seqcolapi, pushes to DockerHub, and deploys to ECS.

Testing

Unit tests

pytest

Integration tests (requires Docker)

Integration tests run against an ephemeral PostgreSQL database in Docker:

./scripts/test-integration.sh

This starts the test database, runs tests, and cleans up automatically.

Development and deployment: Backend

Store-backed (no database)

The store-backed seqcolapi uses a RefgetStore (local files) instead of PostgreSQL. This is the simplest way to run the API, and it needs only pip install 'refget[seqcolapi]' — fastapi and uvicorn, no sqlmodel, no sqlalchemy, no psycopg2.

For safe concurrent serving, the store is fully loaded and converted to a read-only store (RefgetStore.into_readonly()) before serving, so HTTP reads borrow immutably across request threads. The refget store serve CLI does this by default; pass --lazy to serve directly from the mutable, lazy-loading store instead (single-reader-oriented, not recommended for concurrent production serving).

Quick start

Requires a checkout of this repository (the script and the demo FASTA files live here):

bash deployment/store_demo_up.sh

This will:

  • Build a local RefgetStore from test FASTA files
  • Run the store-backed seqcolapi with uvicorn
  • Block the terminal until you press Ctrl+C, which cleans up

No Docker or database required.

Step-by-step

  1. Build a store from FASTA files. data_loaders/ is not part of the wheel, so this step needs a checkout of this repository:
python data_loaders/demo_build_store.py test_fasta /tmp/refget_demo_store

From a pip install, build a store with the CLI instead:

refget store init -p /tmp/refget_demo_store
refget store add -p /tmp/refget_demo_store <your.fa>

Use refget store list to browse collections and collection-local sequence names, and refget store match to translate chromosome names between stored collections. See the Refget CLI reference for selectors, examples, and output schemas.

  1. Start the store-backed API. This works anywhere refget[seqcolapi] is installed:
REFGET_STORE_PATH=/tmp/refget_demo_store uvicorn refget.seqcolapi.main:store_app --reload --port 8100

Remote store

To run against a remote (S3) store:

REFGET_STORE_URL=https://example.com/store uvicorn refget.seqcolapi.main:store_app --port 8100

DB-backed (PostgreSQL)

If you need a database-backed instance (e.g., for mutable data, advanced queries), use the DB-backed workflow. This one needs pip install 'refget[seqcolapi-db]'. In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way to get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):

bash deployment/demo_up.sh

This will:

  • populate env vars
  • launch postgres container with docker
  • run the refget service with uvicorn
  • load up the demo data
  • block the terminal until you press Ctrl+C, which will shut down all services.

Step-by-step process (DB-backed)

Alternatively, if you want to run each step separately to see what's really going on, start here.

Setting up a database connection

First configure a database connection through environment variables. Choose one of these:

source deployment/local_demo/local_demo.env # local demo (see below to create the database using docker)
source deployment/seqcolapi.databio.org/production.env # connect to production database

If you're using the local_demo, then use docker to launch a local postgres database service like this:

docker run --rm --name refget-postgres -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD \
  -e POSTGRES_USER \
  -e POSTGRES_DB \
  -e POSTGRES_HOST \
  pgvector/pgvector:pg18

If you need to load test data into your server, then you have to install gtars (with pip install gtars), a Python package for computing GA4GH digests. You can then load test data like this:

PYTHONPATH=. python data_loaders/load_demo_seqcols.py

or, with the CLI (refget add-fasta was replaced by refget admin load):

refget admin load --pep test_fasta/test_fasta_metadata.csv --fa-root test_fasta

Running the seqcolapi API backend

Run the demo seqcolapi service like this:

uvicorn refget.seqcolapi.main:app --reload --port 8100

Running with docker

To build the docker file, first build the image from a checkout of this repository (the build context is the seqcolapi/ compatibility shim, which exists only here):

docker build -f deployment/dockerhub/Dockerfile -t databio/seqcolapi seqcolapi

To run in container:

source deployment/seqcolapi.databio.org/production.env
docker run --rm -p 8000:80 --name seqcolapi \
  --env "POSTGRES_USER" \
  --env "POSTGRES_DB" \
  --env "POSTGRES_PASSWORD" \
  --env "POSTGRES_HOST" \
  databio/seqcolapi

Deploying container to dockerhub

Use the github action in this repo which deploys on release, or through manual dispatch.

Running the frontend

Once you have a backend running, you can run a frontend to interact with it

Local client with local server

cd frontend
npm i
VITE_API_BASE="http://localhost:8100" npm run dev

Local client with production server

cd frontend
npm i
VITE_API_BASE="https://seqcolapi.databio.org" npm run dev

Development with local WASM

The /digest feature uses @databio/gtars for WASM-based FASTA processing. To use a local gtars-wasm build instead of the npm package:

LOCAL_GTARS=../../gtars/gtars-wasm/pkg npm run dev

The LOCAL_GTARS env var should point to the pkg/ directory of a built gtars-wasm package (run wasm-pack build --target web in gtars-wasm to build it).

gtars WASM API Reference

The streaming API handles files of any size:

import * as gtars from '@databio/gtars';
await gtars.default();  // Initialize WASM

// Streaming API (for large files)
const handle = gtars.fastaHasherNew();
gtars.fastaHasherUpdate(handle, chunk);  // Feed Uint8Array chunks
const result = gtars.fastaHasherFinish(handle);  // Get SeqColResult

// Batch API (for small files)
const result = gtars.digestSeqcol(fastaBytes);

Result object:

interface SeqColResult {
  digest: string;           // Collection digest (SHA512t24u)
  names_digest: string;
  sequences_digest: string;
  lengths_digest: string;
  n_sequences: number;
  sequences: Array<{
    name: string;
    length: number;
    alphabet: string;       // dna2bit, dna3bit, etc.
    sha512t24u: string;
    md5: string;
    description?: string;
  }>;
}

Deploying

  1. Ensure the refget package master branch is as you want it.
  2. Deploy the updated secqolapi app to dockerhub (using manual dispatch, or deploy on github release).
  3. Finally, deploy the instance with manual dispatch using the included GitHub action.

Developer notes

Models

The database objects and attributes are represented as SQLModel objects in refget/models.py (requires refget[db]). To add a new attribute:

  1. create a new model. This will create a table for that model, etc.
  2. change the function that creates the objects, to populate the new attribute.

HTTP response bodies are not defined there. They are plain pydantic models in refget/response_models.py, so that refget.router — and therefore the store-backed service — can be imported without an ORM. Put a new response schema there unless it genuinely maps to a database table.

Example of loading reference fasta datasets:

Needs pip install 'refget[seqcolapi-db]' and a configured PostgreSQL connection (see DB-backed (PostgreSQL)):

refget admin load --pep ref_fasta.csv --fa-root $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta

Metadata

Release files for refget 0.13.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for refget 0.13.0
File Size Uploaded Provenance Actions
refget-0.13.0.tar.gz 767.4 kB With provenance Details for refget-0.13.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for refget 0.13.0
File Interpreter ABI Platform Actions
refget-0.13.0-py3-none-any.whl Python 3 none any Details for refget-0.13.0-py3-none-any.whl

Total release size: 920.2 kB

Release files / refget-0.13.0.tar.gz

Download URL refget-0.13.0.tar.gz
Size 767.4 kB
Tags Source
SHA-256 checksum
How to use checksums
0ea9b918b74bc4a1ce312260364e82ffd6ee27b1fa9e7f0fda8cb08e742a6b2b
BLAKE2b-256 checksum
How to use checksums
d05da1794f7ca8150230a4b23ea1177c50504136661d004956b4eb4d0482af08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / refget-0.13.0-py3-none-any.whl

Download URL refget-0.13.0-py3-none-any.whl
Size 152.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
122d6edb2cabc4c684eee33ad52a57e59c9fca0985352168f0f5b0365e309dd2
BLAKE2b-256 checksum
How to use checksums
fd582f2c4b0777b82ce95f423bb45b1bd746d7de4b1cdbd88c15357e1606ad5a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Security for refget 0.13.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for refget-0.13.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page