refget 0.10.1

Python client for refget

Refget

User-facing documentation is hosted at refgenie.org/refget.

This repository includes:

1. /refget: The refget Python package, which provides a Python interface to both remote and local use of refget standards. It has clients and functions for both refget sequences and refget sequence collections (seqcol).
2. /seqcolapi: Sequence collections API software, a FastAPI wrapper built on top of the refget package. It provides a bare-bones Sequence Collections API service.
3. /deployment: Server configurations for demo instances and public deployed instances. There are also github workflows (in .github/workflows) that deploy the demo server instance from this repository.
4. /test_fasta and /test_api: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
5. /frontend: a React seqcolapi front-end.

Deploy to AWS ECS

To deploy the public demo instance, you can either:

1. Create a GitHub release - This triggers the deploy_release_software.yml workflow, which builds and pushes the Docker image to DockerHub. After that completes, it automatically triggers deploy_primary.yml to deploy to AWS ECS.

2. Manual dispatch - You can manually trigger either workflow from the GitHub Actions tab.

This builds seqcolapi, pushes to DockerHub, and deploys to ECS.

Testing

Unit tests

pytest

Integration tests (requires Docker)

Integration tests run against an ephemeral PostgreSQL database in Docker:

./scripts/test-integration.sh

This starts the test database, runs tests, and cleans up automatically.

Development and deployment: Backend

Easy-peasy way

In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):

bash deployment/demo_up.sh

This will:

• populate env vars
• launch postgres container with docker
• run the refget service with uvicorn
• load up the demo data
• block the terminal until you press Ctrl+C, which will shut down all services.

Step-by-step process

Alternatively, if you want to run each step separately to see what's really going on, start here.

Setting up a database connection

First configure a database connection through environment variables. Choose one of these:

source deployment/local_demo/local_demo.env # local demo (see below to create the database using docker)
source deployment/seqcolapi.databio.org/production.env # connect to production database

If you're using the local_demo, then use docker to launch a local postgres database service like this:

docker run --rm --name refget-postgres -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD \
  -e POSTGRES_USER \
  -e POSTGRES_DB \
  -e POSTGRES_HOST \
  postgres:17.0

If you need to load test data into your server, then you have to install gtars (with pip install gtars), a Python package for computing GA4GH digests. You can then load test data like this:

PYTHONPATH=. python data_loaders/load_demo_seqcols.py

or:

refget add-fasta -p test_fasta/test_fasta_metadata.csv -r test_fasta

Running the seqcolapi API backend

Run the demo seqcolapi service like this:

uvicorn seqcolapi.main:app --reload --port 8100

Running with docker

To build the docker file, first build the image from the root of this repository:

docker build -f deployment/dockerhub/Dockerfile -t databio/seqcolapi seqcolapi

To run in container:

source deployment/seqcolapi.databio.org/production.env
docker run --rm -p 8000:80 --name seqcolapi \
  --env "POSTGRES_USER" \
  --env "POSTGRES_DB" \
  --env "POSTGRES_PASSWORD" \
  --env "POSTGRES_HOST" \
  databio/seqcolapi

Deploying container to dockerhub

Use the github action in this repo which deploys on release, or through manual dispatch.

Running the frontend

Once you have a backend running, you can run a frontend to interact with it

Local client with local server

cd frontend
npm i
VITE_API_BASE="http://localhost:8100" npm run dev

Local client with production server

cd frontend
npm i
VITE_API_BASE="https://seqcolapi.databio.org" npm run dev

Development with local WASM

The /digest feature uses @databio/gtars for WASM-based FASTA processing. To use a local gtars-wasm build instead of the npm package:

LOCAL_GTARS=../../gtars/gtars-wasm/pkg npm run dev

The LOCAL_GTARS env var should point to the pkg/ directory of a built gtars-wasm package (run wasm-pack build --target web in gtars-wasm to build it).

gtars WASM API Reference

The streaming API handles files of any size:

import * as gtars from '@databio/gtars';
await gtars.default();  // Initialize WASM

// Streaming API (for large files)
const handle = gtars.fastaHasherNew();
gtars.fastaHasherUpdate(handle, chunk);  // Feed Uint8Array chunks
const result = gtars.fastaHasherFinish(handle);  // Get SeqColResult

// Batch API (for small files)
const result = gtars.digestSeqcol(fastaBytes);

Result object:

interface SeqColResult {
  digest: string;           // Collection digest (SHA512t24u)
  names_digest: string;
  sequences_digest: string;
  lengths_digest: string;
  n_sequences: number;
  sequences: Array<{
    name: string;
    length: number;
    alphabet: string;       // dna2bit, dna3bit, etc.
    sha512t24u: string;
    md5: string;
    description?: string;
  }>;
}

Deploying

1. Ensure the refget package master branch is as you want it.
2. Deploy the updated secqolapi app to dockerhub (using manual dispatch, or deploy on github release).
3. Finally, deploy the instance with manual dispatch using the included GitHub action.

Developer notes

Models

The objects and attributes are represented as SQLModel objects in refget/models.py. To add a new attribute:

1. create a new model. This will create a table for that model, etc.
2. change the function that creates the objects, to populate the new attribute.

Example of loading reference fasta datasets:

refget add-fasta -p ref_fasta.csv -r $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta