Skip to main content

LabCAS Backend

Repository containing back-end services and configuration for executing EDRN LabCAS data processing workflows.

Starting the Service

Create a .env file (or specify one with --env) and launch it with labcas-backend. Run labcas-backend --help for more information.

By default, API endpoints are served from the root path, such as /auth. If the service is behind a reverse proxy that keeps a subpath in forwarded requests, set LABCAS_SUBPATH_PREFIX in .env, for example LABCAS_SUBPATH_PREFIX=labcas-backend-data-access-api, to serve endpoints such as /labcas-backend-data-access-api/auth.

Auxiliary File Services

Auxiliary assets (JavaScript, CSV, images, icons, and other non-Solr-cataloged files) can be served from configured filesystem roots via:

GET|HEAD /{subpath}/auxfiles/{service-key}/{relative/path}

Point LABCAS_AUXFILES_CONFIG (or --auxfiles-config) at an INI file. See file-services.ini.example. Each [file-service <key>] section maps a URL key to a filesystem root and a groups ACL (* for public, otherwise |-delimited LDAP group DNs). Path traversal outside a service root is always denied. Pass ?download=true to force Content-Disposition: attachment.

Zipperlab Integration

To test sending queries to Zipperlab, first get your JWT (above) then do:

curl --request POST --verbose --insecure \
    --cookie "JasonWebToken=`cat /tmp/jwt`" \
    --header "Authentication: Bearer `cat /tmp/jwt`" \
    --header 'Content-Type: application/x-www-form-urlencoded' \
    --data 'email=hello@a.co&query=id:Pre-diagnostic_PDAC_Images/City_of_Hope/COH_0171/COH_01710003/DICOM/I883*' \
    https://localhost:8444/labcas-backend-data-access-api/zip

Make sure you have Zipperlab running and set its URL in ~/labcas.properties.

If you want to send file IDs instead, do:

curl --request POST --verbose --insecure \
    --cookie "JasonWebToken=`cat /tmp/jwt`" \
    --header "Authentication: Bearer `cat /tmp/jwt`" \
    --header 'Content-Type: application/x-www-form-urlencoded' \
    --data 'email=hello@a.co&id=FILE1&id=FILE2&id=FILE3' \
    https://localhost:8444/labcas-backend-data-access-api/zip

Loading Solr Data

If you have downloaded backups of Solr data you can reload it generally as follows:

curl --insecure --verbose "https://localhost:8984/solr/collections/replication?command=restore&location=$BACKUP_PATH/collections"
curl --insecure --verbose "https://localhost:8984/solr/datasets/replication?command=restore&location=$BACKUP_PATH/datasets"
curl --insecure --verbose "https://localhost:8984/solr/files/replication?command=restore&location=$BACKUP_PATH/files"

Replace $BACKUP_PATH with the location of the downloaded backups.# LabCAS Backend (Python)

This package is the Python rewrite of the LabCAS Data Access API. It mirrors the existing Java functionality while providing a modern FastAPI-based stack with an extensible directory abstraction, Redis-backed session management, and a pluggable search layer.

Development

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install --editable .
labcas-backend --help

Create a .env file based on env.example before running the service or tests.

Solr Migration

This guide explains how to export data from your old Solr instance (https://localhost:8984/solr) and import it into a newer Solr instance.

The migration is handled by the migrate-solr console script, which is part of the jpl.labcas.backend package. After installing the package, you can use migrate-solr directly.

Prerequisites

  • Python 3.13+ (as required by the package)
  • The jpl.labcas.backend package installed (includes required dependencies like httpx and tqdm)

Quick Start

The migration tool operates in two modes: export (to save cores to files) and import (to load cores from files).

Export Mode

# Export all cores from a Solr instance
migrate-solr --mode export https://localhost:8984/solr

# Export specific cores only
migrate-solr --mode export https://localhost:8984/solr --cores collections,datasets

# Specify backup directory (default: /tmp/solr-backup-<timestamp>)
migrate-solr --mode export https://localhost:8984/solr --backup-dir /path/to/backup

Import Mode

# Import all cores into a Solr instance (from current directory)
migrate-solr --mode import https://newhost:8983/solr

# Import from a specific backup directory
migrate-solr --mode import https://newhost:8983/solr --backup-dir /tmp/solr-backup-20240101-120000

# Import specific cores only
migrate-solr --mode import https://newhost:8983/solr --backup-dir /tmp/solr-backup-20240101-120000 --cores collections,datasets

Your Solr Cores

The LabCAS Solr engine has the following cores:

  • collections - Collection metadata
  • datasets - Dataset metadata
  • files - File metadata
  • userdata - User data

Migration Process

The migration process consists of two phases:

1. Export Phase

  • Connects to your old Solr instance
  • Exports all documents from each core using cursorMark pagination
  • Saves data as JSON files in the backup directory

2. Import Phase

  • Connects to your new Solr instance
  • Imports documents into corresponding cores
  • Commits changes after import

Important Notes

Before Migration

  1. Create cores in new Solr: The new Solr instance must have the cores created first. You can:

    • Copy the core configuration from solr/src/main/resources/solr-home/ to your new Solr instance
    • Or use the Solr Admin UI: http://newhost:8983/solr/admin/cores?action=CREATE&name=<core>&instanceDir=<core>
  2. Schema compatibility: Ensure the schema.xml files are compatible between old and new Solr versions. You may need to update field types or configurations.

  3. Backup first: Always backup your data before migration!

During Migration

  • The tool will prompt you if a core doesn't exist in the target instance
  • Large cores may take significant time to export/import
  • Progress is shown during the process with progress bars

After Migration

  1. Verify data: Check document counts match:

    curl "https://oldhost:8984/solr/collections/select?q=*:*&rows=0"
    curl "https://newhost:8983/solr/collections/select?q=*:*&rows=0"
    
  2. Update configuration: Update your application's LABCAS_SOLR_URL environment variable

  3. Test thoroughly: Run your application tests to ensure everything works

Alternative: Using Solr Backup API

For very large datasets, you might prefer using Solr's built-in backup API:

Export (on old Solr)

# For each core
curl "https://localhost:8984/solr/collections/replication?command=backup&location=/path/to/backup"

Import (on new Solr)

# Copy backup files to new Solr server, then restore
curl "https://newhost:8983/solr/collections/replication?command=restore&location=/path/to/backup"

Note: This method requires filesystem access to both Solr servers and the cores must have identical configurations.

Troubleshooting

SSL Certificate Errors

If you get SSL errors, the tool skips verification by default. Use --verify-ssl if you want to verify certificates.

Core Not Found

If a core doesn't exist in the new instance, the script will prompt you to create it. Make sure to:

  1. Copy the conf/ directory from the old core
  2. Create the core using Solr Admin API or UI
  3. Then continue with the import

Large Datasets

For very large cores (>1M documents), consider:

  • Running exports during off-peak hours
  • Running export and import separately (export first, then import in a separate command)
  • The tool processes documents in batches of 1000 to manage memory efficiently

Memory Issues

If you encounter memory issues:

  • The tool processes documents in batches of 1000
  • Consider migrating one core at a time using the --cores option
  • Ensure sufficient disk space for backup files

Support

For issues or questions, check:

Metadata

Release files for jpl.labcas.backend 2.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for jpl.labcas.backend 2.0.3
File Size Uploaded Provenance Actions
jpl_labcas_backend-2.0.3.tar.gz 608.8 kB Without provenance Details for jpl_labcas_backend-2.0.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for jpl.labcas.backend 2.0.3
File Interpreter ABI Platform Actions
jpl_labcas_backend-2.0.3-py3-none-any.whl Python 3 none any Details for jpl_labcas_backend-2.0.3-py3-none-any.whl

Total release size: 671.4 kB

Release files / jpl_labcas_backend-2.0.3.tar.gz

Download URL jpl_labcas_backend-2.0.3.tar.gz
Size 608.8 kB
Tags Source
SHA-256 checksum
How to use checksums
f9b9c2fba83e549d3a8709a18bc5ae78855d5a11467037e71a402e7041412914
BLAKE2b-256 checksum
How to use checksums
89b9402d0ecce893db08467a5c2cbb68c877f308c938ed6d104da76aee62ddb6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.17

Release files / jpl_labcas_backend-2.0.3-py3-none-any.whl

Download URL jpl_labcas_backend-2.0.3-py3-none-any.whl
Size 62.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
399e08f37c72295aa2c508fe78794a6e20d4c3ed7d490b1933f167f2c1bddd03
BLAKE2b-256 checksum
How to use checksums
4a21fe4a43e09a4eb655a3b543a70f2eaa13babbafa02e56f70d79ae4c9f48ae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.17

Security for jpl.labcas.backend 2.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page