omnibusx-sdk 2.0.0

No project description provided

OmnibusX SDK

OmnibusX SDK is a Python package for submitting data programmatically to the OmnibusX Enterprise platform.

Features

• Seamless integration with OmnibusX Enterprise APIs
• OAuth2 device flow authentication with token caching
• Chunked file upload with automatic retry logic
• Progress tracking for uploads
• Type-safe interfaces

Installation

pip install omnibusx-sdk

Quick Start

Authentication

from omnibusx_sdk import SDKClient

# Initialize the client
client = SDKClient(server_url="https://api-prod.omnibusx.com")

# Authenticate (opens browser for login)
client.authenticate()

# Test connection
client.test_connection()

File Upload

Upload files to OmnibusX with automatic chunking and built-in progress tracking:

from omnibusx_sdk import SDKClient

# Initialize and authenticate
client = SDKClient(server_url="https://api-prod.omnibusx.com")
client.authenticate()

# Get available groups to find your group_id
groups = client.get_available_groups()
group_id = groups[0].user_group_id  # or specify your group ID directly

# Upload files - progress is displayed automatically with a clean progress bar!
response = client.upload_files(
    file_paths=["/path/to/file1.h5", "/path/to/file2.csv"],
    group_id=group_id
)

# Output (live updating progress bar):
# [1/2] file1.h5:  67%|████████████████          | 67.5M/100M [00:05<00:02, 10.2MB/s]
# ✓ Upload complete! All 2 file(s) uploaded successfully.

print(f"Folder ID: {response.folder_id}")

Features:

• Automatic 5MB chunking for large files
• Clean progress bar showing uploaded size, total size, speed (MB/s), and ETA
• Retry logic with exponential backoff (up to 5 retries)
• Optional custom progress callback for additional handling
• Multiple file upload to the same folder
• Automatic inclusion of user email and group ID headers

Advanced Usage:

# Silent upload (no progress display)
response = client.upload_files(file_paths, group_id=group_id, show_progress=False)

# Custom progress callback for additional handling
def log_progress(progress):
    # Log to file, update database, etc.
    if progress.done_chunks % 10 == 0:
        print(f"Checkpoint: {progress.done_chunks} chunks uploaded")

response = client.upload_files(file_paths, group_id=group_id, progress_callback=log_progress)

Preprocessing Datasets

Preprocess datasets with type-safe, validated parameters:

Option 1: Server-side files (files already on server)

from omnibusx_sdk import (
    SDKClient, PreprocessDatasetParams, BatchInfo,
    Species, SequencingTechnology, SequencingPlatform, DataFormat
)

# Initialize and authenticate
client = SDKClient(server_url="https://api-prod.omnibusx.com")
client.authenticate()

# Get available groups
groups = client.get_available_groups()
group_id = groups[0].user_group_id

# Create preprocessing parameters with SERVER paths
params = PreprocessDatasetParams(
    name="My Dataset",
    description="Dataset description",
    batches=[
        BatchInfo(
            file_path="/data/server/path/data.h5ad",  # Server path
            batch_name="Batch 1"
        )
    ],
    gene_reference_version=111,
    gene_reference_id=Species.HUMAN,  # or Species.MOUSE
    technology=SequencingTechnology.SC_RNA_SEQ,
    platform=SequencingPlatform.ScRnaSeq.CHROMIUM_10X,
    data_format=DataFormat.SCANPY,  # or DataFormat.SEURAT
)

# Submit preprocessing task
task_id = client.preprocess_dataset(params, group_id=group_id)

# Monitor task progress
client.get_task_info(task_id)

Option 2: Local files (upload + preprocess in one step)

# Create preprocessing parameters with LOCAL paths
params = PreprocessDatasetParams(
    name="My Local Dataset",
    description="Dataset from local files",
    batches=[
        BatchInfo(
            file_path="/Users/me/data/sample1.h5ad",  # Local path!
            batch_name="Sample 1"
        ),
        BatchInfo(
            file_path="/Users/me/data/sample2.h5ad",  # Local path!
            batch_name="Sample 2"
        )
    ],
    gene_reference_version=111,
    gene_reference_id=Species.HUMAN,
    technology=SequencingTechnology.SC_RNA_SEQ,
    platform=SequencingPlatform.ScRnaSeq.CHROMIUM_10X,
    data_format=DataFormat.SCANPY,
)

# Upload files and preprocess in one step!
task_id = client.upload_and_preprocess_dataset(params, group_id=group_id)

# Output:
# Uploading 2 file(s)...
# [1/2] sample1.h5ad: 100%|████| 50.0M/50.0M [00:10<00:00, 5.0MB/s]
# ✓ Upload complete! All 2 file(s) uploaded successfully.
# Upload complete! Files uploaded to: /tmp/abc123/
#
# Submitting preprocessing task...
# Preprocessing task submitted! Task ID: task_xyz

# Monitor preprocessing progress
client.get_task_info(task_id)

Supported Values:

• Species: Species.HUMAN, Species.MOUSE
• Technology: SequencingTechnology.SC_RNA_SEQ (only sc_rna_seq for now)
• Platforms: CHROMIUM_10X, CITE_SEQ, SMART_SEQ_2, DROP_SEQ, OTHERS
• Data Formats: DataFormat.SCANPY, DataFormat.SEURAT

The SDK validates all parameters and provides clear error messages for invalid configurations.

Processing Datasets

After preprocessing a dataset, you can trigger the processing pipeline to apply quality control filters and analysis parameters:

from omnibusx_sdk import (
    SDKClient,
    ProcessDatasetParams,
    QCFilter,
    QCFilterRange,
    ProcessingParameters,
    Species,
    SequencingTechnology
)

# Initialize and authenticate
client = SDKClient(server_url="https://api-prod.omnibusx.com")
client.authenticate()

# Option 1: Skip QC filtering (default)
params = ProcessDatasetParams(
    dataset_id="053ace73fcc84de3a6b0b47aaa335312",
    technology=SequencingTechnology.SC_RNA_SEQ,
    skip_processing_pipeline=True,  # QC filters are ignored
    subcluster=False
)

task_id = client.process_dataset(params)
print(f"Processing task ID: {task_id}")

# Option 2: Apply custom QC filters and processing parameters
qc_filter = QCFilter(
    total_rna=QCFilterRange(min=300, max=3093),
    genes_count=QCFilterRange(min=82, max=1980),
    mt_genes_ratio=58
)

parameters = ProcessingParameters(
    doublet_detection_method="scrublet",  # or "none", "doubletfinder"
    normalization_method="2",  # "1", "2", or "3"
    top_highly_variable_genes=2000,
    pca_method="pca",  # or "incremental_pca", "sparse_pca"
    dimensionality_reduction_method="umap",  # or "tsne"
    cell_type_prediction_version=1,
    species=Species.HUMAN,
    well_aggregation_method="none",  # or "mean", "sum"
    platform="10x"
)

params = ProcessDatasetParams(
    dataset_id="053ace73fcc84de3a6b0b47aaa335312",
    technology=SequencingTechnology.SC_RNA_SEQ,
    qc_filter=qc_filter,
    parameters=parameters,
    skip_processing_pipeline=False,  # Apply QC filters
    subcluster=True
)

task_id = client.process_dataset(params)

# Monitor processing progress
task_info = client.get_task_info(task_id)

Supported Technologies:

• SequencingTechnology.SC_RNA_SEQ
• SequencingTechnology.BULK_RNA_SEQ
• SequencingTechnology.SC_ATAC_SEQ
• SequencingTechnology.BULK_ATAC_SEQ
• SequencingTechnology.WELL_BASED_SPATIAL

Working with Tasks

# Get available user groups
groups = client.get_available_groups()
for group in groups:
    print(f"{group.name}: {group.description}")

# Import OmnibusX file
task_id = client.import_omnibusx_file(
    omnibusx_file_path="/path/to/file.omnibusx",
    group_id="your-group-id"
)

# Monitor task progress
task_info = client.get_task_info(task_id)

Examples

See the example files for detailed usage:

• example_simple_upload.py - Quick start for file uploads
• example_file_upload.py - Advanced file upload scenarios with custom callbacks
• example_preprocess_dataset.py - Dataset preprocessing with type-safe parameters

For more examples, see the documentation

API Reference

SDKClient

Methods:

• authenticate(cache_token=True) - Authenticate with OAuth2 device flow and extract user email
• test_connection() - Test API connectivity
• upload_files(file_paths, group_id, progress_callback=None, show_progress=True) - Upload files with chunking, automatic progress display, and required headers
• get_available_groups() - Get list of user groups
• import_omnibusx_file(omnibusx_file_path, group_id) - Import OmnibusX file
• preprocess_dataset(params: PreprocessDatasetParams, group_id) - Preprocess a dataset with server-side file paths
• upload_and_preprocess_dataset(params: PreprocessDatasetParams, group_id, progress_callback=None, show_progress=True) - Upload local files and preprocess in one step
• process_dataset(params: ProcessDatasetParams) - Process a preprocessed dataset with QC filters and analysis parameters
• get_task_info(task_id, interval=5) - Monitor task progress

Note: The SDK automatically includes OmnibusX-Email (from Auth0 authentication) and OmnibusX-GroupId headers in all API requests.

UploadProgress

Fields:

• total_files - Total number of files to upload
• total_chunks - Total number of chunks across all files
• done_files - Number of files completed
• done_chunks - Number of chunks completed
• current_file - Name of the file currently being uploaded