Skip to main content
OSA

OSA Python SDK

The developer toolkit for the Open Science Archive — define metadata schemas, write validation hooks, build ingesters, and deploy conventions.

License Python

Pre-release — APIs will change without notice.


Install

pip install osa-py

Quickstart

A convention defines what data your archive accepts. Here's an example that archives protein structures and runs analysis on the deposited files:

from datetime import date
import httpx
from pydantic import BaseModel
from osa import (
    Schema, Field, Record, Reject, hook, convention, Example,
    IngesterContext, IngesterRecord,
)

# 1. Define the metadata schema
class PDBStructure(Schema):
    __schema_id__ = "pdb-structure"

    pdb_id: str
    title: str
    method: str
    resolution: float | None = Field(default=None, unit="Å")
    deposition_date: date
    molecular_weight: float = Field(unit="kDa")
    chain_count: int

# 2. Validate incoming records
@hook
def validate_structure(record: Record[PDBStructure]) -> None:
    if record.metadata.resolution and record.metadata.resolution > 4.0:
        raise Reject("Resolution too low for reliable analysis")
    if not record.files.glob("*.cif"):
        raise Reject("At least one CIF file is required")

# 3. Extract features from the data
class Pocket(BaseModel):
    pocket_id: int
    score: float
    volume: float

@hook
def find_pockets(record: Record[PDBStructure]) -> list[Pocket]:
    cif = record.files["structure.cif"]
    size_kb = cif.size / 1024
    return [Pocket(pocket_id=0, score=round(size_kb / 100, 2), volume=size_kb)]

# 4. Pull records from an external source
class PDBIngester:
    name = "pdb-rcsb"
    schedule = None
    initial_run = None
    max_file_mb = 50.0

    async def pull(self, *, ctx: IngesterContext, limit=None, session=None):
        # Continuation comes from `session`: the server hands back whatever the
        # previous batch set. Read your start position out of it, and set no
        # session on the final batch — that is what ends the run.
        ids = ["4TOS", "1TIM", "6LU7"]
        start = int((session or {}).get("offset", 0))
        window = ids[start : start + limit] if limit is not None else ids[start:]
        if start + len(window) < len(ids):
            ctx.set_session({"offset": start + len(window)})

        for pdb_id in window:
            entry = httpx.get(f"https://data.rcsb.org/rest/v1/core/entry/{pdb_id}").json()
            file_ref = await ctx.add_file(
                pdb_id, "structure.cif",
                url=f"https://files.rcsb.org/download/{pdb_id}.cif",
            )
            yield IngesterRecord(
                source_id=pdb_id,
                metadata={"pdb_id": pdb_id, "title": entry["struct"]["title"], ...},
                files=[file_ref],
            )

# 5. Register the convention
convention(
    title="Protein Structures",
    description="Experimentally determined protein structures with pocket analysis.",
    version="1.0.0",
    schema=PDBStructure,
    ingester=PDBIngester,
    hooks=[validate_structure, find_pockets],
    files={"accepted_types": [".cif", ".pdb"], "max_count": 5},
    # Docs are mandatory, and an agent reads them to decide whether this
    # archive can answer a question at all: a purpose, at least one worked
    # example, and three distinct trigger questions (a worked example's
    # question counts toward the three).
    purpose=(
        "Look up protein structures by experimental method, resolution and "
        "molecular weight, and the binding pockets found in their coordinates."
    ),
    examples=[
        Example(
            question="Which X-ray structures resolve better than 2 A?",
            query="POST /api/v1/data/pdb-structure@1.0.0/records",
            interpretation=(
                "Filter method = 'X-RAY DIFFRACTION' and resolution < 2.0; "
                "resolution is in angstroms and lower is better."
            ),
        )
    ],
    example_questions=[
        "What is the molecular weight of 4TOS?",
        "Which structures have the largest binding pockets?",
    ],
    when_not_to_use="Not a sequence-search service, and it holds no predicted structures.",
)

Register the convention as a setuptools entry point in pyproject.toml:

[project.entry-points."osa.conventions"]
my_convention = "my_package"

Testing

Test the full convention pipeline end-to-end — the ingester pulls real data, then every hook runs against each record:

osa test --limit 2
Running ingester pdb-rcsb...
  Fetched 2 record(s) (4TOS, 1TIM)

Running hooks...

  4TOS
    ✓ validate_structure
    ✓ find_pockets → 1 Pocket(s)

  1TIM
    ✓ validate_structure
    ✓ find_pockets → 1 Pocket(s)

2 record(s), 2 accepted, 0 rejected

You can also test individual hooks in-process with synthetic data:

from osa.testing import run_hook

run_hook(validate_structure, meta={"pdb_id": "TEST", ...}, files=tmp_path)

If your source has more records than fit in one batch, test the pagination too. run_ingester runs a single batch, so it cannot tell you whether your continuation works — one batch always looks fine. run_ingester_batches drives the loop the way the server does, threading each batch's session into the next, and raises StalledIngesterError if a batch hands back the session it was given and re-emits the same records:

from osa.testing import run_ingester_batches

# PDBIngester has three PDB IDs, so a batch size of one drives three batches.
result = run_ingester_batches(PDBIngester, batch_size=1)
assert len(result.records) == 3
assert len(result.batches) == 3

Local development

Run a full OSA stack locally with Docker:

osa init my-archive
cd my-archive
osa start

This scaffolds a project directory with osa.yaml, .env, and a Docker Compose stack (Postgres, OSA server, docker-socket-proxy). Authentication is handled automatically — osa start mints a dev JWT so osa deploy and osa ingestion start work immediately.

Deploy

Deploy conventions to a running archive:

osa deploy

This builds OCI images for your hooks and ingesters, pushes them to the server's registry, and registers the convention.

CLI

Local instance

Command Description
osa init [dir] Scaffold a new project (osa.yaml, .env, .gitignore)
osa start Start the local OSA stack
osa stop Stop the local stack
osa logs [-f] [service] View container logs
osa status Show running containers

Convention workflow

Command Description
osa test [--limit N] Test the full pipeline: ingester → hooks
osa deploy Build OCI images and register conventions
osa meta Print the convention manifest as JSON
osa ingestion start Trigger an ingestion run

Authentication

Command Description
osa login Authenticate via ORCID device flow
osa logout Remove stored credentials
osa link --server <url> Link project to a remote archive

Concepts

Schema — a Pydantic model defining typed metadata fields. Each field can carry units and constraints. The schema is the contract between depositors, hooks, and the archive.

Hook — a pure function decorated with @hook that receives a Record[T] and returns structured results. The return annotation defines the feature table: -> Model is one row per record, -> list[Model] is many, and either may be | None for a hook that sometimes has nothing to contribute. -> None is a pure validator with no table. Anything else is rejected when the hook is defined. Hooks run as OCI containers — the SDK builds the image, the server orchestrates execution.

Convention — a bundle of a schema, hooks, file requirements, and an optional ingester. Conventions are the unit of deployment.

Ingester — an async generator that pulls records from external systems (APIs, databases, file servers) into the archive on a schedule. Each ingester gets its own image, and osa deploy bakes the ingester's name into it as OSA_INGESTER_NAME, so a package with several conventions runs the right ingester for each.

Continuation — how an ingester paginates. The server calls pull() once per batch and hands back whatever the previous batch passed to ctx.set_session(...), starting with None. So read your start position out of the session argument, make sure the state you set advances, and set no session on the final batch — that is what ends the run. An ingester that hands back the state it was given would pull the same window forever, so the server aborts the run instead.

Record[T] — generic container binding a schema type T to its metadata, files, and SRN (Scientific Resource Name).

License

Apache 2.0

Metadata

Release files for osa-py 0.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for osa-py 0.10.0
File Size Uploaded Provenance Actions
osa_py-0.10.0.tar.gz 118.6 kB Without provenance Details for osa_py-0.10.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for osa-py 0.10.0
File Interpreter ABI Platform Actions
osa_py-0.10.0-py3-none-any.whl Python 3 none any Details for osa_py-0.10.0-py3-none-any.whl

Total release size: 192.6 kB

Release files / osa_py-0.10.0.tar.gz

Download URL osa_py-0.10.0.tar.gz
Size 118.6 kB
Tags Source
SHA-256 checksum
How to use checksums
572a751324f2e616f201066acc77dc8d7277a5d75e4cd706803d005c29765fe6
BLAKE2b-256 checksum
How to use checksums
37a643e199a0a4815629a43b06571b7b11eef5aafebf7668e7d64884846045b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / osa_py-0.10.0-py3-none-any.whl

Download URL osa_py-0.10.0-py3-none-any.whl
Size 74.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7c1b3e644598c234b66c45b199d074b0522be5937d7f1ab6bbb0f446e5fe44d3
BLAKE2b-256 checksum
How to use checksums
7fedba668e8cbbbd515daf1ba114c82e79af52f3114addf46e02a98c783ec644
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for osa-py 0.10.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page