Skip to main content

SciTeX Dataset (scitex-dataset)

SciTeX

Unified access to neuroscience and scientific datasets

Full Documentation · uv pip install scitex-dataset[all]

pypi python docs docs

tests cov


Problem and Solution

# Problem Solution
1 Public dataset repositories balkanized -- OpenNeuro (BIDS) + DANDI (NWB) + PhysioNet (WFDB) + Zenodo (generic) + GEO / ChEMBL / ClinicalTrials — different APIs, auth, download tools Unified fetcher -- stx.dataset.neuroscience.openneuro.fetch_all_datasets() same call shape across all; local FTS5 search across metadata
2 DataLad setup required — fetching one BIDS dataset means reading DataLad docs first; the barrier is tooling, not knowledge One-line fetch -- no DataLad setup; the module handles auth, resumption, checksums transparently

Supported repositories

Domain Repository Description Data Types
neuroscience OpenNeuro Open BIDS neuroimaging platform MRI, EEG, MEG, iEEG, PET
neuroscience DANDI BRAIN Initiative archive (NWB) Electrophysiology, Ophys
neuroscience PhysioNet Physiological signal databases ECG, EEG, clinical data
neuroscience GIN G-Node Infrastructure (Gogs + git-annex) iEEG, ephys, behaviour
general Zenodo General scientific data (CERN) Any research data
general Figshare Research data sharing platform Any research data
general OpenML Machine-learning datasets Tabular ML benchmarks
general HuggingFace Hub ML datasets / models (on-demand) Any
biology GEO Gene Expression Omnibus (NCBI) Transcriptomics, microarray
pharmacology MoleculeNet Molecular ML benchmark suite SMILES, properties
pharmacology ChEMBL Bioactivity database (EBI) IC50/Ki/EC50 assays
medical ClinicalTrials.gov NIH study registry Trial metadata

Table 1. Supported data repositories. Each source is queried via its public API; no authentication required for metadata access.

Demo

flowchart LR
    Q["query<br/>'eeg motor imagery'"] --> A["scitex_dataset"]
    A --> N["neuroscience/<br/>openneuro · dandi · physionet"]
    A --> B["biology/<br/>geo · figshare · zenodo"]
    A --> M["medical/<br/>clinicaltrials.gov"]
    A --> P["pharmacology/<br/>chembl · moleculenet"]
    A --> G["general/<br/>huggingface · openml"]
    N & B & M & P & G --> F["filter_results()"]
    F --> D["shared SciTeX store<br/>(database.py)"]
    D --> O["DataFrame · JSON · download"]

Figure 1. Query flow — one call fans out across domain fetchers, in-memory filtering, and the shared SciTeX store.

Installation

uv pip install "scitex-dataset[all]"

Requires Python >= 3.10.

MCP support: pip install scitex-dataset[mcp]

Per-feature extras
Extra Provides
mcp MCP server (scitex-dataset mcp ..., 20+ FastMCP tools)
huggingface HuggingFace Hub fetch/search/download (huggingface-hub)
all Everything above: uv pip install "scitex-dataset[all]"
Targeting a specific Python (agent containers, system venvs, Spartan modules)

When the host has multiple Python interpreters — e.g. a pre-built agent venv at /opt/venv-agent, a Spartan module load Python/3.11.3, or an existing ~/.venv you want to leave alone — point uv at the exact interpreter you want with --python:

# install into a specific venv WITHOUT touching project-local .venv
uv pip install --python /opt/venv-agent/bin/python "scitex-dataset[all]"

# one-shot CLI run under that interpreter
uv run --python /opt/venv-agent/bin/python scitex-dataset list-python-apis

# install the CLI as a tool, pinned to the same interpreter
uv tool install --python /opt/venv-agent/bin/python "scitex-dataset[mcp]"

# in a downstream script that bootstraps scitex-dataset, honour
# SCITEX_PYTHON if the caller sets it:
SCITEX_PYTHON="${SCITEX_PYTHON:-/opt/venv-agent/bin/python}"
uv pip install --python "$SCITEX_PYTHON" "scitex-dataset[all]"

The Makefile honours PYTHON=:

PYTHON=/opt/venv-agent/bin/python make install

See issue #38.

Architecture

flowchart LR
    U["user code / agent"] --> P["Python API<br/>_api.py"]
    U --> C["CLI<br/>scitex-dataset"]
    C --> D["dispatch<br/>_sources.py"]
    P --> D
    M["MCP server<br/>_mcp/"] --> D
    D --> N["domain fetchers<br/>neuroscience · biology · medical · pharmacology · general"]
    N --> F["filter_results()"]
    F --> S["shared SciTeX store<br/>database.py"]
    S --> O["DataFrame · JSON · download"]

Figure 2. Module data flow — every interface funnels through the source registry into domain fetchers and the shared store.

Sub-packages group fetchers by scientific domain; each leaf module exposes a <source>_fetch(query, ...) entry point registered in _sources.py and re-exported from __init__.py.

Four Interfaces (Python · CLI · MCP · Skills)

Python API ⭐⭐⭐ (primary)
from scitex_dataset import (
    openneuro_fetch, dandi_fetch, huggingface_search,
    filter_results, list_sources,
    db_build, db_search,
)

# 1) Fetch from any catalog source — every <src>_fetch alias is 1:1
#    with the dataset_<src>_fetch MCP tool.
records = openneuro_fetch(max_datasets=100)

# 2) Filter + rank in memory.
top = filter_results(
    records, modality="eeg", min_subjects=20,
    sort_by="downloads", limit=10,
)

# 3) Search HuggingFace Hub directly (on-demand, no catalog).
hf_hits = huggingface_search("biology", limit=20)

# 4) Build the full-text dataset index in the shared SciTeX store.
db_build()
db_search("Alzheimer EEG")

# 5) Inspect the supported sources.
list_sources()["count"]   # 11

Full API reference

CLI Commands ⭐⭐
scitex-dataset --help-recursive             # Show all commands

# Grammar: scitex-dataset <domain> <dataset> <action>
scitex-dataset neuroscience openneuro fetch -n 100 -o datasets.json -v
scitex-dataset neuroscience dandi fetch -n 50 -o dandi.json -v
scitex-dataset neuroscience physionet fetch -n 50 -v
scitex-dataset general zenodo fetch -q "neuroscience" -n 20

# HuggingFace (general/huggingface noun-group has 4 verbs)
scitex-dataset general huggingface fetch Anthropic/BioMysteryBench-full
scitex-dataset general huggingface search "biology" -n 20 --json

# Local database
scitex-dataset db build                     # index all catalog sources
scitex-dataset db search "epilepsy EEG"     # full-text search
scitex-dataset db show-stats                # show statistics

# Introspection
scitex-dataset list-python-apis -v          # list Python API tree
scitex-dataset mcp list-tools -v            # list MCP tools

Full CLI reference

MCP Server ⭐⭐

AI agents can discover and query neuroscience datasets autonomously.

Tool Description
dataset_list_sources Enumerate the 11 supported sources
dataset_filter_results Filter / rank fetched datasets in memory
dataset_<src>_fetch One per catalog source (10 total)
dataset_huggingface_fetch / _search / _info / _download_file HuggingFace family
dataset_db_build / _search / _show_stats Full-text dataset index
dataset_skills_list / _get Bundled skill pages

Table 2. 21 MCP tools across catalog fetchers, HuggingFace, the dataset index, and skill introspection. Every MCP tool has a matching public Python alias (e.g. scitex_dataset.openneuro_fetch).

scitex-dataset mcp start

Full MCP specification

Skills ⭐

Skills provide workflow-oriented guides that AI agents query to discover capabilities and usage patterns.

scitex-dataset skills list              # List available skill pages
scitex-dataset skills get SKILL         # Show main skill page
scitex-dev skills export --package scitex-dataset  # Export to Claude Code
Skill Content
installation pip install + extras + verify
quick-start Search, fetch, sort across sources
python-api Top-level exports, domain submodules, examples
cli-reference CLI grammar, domains, flags, config precedence
mcp-tools MCP tools for AI agents
env-vars SCITEX_* environment variables
data-sources All 11 supported repositories

Part of SciTeX

scitex-dataset is part of SciTeX. Install via the umbrella with pip install scitex[dataset] to use as scitex.dataset (Python) or scitex dataset ... (CLI).

import scitex
from scitex_dataset import fetch_all_datasets, format_dataset

@scitex.session
def main(logger=scitex.INJECTED):
    datasets = fetch_all_datasets(max_datasets=100, logger=logger)
    formatted = [format_dataset(ds) for ds in datasets]
    scitex.io.save(formatted, "openneuro_datasets.json")
    return 0

The SciTeX ecosystem follows the Four Freedoms for Research, inspired by the Free Software Definition:

Four Freedoms for Research

  1. The freedom to run your research anywhere -- your machine, your terms.
  2. The freedom to study how every step works -- from raw data to final manuscript.
  3. The freedom to redistribute your workflows, not just your papers.
  4. The freedom to modify any module and share improvements with the community.

AGPL-3.0 -- because we believe research infrastructure deserves the same freedoms as the software it runs on.


SciTeX

Metadata

Release files for scitex-dataset 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scitex-dataset 0.8.0
File Size Uploaded
scitex_dataset-0.8.0.tar.gz 4.8 MB Details

Built distribution (wheel)

Table of built distributions (wheels) for scitex-dataset 0.8.0
File Interpreter ABI Platform
scitex_dataset-0.8.0-py3-none-any.whl Python 3 none any Details

Total release size: 9.2 MB

Release files / scitex_dataset-0.8.0.tar.gz

Download URL scitex_dataset-0.8.0.tar.gz
Size 4.8 MB
Tags Source
SHA-256 checksum
How to use checksums
dbf8f6666a08a30a4e96a1c4dc4d853ce064a794781f4c3a5695a3f10ee109cd
BLAKE2b-256 checksum
How to use checksums
f132599868bb86d4feb49ca261ad1bc2c258127f0a5982713a6fce7271449d25
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.

Transparency log

Release files / scitex_dataset-0.8.0-py3-none-any.whl

Download URL scitex_dataset-0.8.0-py3-none-any.whl
Size 4.4 MB
Tags Python 3
SHA-256 checksum
How to use checksums
f7267274cc844066351b60a7c1e229bf4c1bfbb94ad6ae102264cfabd2155efa
BLAKE2b-256 checksum
How to use checksums
154ecc9bcfd95eba9704d14ca2326fd6b2c565c11ec183d31dc0d9806a8aa050
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.

Transparency log

Release history Release notifications | RSS feed

0.8.1

2 release files

This release

0.8.0 This release

2 release files

0.7.1

2 release files

0.7.0

2 release files

0.6.0

2 release files

0.5.0

2 release files

0.4.0

2 release files

0.3.10

2 release files

0.3.9

2 release files

0.3.8

2 release files

0.3.7

2 release files

0.3.6

2 release files

0.3.5

2 release files

0.3.4

2 release files

0.3.3

2 release files

0.3.2

2 release files

0.3.1

2 release files

0.3.0

2 release files

0.2.1

2 release files

0.1.5

2 release files

0.1.4

2 release files

0.1.3

2 release files

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page