SciTeX Dataset (scitex-dataset)
Unified access to neuroscience and scientific datasets
Full Documentation · uv pip install scitex-dataset[all]
Problem and Solution
| # | Problem | Solution |
|---|---|---|
| 1 | Public dataset repositories balkanized -- OpenNeuro (BIDS) + DANDI (NWB) + PhysioNet (WFDB) + Zenodo (generic) + GEO / ChEMBL / ClinicalTrials — different APIs, auth, download tools | Unified fetcher -- stx.dataset.neuroscience.openneuro.fetch_all_datasets() same call shape across all; local FTS5 search across metadata |
| 2 | DataLad setup required — fetching one BIDS dataset means reading DataLad docs first; the barrier is tooling, not knowledge | One-line fetch -- no DataLad setup; the module handles auth, resumption, checksums transparently |
Supported repositories
| Domain | Repository | Description | Data Types |
|---|---|---|---|
| neuroscience | OpenNeuro | Open BIDS neuroimaging platform | MRI, EEG, MEG, iEEG, PET |
| neuroscience | DANDI | BRAIN Initiative archive (NWB) | Electrophysiology, Ophys |
| neuroscience | PhysioNet | Physiological signal databases | ECG, EEG, clinical data |
| neuroscience | GIN | G-Node Infrastructure (Gogs + git-annex) | iEEG, ephys, behaviour |
| general | Zenodo | General scientific data (CERN) | Any research data |
| general | Figshare | Research data sharing platform | Any research data |
| general | OpenML | Machine-learning datasets | Tabular ML benchmarks |
| general | HuggingFace Hub | ML datasets / models (on-demand) | Any |
| biology | GEO | Gene Expression Omnibus (NCBI) | Transcriptomics, microarray |
| pharmacology | MoleculeNet | Molecular ML benchmark suite | SMILES, properties |
| pharmacology | ChEMBL | Bioactivity database (EBI) | IC50/Ki/EC50 assays |
| medical | ClinicalTrials.gov | NIH study registry | Trial metadata |
Table 1. Supported data repositories. Each source is queried via its public API; no authentication required for metadata access.
Demo
flowchart LR
Q["query<br/>'eeg motor imagery'"] --> A["scitex_dataset"]
A --> N["neuroscience/<br/>openneuro · dandi · physionet"]
A --> B["biology/<br/>geo · figshare · zenodo"]
A --> M["medical/<br/>clinicaltrials.gov"]
A --> P["pharmacology/<br/>chembl · moleculenet"]
A --> G["general/<br/>huggingface · openml"]
N & B & M & P & G --> F["filter_results()"]
F --> D["shared SciTeX store<br/>(database.py)"]
D --> O["DataFrame · JSON · download"]
Figure 1. Query flow — one call fans out across domain fetchers, in-memory filtering, and the shared SciTeX store.
Installation
uv pip install "scitex-dataset[all]"
Requires Python >= 3.10.
MCP support:
pip install scitex-dataset[mcp]
Per-feature extras
| Extra | Provides |
|---|---|
mcp |
MCP server (scitex-dataset mcp ..., 20+ FastMCP tools) |
huggingface |
HuggingFace Hub fetch/search/download (huggingface-hub) |
all |
Everything above: uv pip install "scitex-dataset[all]" |
Targeting a specific Python (agent containers, system venvs, Spartan modules)
When the host has multiple Python interpreters — e.g. a pre-built
agent venv at /opt/venv-agent, a Spartan module load Python/3.11.3,
or an existing ~/.venv you want to leave alone — point uv at the
exact interpreter you want with --python:
# install into a specific venv WITHOUT touching project-local .venv
uv pip install --python /opt/venv-agent/bin/python "scitex-dataset[all]"
# one-shot CLI run under that interpreter
uv run --python /opt/venv-agent/bin/python scitex-dataset list-python-apis
# install the CLI as a tool, pinned to the same interpreter
uv tool install --python /opt/venv-agent/bin/python "scitex-dataset[mcp]"
# in a downstream script that bootstraps scitex-dataset, honour
# SCITEX_PYTHON if the caller sets it:
SCITEX_PYTHON="${SCITEX_PYTHON:-/opt/venv-agent/bin/python}"
uv pip install --python "$SCITEX_PYTHON" "scitex-dataset[all]"
The Makefile honours PYTHON=:
PYTHON=/opt/venv-agent/bin/python make install
See issue #38.
Architecture
flowchart LR
U["user code / agent"] --> P["Python API<br/>_api.py"]
U --> C["CLI<br/>scitex-dataset"]
C --> D["dispatch<br/>_sources.py"]
P --> D
M["MCP server<br/>_mcp/"] --> D
D --> N["domain fetchers<br/>neuroscience · biology · medical · pharmacology · general"]
N --> F["filter_results()"]
F --> S["shared SciTeX store<br/>database.py"]
S --> O["DataFrame · JSON · download"]
Figure 2. Module data flow — every interface funnels through the source registry into domain fetchers and the shared store.
Sub-packages group fetchers by scientific domain; each leaf module
exposes a <source>_fetch(query, ...) entry point registered in
_sources.py and re-exported from __init__.py.
Four Interfaces (Python · CLI · MCP · Skills)
Python API ⭐⭐⭐ (primary)
from scitex_dataset import (
openneuro_fetch, dandi_fetch, huggingface_search,
filter_results, list_sources,
db_build, db_search,
)
# 1) Fetch from any catalog source — every <src>_fetch alias is 1:1
# with the dataset_<src>_fetch MCP tool.
records = openneuro_fetch(max_datasets=100)
# 2) Filter + rank in memory.
top = filter_results(
records, modality="eeg", min_subjects=20,
sort_by="downloads", limit=10,
)
# 3) Search HuggingFace Hub directly (on-demand, no catalog).
hf_hits = huggingface_search("biology", limit=20)
# 4) Build the full-text dataset index in the shared SciTeX store.
db_build()
db_search("Alzheimer EEG")
# 5) Inspect the supported sources.
list_sources()["count"] # 11
CLI Commands ⭐⭐
scitex-dataset --help-recursive # Show all commands
# Grammar: scitex-dataset <domain> <dataset> <action>
scitex-dataset neuroscience openneuro fetch -n 100 -o datasets.json -v
scitex-dataset neuroscience dandi fetch -n 50 -o dandi.json -v
scitex-dataset neuroscience physionet fetch -n 50 -v
scitex-dataset general zenodo fetch -q "neuroscience" -n 20
# HuggingFace (general/huggingface noun-group has 4 verbs)
scitex-dataset general huggingface fetch Anthropic/BioMysteryBench-full
scitex-dataset general huggingface search "biology" -n 20 --json
# Local database
scitex-dataset db build # index all catalog sources
scitex-dataset db search "epilepsy EEG" # full-text search
scitex-dataset db show-stats # show statistics
# Introspection
scitex-dataset list-python-apis -v # list Python API tree
scitex-dataset mcp list-tools -v # list MCP tools
MCP Server ⭐⭐
AI agents can discover and query neuroscience datasets autonomously.
| Tool | Description |
|---|---|
dataset_list_sources |
Enumerate the 11 supported sources |
dataset_filter_results |
Filter / rank fetched datasets in memory |
dataset_<src>_fetch |
One per catalog source (10 total) |
dataset_huggingface_fetch / _search / _info / _download_file |
HuggingFace family |
dataset_db_build / _search / _show_stats |
Full-text dataset index |
dataset_skills_list / _get |
Bundled skill pages |
Table 2. 21 MCP tools across catalog fetchers, HuggingFace,
the dataset index, and skill introspection. Every MCP tool has a matching
public Python alias (e.g. scitex_dataset.openneuro_fetch).
scitex-dataset mcp start
Skills ⭐
Skills provide workflow-oriented guides that AI agents query to discover capabilities and usage patterns.
scitex-dataset skills list # List available skill pages
scitex-dataset skills get SKILL # Show main skill page
scitex-dev skills export --package scitex-dataset # Export to Claude Code
| Skill | Content |
|---|---|
installation |
pip install + extras + verify |
quick-start |
Search, fetch, sort across sources |
python-api |
Top-level exports, domain submodules, examples |
cli-reference |
CLI grammar, domains, flags, config precedence |
mcp-tools |
MCP tools for AI agents |
env-vars |
SCITEX_* environment variables |
data-sources |
All 11 supported repositories |
Part of SciTeX
scitex-dataset is part of SciTeX. Install via
the umbrella with pip install scitex[dataset] to use as
scitex.dataset (Python) or scitex dataset ... (CLI).
import scitex
from scitex_dataset import fetch_all_datasets, format_dataset
@scitex.session
def main(logger=scitex.INJECTED):
datasets = fetch_all_datasets(max_datasets=100, logger=logger)
formatted = [format_dataset(ds) for ds in datasets]
scitex.io.save(formatted, "openneuro_datasets.json")
return 0
The SciTeX ecosystem follows the Four Freedoms for Research, inspired by the Free Software Definition:
Four Freedoms for Research
- The freedom to run your research anywhere -- your machine, your terms.
- The freedom to study how every step works -- from raw data to final manuscript.
- The freedom to redistribute your workflows, not just your papers.
- The freedom to modify any module and share improvements with the community.
AGPL-3.0 -- because we believe research infrastructure deserves the same freedoms as the software it runs on.
Metadata
Release files for scitex-dataset 0.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scitex_dataset-0.8.0.tar.gz | 4.8 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scitex_dataset-0.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 9.2 MB
Release files / scitex_dataset-0.8.0.tar.gz
| Download URL | scitex_dataset-0.8.0.tar.gz |
|---|---|
| Size | 4.8 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dbf8f6666a08a30a4e96a1c4dc4d853ce064a794781f4c3a5695a3f10ee109cd
|
|
BLAKE2b-256 checksum How to use checksums |
f132599868bb86d4feb49ca261ad1bc2c258127f0a5982713a6fce7271449d25
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / scitex_dataset-0.8.0-py3-none-any.whl
| Download URL | scitex_dataset-0.8.0-py3-none-any.whl |
|---|---|
| Size | 4.4 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f7267274cc844066351b60a7c1e229bf4c1bfbb94ad6ae102264cfabd2155efa
|
|
BLAKE2b-256 checksum How to use checksums |
154ecc9bcfd95eba9704d14ca2326fd6b2c565c11ec183d31dc0d9806a8aa050
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency log