NeuroVLM

NeuroVLM maps between neuroimaging activation maps and neuroscience text.

System Requirements

• python >=3.10, <3.14
• ubuntu-latest, macos-latest
• GPU (NVIDIA or Apple MPS) or CPU

Installation

Minimal, inference-only installation:

pip install neurovlm

With optional dependencies needed to train and reproduce analyses:

pip install "neurovlm[full]"

Installation take a couple minutes. After installation, calling neurovlm.data.fetch_data() will fetch datasets and models from huggingface, which will be slower.

Demo

See here for the introductory notebook that walks through using all NeuroVLM models. In short:

Fetch NeuroVLM's datasets and models:

from neurovlm.data import fetch_data
fetch_data()

Use the four text/brain inference paths:

from neurovlm import NeuroVLM
from neurovlm.data import load_latent

nvlm = NeuroVLM(device="cuda") # use device="cpu" if GPU not available

# Text-to-brain generation (MSE)
brain_map = nvlm.text("auditory processing").to_brain(head="mse")
brain_map.plot(0, threshold=0.1)

# Brain-to-text generation (QFormer)
auditory = load_latent("networks_neuro")["Du"]["AUD"]
description = nvlm.brain(auditory).to_text(head="qformer")
print(description)

# Text-to-brain retrieval (InfoNCE)
brain_matches = nvlm.text("auditory processing").to_brain(head="infonce")
df_text_to_brain = brain_matches.top_k(3)

# Brain-to-text retrieval (InfoNCE)
text_matches = nvlm.brain(auditory).to_text(head="infonce")
df_brain_to_text = text_matches.top_k(3)

Documentation

See the docs for the API and tutorials.

Reproducibility

All analyses are in Juptyer notebooks. Their are three directories:

1. docs/01_data: Fetch raw data and preprocess
2. docs/02_models: Trains all models
3. docs/03_evaluation: Evaluates models and reproduces publication figures.

License

Apache-2.0 (LICENSE).