radiens-drive-catalog 0.0.4

No project description provided

radiens-drive-catalog

A Python package for programmatically managing large neural datasets stored on Google Drive. It handles Drive scanning, local cataloging, and selective dataset download. Analysis is done locally — this package is purely about data management.

Overview

Neural data is stored as xdat filesets (NeuroNexus format) on a shared Google Drive. Each dataset consists of 3 files sharing a common base_name:

{base_name}_data.xdat
{base_name}.xdat.json
{base_name}_timestamp.xdat

radiens-drive-catalog scans the Drive hierarchy, builds a local catalog indexed by base_name, and lets you query and download datasets selectively. Non-xdat content found alongside datasets — logs directories, PowerPoints, writeups — is also discovered and tracked as assets.

Usage

Datasets

from radiens_drive_catalog import Catalog, Config

config = Config.from_file("config.json")
catalog = Catalog(config)

# Scan Drive and build the catalog (discovers datasets and assets)
catalog.scan()

# Query datasets
catalog.list()                                                          # everything
catalog.list(date_folder="2026-02-15_batch")                            # one date folder
catalog.list(date_folder="2026-02-15_batch", experiment="reaching")     # one experiment

# Access the raw DataFrame
catalog.df

# Check what's available locally
catalog.status()

# Download a dataset
catalog.download("rat01_session3")

# Get the local path, downloading automatically if needed
path = catalog.get_path("rat01_session3")

Assets (non-xdat content)

Non-xdat files and folders (e.g. logs/, PowerPoints, writeups) found inside experiment folders are automatically cataloged as assets during scan().

# Query assets
catalog.assets_df                                                           # all assets
catalog.list_assets(date_folder="2026-02-15_batch")                         # assets in a date folder
catalog.list_assets(experiment="reaching", asset_type="folder")             # folder assets only

# Download an asset
# drive_path is the slash-joined path to the asset's parent folder
catalog.download_asset("2026-02-15_batch/reaching", "logs")

# Get the local path, downloading automatically if needed
path = catalog.get_asset_path("2026-02-15_batch/reaching", "logs")

Assets land under local_data_dir/assets/{drive_path}/{asset_name}, separate from the flat xdat dataset files.

Configuration

Create a config.json (outside your repo — do not commit it):

{
    "credentials_path": "/path/to/service_account.json",
    "root_folder_id": "your-drive-folder-id",
    "local_data_dir": "/path/to/local/data",
    "catalog_path": "/path/to/local/data/catalog.json"
}

Config.from_file() locates the config file using this resolution order:

1. Explicit path argument.
2. RADIENS_DRIVE_CATALOG_CONFIG environment variable.
3. .secrets/config.json in the current working directory.
4. config.json in the current working directory.

# Automatic discovery (env var or well-known paths)
config = Config.from_file()

# Explicit path
config = Config.from_file("/path/to/config.json")

The root_folder_id is the alphanumeric string in the Drive URL when you're inside the root data folder.

Authentication

This package uses a Google service account for shared access among collaborators. To set it up:

1. Create a project in Google Cloud Console
2. Enable the Google Drive API
3. Create a service account and download its JSON credentials file
4. Share your root Drive data folder with the service account's email address (Viewer access is sufficient)
5. Point credentials_path in your config at the downloaded JSON file

Distribute the credentials file to collaborators securely — treat it like a password.

Installation

This project uses uv for dependency management. If you don't have it:

macOS / Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows:

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Then install the project:

uv sync

Development

uv run pytest          # run tests
uv run mypy            # type checking
uv run ruff check .    # linting
uv run ruff format .   # formatting