zombie-squirrel 0.28.0

Generated from aind-library-template

ZOMBIE Squirrel

zombie-squirrel is a set of one-line functions that handle the entire process of caching and retrieving data (and metadata) from AIND data assets.

In the background, the ZOMBIE squirrel repackages data/metadata into dataframes and stores them on S3 in a flat bucket, or in memory for testing.

Installation

pip install zombie-squirrel

Usage

Set backend

export FOREST_TYPE='S3'

Options are 'S3', 'MEMORY'.

Scurry (fetch) data

from zombie_squirrel import unique_project_names

project_names = unique_project_names()

Acorns

get_squirrel_info returns the following information about all available acorns:

Acorn	Description	Location	Type	Partitioned	Columns
unique_project_names	Unique project names across all assets	s3://allen-data-views/data-asset-cache/zs_unique_project_names.pqt	metadata	False	project_name
unique_subject_ids	Unique subject_ids across all assets	s3://allen-data-views/data-asset-cache/zs_unique_subject_ids.pqt	metadata	False	subject_id
unique_genotypes	Unique genotypes across all assets where subject.subject_details.genotype is present	s3://allen-data-views/data-asset-cache/zs_unique_genotypes.pqt	metadata	False	genotype
asset_basics	Commonly used asset metadata, one row per data asset	s3://allen-data-views/data-asset-cache/zs_asset_basics.pqt	metadata	False	_id, _last_modified, modalities, project_name, data_level, subject_id, acquisition_start_time, acquisition_end_time, code_ocean, process_date, genotype, age, acquisition_type, location, name, experimenters, experimenters_normalized, instrument_id, instrument_id_normalized, investigators, investigators_normalized
source_data	Mapping from derived asset names to their source raw asset names	s3://allen-data-views/data-asset-cache/zs_source_data.pqt	metadata	False	name, source_data, pipeline_name, processing_time
quality_control	Quality control table with one row per QC metric	s3://allen-data-views/data-asset-cache/zs_qc/	asset	True (by subject_id)	name, stage, modality, value, status, asset_name
platform_qc	Tag-level QC statuses aggregated per platform, one row per asset/tag combination	s3://allen-data-views/data-asset-cache/zs_platform_qc/	platform	True (by platform)	asset_name, tag, status, timestamp
assets_smartspim	SmartSPIM assets with processing status and neuroglancer links, one row per (asset, channel)	s3://allen-data-views/data-asset-cache/zs_assets_smartspim.pqt	metadata	False	name, raw_name, processed, institution, processing_end_time, stitched_link, raw_link, channel, segmentation_link, quantification_link, alignment_link
platform_fib	Fiber photometry assets in long form, one row per asset/fiber/channel combination	s3://allen-data-views/data-asset-cache/zs_platform_fib.pqt	metadata	False	asset_name, fiber, patch_cord, channel, intended_measurement, targeted_structure
metadata_core	Presence of core aind-data-schema metadata files per asset (True if file is not null)	s3://allen-data-views/data-asset-cache/zs_metadata_core.pqt	metadata	False	_id, _last_modified, subject, data_description, procedures, instrument, acquisition, processing, quality_control
foraging_sessions	Foraging behavior sessions with key performance metrics, one row per session	s3://allen-data-views/data-asset-cache/zs_foraging_sessions.pqt	metadata	False	subject_id, session_date, session, nwb_suffix, rig, trainer, trainer_normalized, task, curriculum_name, curriculum_version, current_stage_actual, foraging_eff, foraging_eff_random_seed, finished_trials, finished_rate, total_trials, bias_naive
behavior_curriculum	Behavior assets with curriculum name and stage, one row per behavior asset	s3://allen-data-views/data-asset-cache/zs_behavior_curriculum.pqt	asset	False	asset_name, curriculum_name, stage_name, stage_node_id

The raw_to_derived function is not a table stored in S3, instead it is used by passing an asset_name (or list of asset names) and a modality. The function returns the latest derived asset matching the requested pattern.

Custom acorn

The custom function allows you to store and retrieve your own user-defined DataFrames in the cache by name. This requires write authentication to the active backend.

from zombie_squirrel import custom
import pandas as pd

df = pd.DataFrame({"col": [1, 2, 3]})
custom("my_data", df)

retrieved_df = custom("my_data")

Hide all the acorns

We run a nightly capsule on Code Ocean with this code to hide all acorns (not the custom ones).

from zombie_squirrel.sync import hide_acorns
hide_acorns()